Relationships
#909 Add core-concept explanation pages to the manual
Opened by stack72 · 7/1/2026· Shipped 7/1/2026
Problem
The manual explanation section (content/manual/explanation/) has 10 pages, but the three foundational concepts of Swamp — models, data, and workflows — have no dedicated explanation pages. "How Swamp Works" (explanation/how-swamp-works.md, order 1) provides a one-paragraph overview of each concept, but a user who finishes that page has no deeper explanation to follow.
The existing explanation pages cover peripheral topics well (AI agent integration, extension scorecard, extension publishing lifecycle, background autoupdating, API key scoping, workflow suspension, giga-swamp, remote execution, swamp serve). But if someone asks "what is a model type vs a model definition?" or "how does the data layer work?" or "what happens when a workflow runs?", there is no explanation page to point them to.
Reference pages document the WHAT (YAML schemas, CLI commands, JSON shapes). Explanation pages document the WHY (design decisions, conceptual relationships, mental models). The reference docs shipped in #907 / PR #783 made the WHAT comprehensive. This issue fills the WHY gap.
Scope
This issue covers explanation documentation only (diataxis type: explanation). Every page must follow the explanation rules: explain why things work the way they do, help the reader build a mental model, do not include step-by-step instructions (that's how-to) or teach through a project (that's tutorial).
Navigation: Tile Budget
The explanation section currently has 10 top-level tiles. This issue adds 6 new standalone pages and integrates 2 topics into existing pages, for a net of +6 tiles (10 → 16). This is acceptable because:
- Explanation pages are conceptual reading — users browse them selectively, not as a flat list to scan.
- The three core-concept pages (models, data, workflows) should be the most prominent tiles after "How Swamp Works".
- Two smaller topics (webhooks, configuration) are folded into existing pages rather than adding tiles.
If 16 feels heavy after implementation, consider grouping the 3 core-concept tiles under a visual separator or "Core Concepts" sub-heading in the nav — but don't create sub-pages (explanation pages should stand alone).
Source Material
The swamp source repository at ~/code/swamp-club/swamp contains design documents in design/ that serve as authoritative source material:
design/model.md— model system design (types, definitions, methods, output specs)design/workflow.md— workflow orchestration design (DAG, jobs, steps, dependencies, triggers)design/data-query.md— data query system design (CEL predicates, DataRecord, projections)design/extension.md— extension system design (trust, publishing, registry)design/high-level.md— architecture overviewdesign/repo.md— repository designdesign/reports.md— reporting system design
Read these design docs before writing each explanation page. They contain the design rationale and architectural decisions that explanation pages should communicate. Do NOT copy them verbatim — translate the internal design language into user-facing explanation prose.
Also read the corresponding reference pages in content/manual/reference/ — the explanation should complement the reference, not repeat it. Link to reference pages for details ("for the full YAML schema, see Model Definitions").
Detailed Plan
New Page 1: "Models, Types, and Methods" (explanation/models-types-and-methods.md, order 2)
Why this page is needed: The model system has three distinct concepts that users routinely conflate: model types, model definitions, and model instances. The reference page documents the YAML schema but never explains why these layers exist or how they relate.
What to cover:
Why types and definitions are separate. A model type is a reusable TypeScript module that defines methods, arguments, and output specs. A model definition is a YAML file that configures an instance of that type with specific arguments. The type is code; the definition is configuration. Analogy: a type is a class, a definition is an object instantiated from that class. Explain why this separation matters: types are published as extensions and shared across repos; definitions are repo-local and hold environment-specific values (account IDs, regions, naming conventions).
The method execution model. A method is a named operation on a model (create, start, stop, destroy, sync, lookup). Methods receive arguments, access vault secrets, read data from other models via CEL, and produce data outputs (resources and files). Explain the execution lifecycle: definition evaluation (CEL expressions resolved) → pre-flight checks → method execution → data output write → post-execution reports. Explain why checks exist before methods, not after.
How auto-creation works. When
swamp model method run @type method-name definition-nameis used, swamp auto-creates a definition in.swamp/auto-definitions/if one doesn't exist. Explain why this exists (rapid prototyping, AI agent ergonomics) and how it differs from explicitswamp model create(auto-definitions are ephemeral, not checked into version control).Output specs: resources vs files. Resources are structured JSON data with schemas that can be referenced by other models via
data.latest(). Files are opaque binary/text artifacts identified by MIME type. Explain why the distinction exists: resources are queryable and composable; files are for logs, artifacts, and content that doesn't have a schema.The built-in
command/shelltype. Explain why it exists (escape hatch for ad-hoc commands), why it should NOT be used for building integrations (CLAUDE.md rule: "Thecommand/shellmodel is ONLY for ad-hoc one-off shell commands, NEVER for wrapping CLI tools or building integrations"), and what to use instead (extension models or community extensions).
Cross-references: Link to reference/model-definitions for YAML schema, reference/extensions/model for TypeScript model interface, explanation/how-swamp-works for context.
Source material: ~/code/swamp-club/swamp/design/model.md
New Page 2: "The Data Layer" (explanation/the-data-layer.md, order 3)
Why this page is needed: The data layer ("The Swamp") is the mechanism that makes models composable — one model's output becomes another model's input through CEL expressions. But there is no explanation of how data flows, how versioning works, or why the data layer is designed the way it is.
What to cover:
Data as the composition mechanism. Models don't call each other directly. Instead, each model produces versioned data outputs, and other models (or workflows) reference that data via CEL expressions like
data.latest("model-name", "data-name").attributes.field. Explain why this indirect coupling exists: it allows models to run independently, in any order, and compose without knowing about each other. The data layer is the integration bus.Versioning and immutability. Every method execution produces a new version of its data outputs. Previous versions are retained (up to GC limits). Explain why immutability matters: it provides auditability, rollback capability, and safe concurrent access.
data.latest()always returns the most recent version;data.version()retrieves a specific version.Data lifetime and garbage collection. Data has a lifetime policy (ephemeral, job, workflow, duration, infinite) that controls when it becomes eligible for GC. Explain the design: ephemeral data is for intermediate computation that doesn't need to persist; infinite data is for long-lived state. GC runs via
swamp data gcand respects both lifetime expiry and version retention limits.Querying across models.
data.query()andswamp data querysearch across all data in a repository using CEL predicates. Explain why a query layer exists on top ofdata.latest(): it enables cross-model discovery ("find all data tagged with environment=production"), reporting, and debugging. The query layer treats every data artifact as aDataRecordwith uniform fields regardless of which model produced it.Tags and provenance. Every data artifact carries provenance metadata (which model produced it, which method, which workflow run, timestamps) and user-defined tags. Explain why provenance is automatic: it provides the audit trail that connects a data artifact back to the execution that created it.
Sensitive fields and redaction. Output specs can mark fields as sensitive. These fields are stored but redacted from CLI output and logs. Explain the design: secrets that pass through models (API keys returned by a cloud provider) should not leak into terminal output or audit logs, but they still need to be available to downstream models via
data.latest().
Cross-references: Link to reference/data for CLI commands and output specs, reference/cel-expressions for data.latest() and data.query() syntax, reference/vaults for how secrets reach models.
Source material: ~/code/swamp-club/swamp/design/data-query.md, ~/code/swamp-club/swamp/design/model.md (output spec sections)
New Page 3: "The Workflow Execution Model" (explanation/the-workflow-execution-model.md, order 4)
Why this page is needed: The workflow reference page documents the YAML schema (jobs, steps, dependencies, triggers). But it never explains how the execution engine works: how dependencies are resolved, what happens when a step fails, how forEach iteration interacts with concurrency, or why workflows are structured as DAGs rather than sequential scripts.
What to cover:
Why DAGs, not scripts. A workflow is a directed acyclic graph of steps, not a linear script. Steps declare dependencies on other steps; the engine resolves the topological order and runs independent steps concurrently. Explain why: a DAG expresses the actual dependency structure (step B needs step A's output, but step C is independent of both), which allows maximum parallelism without the user manually managing concurrency.
Jobs and steps. A workflow contains jobs; each job contains steps. Jobs are the unit of concurrency control — you can set a concurrency limit on a job. Steps within a job run according to their dependency graph. Explain when to use multiple jobs vs multiple steps in one job: jobs are for logically distinct phases (build, test, deploy); steps are for operations within a phase that may depend on each other.
Dependency resolution and trigger conditions. Steps can depend on other steps via
depends_on. The default trigger issucceeded— the step runs only if all dependencies succeeded. Other triggers:failed(run cleanup on failure),completed(run regardless of outcome). Explain how this enables error handling patterns: a notification step triggered onfailedsends an alert without needing try/catch in the step itself.Data flow through workflows. Steps produce data outputs that downstream steps reference via CEL. The workflow engine makes each step's data available under the step's name. Explain how
data.latest()in a workflow step resolves: it reads the data produced by the named model in a prior step, not the latest data in the repository (workflow-scoped resolution vs repo-scoped resolution). MentiondataOutputOverridesandvarydimensions for controlling how data is keyed.forEach iteration. A step can iterate over a list using
forEach. Each iteration runs the step's method with a different value from the list, accessible viaself.item. Iterations run concurrently (up to the job's concurrency limit). Explain why forEach exists as a workflow primitive rather than a loop in the model: it allows the workflow engine to track each iteration independently (separate data outputs, separate status, separate logs).Manual approval gates. A
manual_approvalstep suspends the workflow and waits for human sign-off. Explain the design: approval and resume are separate operations because the approver and the executor may be different people, and resume may require new inputs (e.g., a freshly minted auth key). Link to the existingexplanation/understanding-workflow-suspensionfor the deep dive.Triggers and scheduling. Workflows can have a
triggerblock with a cron schedule. Whenswamp serveis running, it executes triggered workflows on schedule. Explain why triggers are in the workflow definition rather than in a separate scheduler: it keeps the "what to run" and "when to run it" together.
Cross-references: Link to reference/workflows for YAML schema, explanation/understanding-workflow-suspension for suspension deep dive, reference/swamp-serve/webhooks for event-triggered execution.
Source material: ~/code/swamp-club/swamp/design/workflow.md
New Page 4: "Extension Trust and the Supply Chain" (explanation/extension-trust.md, order 5)
Why this page is needed: Extensions run arbitrary TypeScript code on the user's machine. The trust model that governs which extensions are allowed, how dependencies are audited, and what "Verified by Swamp" means is spread across multiple reference pages (scorecard rubric, publishing commands, management commands) but never explained as a coherent system.
What to cover:
The trust problem. Extensions are published by collectives (teams) on the registry. Anyone can publish. When a user runs
swamp extension pull @someone/something, they are downloading and executing code from that collective. Explain why explicit trust is needed rather than relying on the registry alone.Trusted collectives. A repository maintains a list of trusted collectives (
swamp extension trust add). Extensions from trusted collectives are auto-resolved duringswamp extension install. Extensions from untrusted collectives require explicitswamp extension pull. Explain why trust is per-repository: different repos have different risk profiles (a production infrastructure repo should trust fewer collectives than a playground repo).Auto-trust for membership collectives.
swamp extension trust auto-trust trueautomatically trusts all collectives the authenticated user belongs to. Explain the design tradeoff: convenient for teams that publish their own extensions, but should be off for repos managed by CI (where the authenticated identity may have broad membership).Dependency trust scoring. The extension quality scorecard includes a "dependency trust" factor that audits npm dependencies against OSV.dev (vulnerability database) and the npm registry (deprecation status). Explain why this is a publishing gate: a single compromised or deprecated dependency in an extension can affect every repo that installs it. The gate blocks
swamp extension pushwhen dependencies have known vulnerabilities or are deprecated.The "Verified by Swamp" badge. This is a separate badge from the quality score — it indicates that the Swamp team has reviewed the extension for safety and correctness. Explain why it's not part of the automated score: automated checks verify surface-level quality (docs, types, dependencies); verification requires human review of the code's behavior. Most extensions don't have this badge and that's expected.
Yanking and deprecation. A published extension can be yanked (removed from resolution but still downloadable by pinned version) or deprecated (flagged with a warning and optional successor). Explain the difference: yanking is for safety (compromised version, accidental publish); deprecation is for lifecycle (superseded by a better extension).
Cross-references: Link to reference/extensions/scorecard-rubric for scoring details, reference/extensions/publishing for yank/deprecate commands, reference/extensions/management for trust commands, explanation/extension-scorecard for quality scoring.
Source material: ~/code/swamp-club/swamp/design/extension.md
New Page 5: "The Audit System" (explanation/the-audit-system.md, order 11)
Why this page is needed: swamp audit shows a timeline of commands run in a repository — both swamp commands and direct CLI commands (aws, terraform, kubectl, etc.) captured by audit hooks. But there is no explanation of why this exists, what it captures, or how the audit hook integration works.
What to cover:
Why audit exists. When AI agents operate swamp repos, the human operator needs to know what happened. The audit timeline shows every command in chronological order with timestamps, session IDs, and whether it was a swamp command or a direct CLI invocation. Explain the design: the audit system provides the "what happened in my repo while I was away" answer.
Two sources of audit data. Swamp commands are logged automatically by the CLI. Direct CLI commands (aws, terraform, gcloud) are captured by audit hooks — shell integration that records commands run in the repo directory. Explain why both sources matter: a swamp workflow might run
swamp model method run deploy, but the model's method internally callsaws ec2 run-instances. The audit timeline shows both, giving a complete picture of what touched the environment.Audit hooks and AI tool integration. Each supported AI tool (Claude Code, Cursor, Kiro, OpenCode, Codex, Copilot) has its own audit hook integration.
swamp repo init --tool <tool>sets up the hook for the chosen tool.swamp doctor auditverifies the hook is healthy. Explain why hooks are tool-specific: each AI coding tool has a different mechanism for intercepting shell commands (Claude Code uses hooks in settings, Cursor uses shell integration, etc.).Sessions. Each interactive terminal session or agent run gets a session ID. The audit timeline can be filtered by session to see what one particular session did. Explain why sessions matter: when multiple agents or humans work in the same repo concurrently, session filtering isolates one actor's commands.
What audit is NOT. The audit system records commands, not outcomes. It does not record whether a command succeeded or failed, what output it produced, or what resources it modified. For execution outcomes, use
swamp model method historyorswamp workflow history. Explain the design boundary: audit is a lightweight command log, not an execution record store.
Cross-references: Link to reference/operational-commands for swamp audit CLI reference, reference/doctor for swamp doctor audit, the how-to guides for AI agent setup.
New Page 6: "Datastore Architecture" (explanation/datastore-architecture.md, order 12)
Why this page is needed: The datastore is the persistence layer beneath everything — definitions, workflow runs, data outputs, secrets, audit logs. The reference page documents configuration options, but users with S3 backends, shared datastores, or distributed teams need to understand how the datastore actually works internally.
What to cover:
What the datastore stores. Evaluated definitions, workflow run records, data outputs (resources and files), vault-encrypted secrets, audit logs, extension catalog metadata, and the WAL. Explain why all of these live in one datastore rather than separate storage systems: a single datastore makes backup, migration, and sharing atomic.
Local vs external vs S3 backends. The default datastore is
.swamp/in the repository (local filesystem). An external filesystem path (NFS mount) enables sharing across machines. An S3-compatible backend enables cloud-native shared state. Explain the tradeoffs: local is simplest (no configuration, no network), external filesystem enables shared access without cloud dependencies, S3 enables geo-distributed access with versioning.The catalog database. The datastore maintains a SQLite catalog that indexes all artifacts for fast queries. The catalog is a cache — it can be rebuilt from the raw files. Explain why a catalog exists on top of the filesystem: scanning the filesystem for every
data.query()ordata.latest()would be slow; the catalog provides indexed lookup.swamp datastore compactcheckpoints the WAL and vacuums the catalog.Hydration strategies. When using an S3 backend, the datastore can hydrate data on-demand (lazy) or eagerly (sync all on start). Explain why lazy hydration is the default: most commands only need a subset of the data, so downloading everything upfront wastes bandwidth and time.
swamp datastore syncforces a full sync when needed.Distributed locking. When multiple machines share a datastore (S3 or NFS), concurrent writes must be serialized. The datastore uses a lock file (filesystem) or conditional writes (S3) to prevent conflicts. Explain why per-model locking exists: it scopes contention to individual models rather than locking the entire datastore.
swamp datastore lock statusandswamp datastore lock releasemanage stuck locks.Relationship to giga-swamp. Namespaces partition a shared datastore by repository. Each namespace has its own data, definitions, and workflow runs, but they share the underlying storage. Cross-namespace queries are possible via
data.query()with namespace prefixes. Link to the existingexplanation/giga-swampfor the full namespace design.
Cross-references: Link to reference/datastore-configuration for CLI commands and YAML config, explanation/giga-swamp for namespaces, reference/vaults for how secrets are stored.
Integration 1: Webhooks → existing "swamp serve" explanation page
Do NOT create a new page. Integrate webhook content into the existing explanation/swamp-serve.md (order 10).
Add a new section ## Webhooks and Event-Driven Workflows covering:
How webhooks bridge external events to workflows.
swamp serve --webhookregisters an HTTP endpoint that verifies an incoming webhook signature, extracts the payload, and triggers a workflow run. The workflow receives the payload inwebhook.*CEL variables.Provider schemes and signature verification. Each provider scheme (github, linear, stripe, slack, generic) implements a different signature verification algorithm. Explain why multiple schemes exist: each SaaS platform signs webhooks differently (GitHub uses HMAC-SHA256 on the body, Stripe uses a timestamp+signature scheme, etc.).
The secret source pattern. Webhook secrets can be provided as literal strings,
@env=VAR(from environment variable), or@file=/path(from a file). Explain why env and file indirection exist: they keep secrets out of process argv (which is visible inpsoutput).When to use webhooks vs cron triggers. Cron triggers (
trigger.schedulein workflow YAML) run workflows on a time schedule. Webhooks run workflows in response to external events. Explain the design: cron is for polling-style automation (check every hour); webhooks are for event-driven automation (deploy when code is pushed).
Source material: The new reference/swamp-serve/webhooks.md page (shipped in PR #783) for the spec details. ~/code/swamp-club/swamp/design/workflow.md for trigger design.
Integration 2: Configuration Hierarchy → existing "How Swamp Works" explanation page
Do NOT create a new page. Integrate configuration content into the existing explanation/how-swamp-works.md (order 1).
Add a new section ## Configuration covering:
The configuration layers.
.swamp.yaml(repo-level, checked into version control),swamp config set(user-level, stored in~/.config/swamp/), environment variables (SWAMP_*), and CLI flags. Explain the precedence order: CLI flags > environment variables > user config > repo config > defaults.What lives where. Repo config (
.swamp.yaml) holds tool selection, trusted collectives, auto-GC settings, datastore config — things that should be shared across the team. User config holds personal preferences (log level, telemetry opt-out). Environment variables override both for CI/CD and scripted environments. Explain why the split exists: team settings should be version-controlled; personal preferences should not.Environment variable conventions.
SWAMP_REPO_DIR,SWAMP_SERVE_URL,SWAMP_SERVER_TOKEN,SWAMP_LOG_LEVEL,SWAMP_NO_TELEMETRY,SWAMP_DATASTORE,SWAMP_MAX_CONCURRENT_STEPS. Explain the naming convention and that env vars always win over config file values.
Cross-references: Link to reference/repository-configuration for the full .swamp.yaml schema.
Order Numbers
Existing explanation pages and their order values for reference:
| Page | Current order |
|---|---|
| How Swamp Works | 1 |
| AI Agent Integration | 2 |
| Extension Scorecard | 3 |
| Extension Publishing Lifecycle | 4 |
| Background Autoupdating | 5 |
| API Key Scoping | 6 |
| Understanding Workflow Suspension | 7 |
| Giga-Swamp and Namespaces | 8 |
| Remote Execution | 9 |
| swamp serve | 10 |
The three core-concept pages should appear immediately after "How Swamp Works" to establish the foundational mental model before the more specific topics. This requires renumbering existing pages:
| Page | New order |
|---|---|
| How Swamp Works | 1 |
| Models, Types, and Methods | 2 |
| The Data Layer | 3 |
| The Workflow Execution Model | 4 |
| Extension Trust and the Supply Chain | 5 |
| AI Agent Integration | 6 |
| Extension Scorecard | 7 |
| Extension Publishing Lifecycle | 8 |
| Background Autoupdating | 9 |
| API Key Scoping | 10 |
| Understanding Workflow Suspension | 11 |
| Giga-Swamp and Namespaces | 12 |
| Remote Execution | 13 |
| swamp serve | 14 |
| The Audit System | 15 |
| Datastore Architecture | 16 |
Implementation Notes
Writing style for explanation pages
Read the existing explanation pages to match the tone and structure:
- Lead with the design decision, not the implementation. "Swamp separates model types from model definitions because..." not "A model type is a TypeScript module that..."
- Use analogies where they illuminate. The existing pages use analogies effectively (e.g., "How Swamp Works" compares the data layer to a shared state bus).
- Explain tradeoffs, not just the chosen design. "The alternative would be X, but that fails when Y."
- No step-by-step instructions. Link to how-to guides for that. The only code in explanation pages should be illustrative snippets showing the concept, not runnable recipes.
- Keep pages focused. Each page should answer one question ("what is a model type and why does it exist?"), not cover a topic exhaustively. If a section grows beyond a few paragraphs, it probably belongs in its own page.
Verify concepts against the CLI
For every concept described, verify it against the actual CLI behavior:
- Create a sandbox repo:
SWAMP_SANDBOX=$(mktemp -d) && cd "$SWAMP_SANDBOX" && swamp repo init - Run the relevant commands to confirm the behavior matches the explanation
- Use
swamp help <command>for authoritative flag/argument descriptions
Terminology
Per CLAUDE.md:
- "Swamp" capitalized in prose,
swamplowercase in command invocations - "The Swamp" (capital T) is reserved as the alias for the data layer
- "operative" not "operator" for users
- "collective" not "organization"
- "agent" lowercase in running prose
Meta descriptions
Per CLAUDE.md SEO rules, frontmatter description fields must be 150-160 characters. Verify character count for every new and modified page.
Acceptance Criteria
- 6 new explanation pages created with correct frontmatter (title, description 150-160 chars, type: explanation, order)
- Webhooks section integrated into existing
explanation/swamp-serve.md - Configuration section integrated into existing
explanation/how-swamp-works.md - All existing explanation pages renumbered (order field updated) per the table above
- Core concepts explained with design rationale, not just implementation description
- Each page links to relevant reference pages (complement, don't repeat)
- Each page links back to "How Swamp Works" for context
- Concepts verified against actual CLI behavior in a sandbox repo
- No step-by-step instructions in explanation pages (link to how-to guides)
-
deno task checkpasses - Terminology rules followed (Swamp capitalization, operative, collective)
Shipped
Click a lifecycle step above to view its details.
Sign in to post a ripple.