THE DATA LAYER

The primary composition mechanism between models in Swamp is data, not calls. Every method execution produces versioned data artifacts, and any other model can reference those artifacts through CEL expressions like data.latest("model-name", "data-name").attributes.field. The data layer -- also known as The Swamp -- is the integration surface. See How Swamp Works for the broader composition model.

Models can also invoke each other directly via context.runModel(), but the data layer remains the default and preferred approach. Direct invocation exists for cases where a model needs to orchestrate another model's execution mid-method and act on the result immediately — the kind of work that data chaining cannot express because the caller needs to make decisions based on the outcome before proceeding. See Models, Types, and Methods for when direct invocation is appropriate and the safety limits that bound it.

Why indirect coupling is the default

Direct model-to-model calls create ordering dependencies: A must know B exists, B must be running, and the two must agree on an interface at call time. Indirect coupling through data removes all three constraints. A model that produces a VPC ID does not know whether zero or ten other models will consume it. A model that reads a VPC ID does not know which model produced it or when. The only shared contract is the data name and its schema, making the system additive: wiring a new model into an existing pipeline requires no changes to the models already there.

context.runModel() reintroduces those ordering dependencies deliberately, for cases where they are worth the cost. A lifecycle model that needs to auto-create a sub-model, run its method, and branch on the result cannot express that through data chaining alone — the decision depends on the outcome, and the outcome does not exist until the sub-model runs. The tradeoff is explicit: direct invocation couples the caller to the callee's existence and interface, but it enables synchronous orchestration that data-layer composition cannot.

Versioning and immutability

Every method execution produces a new version of its data outputs. Previous versions are retained up to garbage collection limits. Data is never updated in place.

Immutability buys three things. Auditability: you can always reconstruct what a model saw at any point in time. Rollback: reverting to a previous version is a read, not a write. Safe concurrency: two models writing to the same data name produce two versions rather than a race condition. data.latest() returns the most recent version; data.version() retrieves a specific one. The CEL expressions reference covers the full retrieval syntax.

Data lifetime and garbage collection

Not all data needs to live forever. Intermediate computation results lose value the moment the workflow completes; long-lived state needs to persist across runs. Lifetime policies -- ephemeral, job, workflow, duration, infinite -- express this distinction. swamp data gc enforces both lifetime expiry and version retention limits. Shorter lifetimes reduce storage and noise in queries; longer lifetimes preserve history and enable rollback. The data reference documents the available policies.

Ephemeral data occupies one end of this spectrum: it lives entirely in memory and is never persisted to disk. When the process exits, ephemeral data disappears. This makes it useful for intermediate computation results that only need to survive long enough for downstream steps to consume them -- a workflow can pass large transient datasets between steps without accumulating storage debt. The tradeoff is durability: if the process crashes, ephemeral data is lost. A configurable memory budget (default 512 MB, controlled by SWAMP_EPHEMERAL_BUDGET) prevents runaway memory consumption. See Data Lifetimes for when to choose each lifetime policy, and the data reference and repository configuration for the specifics.

Orphaned data

Garbage collection handles data whose lifetime has elapsed or whose version count exceeds the declared cap. A different category of stale data falls outside its reach: data whose owning model definition no longer exists. This happens when a definition is deleted outright, or when a model migrates to a different namespace and its historical data stays behind under the old namespace path.

Reclaiming orphaned data is deliberately a separate command (swamp data prune) rather than a phase of data gc. The reason is confidence: gc enforces policies the model author explicitly declared -- lifetimes and version caps are first-class configuration, and enforcing them is always safe. Prune, by contrast, infers that data is orphaned from the absence of a definition, and absence is not always permanent. A branch switch can make a definition disappear and reappear. An in-flight migration might remove the old definition before the new one is committed. Conflating the two would mean that a routine, automatable operation (gc) could irreversibly destroy data that is only transiently unreachable.

The separation lets each command carry the appropriate safety posture: gc runs unattended (including as an automatic post-method hook), while prune defaults to a confirmation prompt and supports --dry-run for preview. See the data reference for the command details.

Querying across models

data.latest() retrieves a known artifact by name. But some operations need to discover data rather than reference it -- find all resources tagged with environment=production, or locate every failed result across a repository. data.query() in CEL and swamp data query on the CLI search across all stored data using CEL predicates. The query layer exists because composition is not always planned in advance: reporting, debugging, and ad-hoc inspection all require asking questions that no single model anticipated. See the data reference for query syntax and filtering.

Tags and provenance

Every data artifact carries provenance metadata: which model produced it, which method, which workflow run (if any), who initiated the run, and timestamps. The initiatedBy tag records the authenticated principal (e.g. user:stack72) or ghost for unauthenticated runs, making it possible to trace an artifact back to the person who triggered the execution. Operatives can also attach user-defined tags. Provenance is automatic because manual tracking does not scale -- when an artifact looks wrong, the audit trail connects it to the exact execution that created it without anyone having planned for that investigation.

Sensitive fields and redaction

Some data that passes through models is sensitive -- an API key returned by a cloud provider, a connection string with embedded credentials. Output specs can mark fields as sensitive. Sensitive fields are stored but redacted from CLI output and logs.

Excluding sensitive fields entirely would break downstream models that need those values. Storing them only in the vault would require models to coordinate on vault paths rather than data names, re-introducing the coupling the data layer is designed to avoid. Redaction preserves the uniform interface -- downstream models read sensitive fields through data.latest() like any other field -- while keeping secrets out of terminal sessions and audit logs.

Redacting what is displayed is not enough on its own, because a value read through an expression travels further than the terminal. Spliced into a shell command, it would sit in the process list, readable by other processes on the host. Written into an evaluated cache or a run record, it would sit on disk for as long as the file does. So the uniform interface stops at the expression: inside it, a sensitive field read through data.latest(), steps.<name>.outputs or model.<name>.resource is the real value, and a guard or comparison behaves as it would on any other field. Once the result leaves the expression, it never reaches a spawned command line or a file Swamp writes in plaintext. A shell step receives the value through an environment variable, the same way it receives a vault.get() secret. The exception is a place no variable can expand, such as a quoted here-document, where the value has to stay in the command line and Swamp warns.

Every sensitive field already has a vault reference, since that is how it was stored, so persisting the reference needs no new storage and keeps a later run or a --last-evaluated replay able to resolve the value again. Caches, run records and auto-created definitions store that reference. Reports and evaluate output, which exist to be read, show ***. Tags, labels, forEach step names and data-name suffixes are compared as exact strings and never resolved, so they cannot carry a reference; they get a sensitive-<vault>.<key> placeholder instead. The cost is that a tag filter on a secret value does not match.

Caches written by an older Swamp may hold values in plaintext, and nothing in the file says which. Rather than replay one and splice a plaintext value back into a command, --last-evaluated refuses a cache that lacks the format marker and whose expressions read data or step outputs; one normal run rewrites it. The vaults reference lists where each value lands, and Environment Variable Mounting covers how a shell step receives it.