THE DATA LAYER

The primary composition mechanism between models in Swamp is data, not calls. Every method execution produces versioned data artifacts, and any other model can reference those artifacts through CEL expressions like data.latest("model-name", "data-name").attributes.field. The data layer -- also known as The Swamp -- is the integration surface. See How Swamp Works for the broader composition model.

Models can also invoke each other directly via context.runModel(), but the data layer remains the default and preferred approach. Direct invocation exists for cases where a model needs to orchestrate another model's execution mid-method and act on the result immediately — the kind of work that data chaining cannot express because the caller needs to make decisions based on the outcome before proceeding. See Models, Types, and Methods for when direct invocation is appropriate and the safety limits that bound it.

Why indirect coupling is the default

Direct model-to-model calls create ordering dependencies: A must know B exists, B must be running, and the two must agree on an interface at call time. Indirect coupling through data removes all three constraints. A model that produces a VPC ID does not know whether zero or ten other models will consume it. A model that reads a VPC ID does not know which model produced it or when. The only shared contract is the data name and its schema, making the system additive: wiring a new model into an existing pipeline requires no changes to the models already there.

context.runModel() reintroduces those ordering dependencies deliberately, for cases where they are worth the cost. A lifecycle model that needs to auto-create a sub-model, run its method, and branch on the result cannot express that through data chaining alone — the decision depends on the outcome, and the outcome does not exist until the sub-model runs. The tradeoff is explicit: direct invocation couples the caller to the callee's existence and interface, but it enables synchronous orchestration that data-layer composition cannot.

Versioning and immutability

Every method execution produces a new version of its data outputs. Previous versions are retained up to garbage collection limits. Data is never updated in place.

Immutability buys three things. Auditability: you can always reconstruct what a model saw at any point in time. Rollback: reverting to a previous version is a read, not a write. Safe concurrency: two models writing to the same data name produce two versions rather than a race condition. data.latest() returns the most recent version; data.version() retrieves a specific one. The CEL expressions reference covers the full retrieval syntax.

Data lifetime and garbage collection

Not all data needs to live forever. Intermediate computation results lose value the moment the workflow completes; long-lived state needs to persist across runs. Lifetime policies -- ephemeral, job, workflow, duration, infinite -- express this distinction. swamp data gc enforces both lifetime expiry and version retention limits. Shorter lifetimes reduce storage and noise in queries; longer lifetimes preserve history and enable rollback. The data reference documents the available policies.

Ephemeral data occupies one end of this spectrum: it lives entirely in memory and is never persisted to disk. When the process exits, ephemeral data disappears. This makes it useful for intermediate computation results that only need to survive long enough for downstream steps to consume them -- a workflow can pass large transient datasets between steps without accumulating storage debt. The tradeoff is durability: if the process crashes, ephemeral data is lost. A configurable memory budget (default 512 MB, controlled by SWAMP_EPHEMERAL_BUDGET) prevents runaway memory consumption. See Data Lifetimes for when to choose each lifetime policy, and the data reference and repository configuration for the specifics.

Orphaned data

Garbage collection handles data whose lifetime has elapsed or whose version count exceeds the declared cap. A different category of stale data falls outside its reach: data whose owning model definition no longer exists. This happens when a definition is deleted outright, or when a model migrates to a different namespace and its historical data stays behind under the old namespace path.

Reclaiming orphaned data is deliberately a separate command (swamp data prune) rather than a phase of data gc. The reason is confidence: gc enforces policies the model author explicitly declared -- lifetimes and version caps are first-class configuration, and enforcing them is always safe. Prune, by contrast, infers that data is orphaned from the absence of a definition, and absence is not always permanent. A branch switch can make a definition disappear and reappear. An in-flight migration might remove the old definition before the new one is committed. Conflating the two would mean that a routine, automatable operation (gc) could irreversibly destroy data that is only transiently unreachable.

The separation lets each command carry the appropriate safety posture: gc runs unattended (including as an automatic post-method hook), while prune defaults to a confirmation prompt and supports --dry-run for preview. See the data reference for the command details.

Querying across models

data.latest() retrieves a known artifact by name. But some operations need to discover data rather than reference it -- find all resources tagged with environment=production, or locate every failed result across a repository. data.query() in CEL and swamp data query on the CLI search across all stored data using CEL predicates. The query layer exists because composition is not always planned in advance: reporting, debugging, and ad-hoc inspection all require asking questions that no single model anticipated. See the data reference for query syntax and filtering.

Tags and provenance

Every data artifact carries provenance metadata: which model produced it, which method, which workflow run (if any), and timestamps. Operatives can also attach user-defined tags. Provenance is automatic because manual tracking does not scale -- when an artifact looks wrong, the audit trail connects it to the exact execution that created it without anyone having planned for that investigation.

Sensitive fields and redaction

Some data that passes through models is sensitive -- an API key returned by a cloud provider, a connection string with embedded credentials. Output specs can mark fields as sensitive. Sensitive fields are stored but redacted from CLI output and logs.

Excluding sensitive fields entirely would break downstream models that need those values. Storing them only in the vault would require models to coordinate on vault paths rather than data names, re-introducing the coupling the data layer is designed to avoid. Redaction preserves the uniform interface -- downstream models read sensitive fields through data.latest() like any other field -- while keeping secrets out of terminal sessions and audit logs.