Skip to main content

THE DATA LAYER

The primary composition mechanism between behaviors in Swamp is data, not calls. Every method execution produces versioned data artifacts, and any other behavior can reference those artifacts through CEL expressions like data.latest("behavior-name", "data-name").attributes.field. The data layer -- also known as The Swamp -- is the integration surface. See How Swamp Works for the broader composition model.

Behaviors can also invoke each other directly via context.runModel(), but the data layer remains the default and preferred approach. Direct invocation exists for cases where a behavior needs to orchestrate another behavior's execution mid-method and act on the result immediately — the kind of work that data chaining cannot express because the caller needs to make decisions based on the outcome before proceeding. See Behaviors, Types, and Methods for when direct invocation is appropriate and the safety limits that bound it.

Why indirect coupling is the default

Direct behavior-to-behavior calls create ordering dependencies: A must know B exists, B must be running, and the two must agree on an interface at call time. Indirect coupling through data removes all three constraints. A behavior that produces a VPC ID does not know whether zero or ten other behaviors will consume it. A behavior that reads a VPC ID does not know which behavior produced it or when. The only shared contract is the data name and its schema, making the system additive: wiring a new behavior into an existing pipeline requires no changes to the behaviors already there.

context.runModel() reintroduces those ordering dependencies deliberately, for cases where they are worth the cost. A lifecycle behavior that needs to auto-create a sub-behavior, run its method, and branch on the result cannot express that through data chaining alone — the decision depends on the outcome, and the outcome does not exist until the sub-behavior runs. The tradeoff is explicit: direct invocation couples the caller to the callee's existence and interface, but it enables synchronous orchestration that data-layer composition cannot.

Versioning and immutability

Every method execution produces a new version of its data outputs. Previous versions are retained up to garbage collection limits. Data is never updated in place.

Immutability buys three things. Auditability: you can always reconstruct what a behavior saw at any point in time. Rollback: reverting to a previous version is a read, not a write. Safe concurrency: two behaviors writing to the same data name produce two versions rather than a race condition. data.latest() returns the most recent version; data.version() retrieves a specific one. The CEL expressions reference covers the full retrieval syntax.

Durability across a power cut

A version is useful downstream only if its content is really there when data.latest() names it. By default, Swamp writes each version atomically, so a crashed process never leaves a half-written version behind, but it does not sync the write to disk. After a power cut, a filesystem such as ext4 can keep the rename and the latest marker yet lose the bytes, and the next reader finds an empty version published as latest.

The datastore's durable setting closes that gap: each version's content reaches the disk before anything publishes it. The cost is several fsyncs per write, about double the write time on a local NVMe disk and more on network or spinning disks. Turn it on where data must survive a power cut, such as the recorded state of real infrastructure a later run reads back; leave it off where a lost newest version can simply be produced again. A v3 datastore's committed versions are already durable. See durable for exactly what it covers, including how macOS orders writes on the default datastore.

Data lifetime and garbage collection

Not all data needs to live forever. Intermediate computation results lose value the moment the workflow completes; long-lived state needs to persist across runs. Lifetime policies -- ephemeral, job, workflow, duration, infinite -- express this distinction. swamp data gc enforces both lifetime expiry and version retention limits. Shorter lifetimes reduce storage and noise in queries; longer lifetimes preserve history and enable rollback. The data reference documents the available policies.

Ephemeral data occupies one end of this spectrum: it lives entirely in memory and is never persisted to disk. When the process exits, ephemeral data disappears. This makes it useful for intermediate computation results that only need to survive long enough for downstream steps to consume them -- a workflow can pass large transient datasets between steps without accumulating storage debt. The tradeoff is durability: if the process crashes, ephemeral data is lost. A configurable memory budget (default 512 MB, controlled by SWAMP_EPHEMERAL_BUDGET) prevents runaway memory consumption. See Data Lifetimes for when to choose each lifetime policy, and the data reference and repository configuration for the specifics.

Orphaned data

Garbage collection handles data whose lifetime has elapsed or whose version count exceeds the declared cap. A different category of stale data falls outside its reach: data whose owning behavior definition no longer exists. This happens when a definition is deleted outright, or when a behavior migrates to a different namespace and its historical data stays behind under the old namespace path.

Reclaiming orphaned data is deliberately a separate command (swamp data prune) rather than a phase of data gc. The reason is confidence: gc enforces policies the behavior author explicitly declared -- lifetimes and version caps are first-class configuration, and enforcing them is always safe. Prune, by contrast, infers that data is orphaned from the absence of a definition, and absence is not always permanent. A branch switch can make a definition disappear and reappear. An in-flight migration might remove the old definition before the new one is committed. Conflating the two would mean that a routine, automatable operation (gc) could irreversibly destroy data that is only transiently unreachable.

The separation lets each command carry the appropriate safety posture: gc runs unattended (including as an automatic post-method hook), while prune defaults to a confirmation prompt and supports --dry-run for preview. See the data reference for the command details.

Querying across behaviors

data.latest() retrieves a known artifact by name. But some operations need to discover data rather than reference it -- find all resources tagged with environment=production, or locate every failed result across a repository. data.query() in CEL and swamp data query on the CLI search across all stored data using CEL predicates. The query layer exists because composition is not always planned in advance: reporting, debugging, and ad-hoc inspection all require asking questions that no single behavior anticipated. See the data reference for query syntax and filtering.

Tags and provenance

Every data artifact carries provenance metadata: which behavior produced it, which method, which workflow run (if any), who initiated the run, and timestamps. The initiatedBy tag records the authenticated principal (e.g. user:stack72) or ghost for unauthenticated runs, making it possible to trace an artifact back to the person who triggered the execution. Operatives can also attach user-defined tags. Provenance is automatic because manual tracking does not scale -- when an artifact looks wrong, the audit trail connects it to the exact execution that created it without anyone having planned for that investigation.

Sensitive fields and redaction

Some data that passes through behaviors is sensitive -- an API key returned by a cloud provider, a connection string with embedded credentials. Output specs can mark fields as sensitive. Sensitive fields are stored but redacted from CLI output and logs.

Excluding sensitive fields entirely would break downstream behaviors that need those values. Storing them only in the vault would require behaviors to coordinate on vault paths rather than data names, re-introducing the coupling the data layer is designed to avoid. Redaction preserves the uniform interface -- downstream behaviors read sensitive fields through data.latest() like any other field -- while keeping secrets out of terminal sessions and audit logs.

Redacting what is displayed is not enough on its own, because a value read through an expression travels further than the terminal. Spliced into a shell command, it would sit in the process list, readable by other processes on the host. Written into an evaluated cache or a run record, it would sit on disk for as long as the file does. So the uniform interface stops at the expression: inside it, a sensitive field read through data.latest(), steps.<name>.outputs or model.<name>.resource is the real value, and a guard or comparison behaves as it would on any other field. Once the result leaves the expression, it never reaches a spawned command line or a file Swamp writes in plaintext. A shell step receives the value through an environment variable, the same way it receives a vault.get() secret. The exception is a place no variable can expand, such as a quoted here-document, where the value has to stay in the command line and Swamp warns.

Every sensitive field already has a vault reference, since that is how it was stored, so persisting the reference needs no new storage and keeps a later run or a --last-evaluated replay able to resolve the value again. Caches, run records and auto-created definitions store that reference. Reports and evaluate output, which exist to be read, show ***. Tags, labels, forEach step names and data-name suffixes are compared as exact strings and never resolved, so they cannot carry a reference; they get a sensitive-<vault>.<key> placeholder instead. The cost is that a tag filter on a secret value does not match.

Caches written by an older Swamp may hold values in plaintext, and nothing in the file says which. Rather than replay one and splice a plaintext value back into a command, --last-evaluated refuses a cache that lacks the format marker and whose expressions read data or step outputs; one normal run rewrites it. The vaults reference lists where each value lands, and Environment Variable Mounting covers how a shell step receives it.