THE DATA LAYER
The primary composition mechanism between behaviors in Swamp is data, not calls.
Every method execution produces versioned data artifacts, and any other behavior
can reference those artifacts through CEL expressions like
data.latest("behavior-name", "data-name").attributes.field. The data layer --
also known as The Swamp -- is the integration surface. See
How Swamp Works for the broader
composition model.
Behaviors can also invoke each other directly via context.runModel(), but the
data layer remains the default and preferred approach. Direct invocation exists
for cases where a behavior needs to orchestrate another behavior's execution
mid-method and act on the result immediately — the kind of work that data
chaining cannot express because the caller needs to make decisions based on the
outcome before proceeding. See
Behaviors, Types, and Methods
for when direct invocation is appropriate and the safety limits that bound it.
Why indirect coupling is the default
Direct behavior-to-behavior calls create ordering dependencies: A must know B exists, B must be running, and the two must agree on an interface at call time. Indirect coupling through data removes all three constraints. A behavior that produces a VPC ID does not know whether zero or ten other behaviors will consume it. A behavior that reads a VPC ID does not know which behavior produced it or when. The only shared contract is the data name and its schema, making the system additive: wiring a new behavior into an existing pipeline requires no changes to the behaviors already there.
context.runModel() reintroduces those ordering dependencies deliberately, for
cases where they are worth the cost. A lifecycle behavior that needs to
auto-create a sub-behavior, run its method, and branch on the result cannot
express that through data chaining alone — the decision depends on the outcome,
and the outcome does not exist until the sub-behavior runs. The tradeoff is
explicit: direct invocation couples the caller to the callee's existence and
interface, but it enables synchronous orchestration that data-layer composition
cannot.
Versioning and immutability
Every method execution produces a new version of its data outputs. Previous versions are retained up to garbage collection limits. Data is never updated in place.
Immutability buys three things. Auditability: you can always reconstruct what a
behavior saw at any point in time. Rollback: reverting to a previous version is
a read, not a write. Safe concurrency: two behaviors writing to the same data
name produce two versions rather than a race condition. data.latest() returns
the most recent version; data.version() retrieves a specific one. The
CEL expressions reference covers the full
retrieval syntax.
Durability across a power cut
A version is useful downstream only if its content is really there when
data.latest() names it. By default, Swamp writes each version atomically, so a
crashed process never leaves a half-written version behind, but it does not sync
the write to disk. After a power cut, a filesystem such as ext4 can keep the
rename and the latest marker yet lose the bytes, and the next reader finds an
empty version published as latest.
The datastore's durable setting closes that gap: each version's content
reaches the disk before anything publishes it. The cost is several fsyncs per
write, about double the write time on a local NVMe disk and more on network or
spinning disks. Turn it on where data must survive a power cut, such as the
recorded state of real infrastructure a later run reads back; leave it off where
a lost newest version can simply be produced again. A v3 datastore's committed
versions are already durable. See
durable for exactly what
it covers, including how macOS orders writes on the default datastore.
Data lifetime and garbage collection
Not all data needs to live forever. Intermediate computation results lose value
the moment the workflow completes; long-lived state needs to persist across
runs. Lifetime policies -- ephemeral, job, workflow, duration, infinite --
express this distinction. swamp data gc enforces both lifetime expiry and
version retention limits. Shorter lifetimes reduce storage and noise in queries;
longer lifetimes preserve history and enable rollback. The
data reference documents the available policies.
Ephemeral data occupies one end of this spectrum: it lives entirely in memory
and is never persisted to disk. When the process exits, ephemeral data
disappears. This makes it useful for intermediate computation results that only
need to survive long enough for downstream steps to consume them -- a workflow
can pass large transient datasets between steps without accumulating storage
debt. The tradeoff is durability: if the process crashes, ephemeral data is
lost. A configurable memory budget (default 512 MB, controlled by
SWAMP_EPHEMERAL_BUDGET) prevents runaway memory consumption. See
Data Lifetimes for when to choose each
lifetime policy, and the
data reference and
repository configuration
for the specifics.
Orphaned data
Garbage collection handles data whose lifetime has elapsed or whose version count exceeds the declared cap. A different category of stale data falls outside its reach: data whose owning behavior definition no longer exists. This happens when a definition is deleted outright, or when a behavior migrates to a different namespace and its historical data stays behind under the old namespace path.
Reclaiming orphaned data is deliberately a separate command (swamp data prune)
rather than a phase of data gc. The reason is confidence: gc enforces policies
the behavior author explicitly declared -- lifetimes and version caps are
first-class configuration, and enforcing them is always safe. Prune, by
contrast, infers that data is orphaned from the absence of a definition, and
absence is not always permanent. A branch switch can make a definition disappear
and reappear. An in-flight migration might remove the old definition before the
new one is committed. Conflating the two would mean that a routine, automatable
operation (gc) could irreversibly destroy data that is only transiently
unreachable.
The separation lets each command carry the appropriate safety posture: gc runs
unattended (including as an automatic post-method hook), while prune defaults to
a confirmation prompt and supports --dry-run for preview. See the
data reference for the command
details.
Querying across behaviors
data.latest() retrieves a known artifact by name. But some operations need to
discover data rather than reference it -- find all resources tagged with
environment=production, or locate every failed result across a repository.
data.query() in CEL and swamp data query on the CLI search across all stored
data using CEL predicates. The query layer exists because composition is not
always planned in advance: reporting, debugging, and ad-hoc inspection all
require asking questions that no single behavior anticipated. See the
data reference for query syntax and filtering.
Tags and provenance
Every data artifact carries provenance metadata: which behavior produced it,
which method, which workflow run (if any), who initiated the run, and
timestamps. The initiatedBy tag records the authenticated principal (e.g.
user:stack72) or ghost for unauthenticated runs, making it possible to trace
an artifact back to the person who triggered the execution. Operatives can also
attach user-defined tags. Provenance is automatic because manual tracking does
not scale -- when an artifact looks wrong, the audit trail connects it to the
exact execution that created it without anyone having planned for that
investigation.
Sensitive fields and redaction
Some data that passes through behaviors is sensitive -- an API key returned by a cloud provider, a connection string with embedded credentials. Output specs can mark fields as sensitive. Sensitive fields are stored but redacted from CLI output and logs.
Excluding sensitive fields entirely would break downstream behaviors that need
those values. Storing them only in the vault would
require behaviors to coordinate on vault paths rather than data names,
re-introducing the coupling the data layer is designed to avoid. Redaction
preserves the uniform interface -- downstream behaviors read sensitive fields
through data.latest() like any other field -- while keeping secrets out of
terminal sessions and audit logs.
Redacting what is displayed is not enough on its own, because a value read
through an expression travels further than the terminal. Spliced into a shell
command, it would sit in the process list, readable by other processes on the
host. Written into an evaluated cache or a run record, it would sit on disk for
as long as the file does. So the uniform interface stops at the expression:
inside it, a sensitive field read through data.latest(),
steps.<name>.outputs or model.<name>.resource is the real value, and a guard
or comparison behaves as it would on any other field. Once the result leaves the
expression, it never reaches a spawned command line or a file Swamp writes in
plaintext. A shell step receives the value through an environment variable, the
same way it receives a vault.get() secret. The exception is a place no
variable can expand, such as a quoted here-document, where the value has to stay
in the command line and Swamp warns.
Every sensitive field already has a vault reference, since that is how it was
stored, so persisting the reference needs no new storage and keeps a later run
or a --last-evaluated replay able to resolve the value again. Caches, run
records and auto-created definitions store that reference. Reports and evaluate
output, which exist to be read, show ***. Tags, labels, forEach step names
and data-name suffixes are compared as exact strings and never resolved, so they
cannot carry a reference; they get a sensitive-<vault>.<key> placeholder
instead. The cost is that a tag filter on a secret value does not match.
Caches written by an older Swamp may hold values in plaintext, and nothing in
the file says which. Rather than replay one and splice a plaintext value back
into a command, --last-evaluated refuses a cache that lacks the format marker
and whose expressions read data or step outputs; one normal run rewrites it. The
vaults reference
lists where each value lands, and
Environment Variable Mounting
covers how a shell step receives it.