DATASTORE ARCHITECTURE
Every behavior method execution, workflow run, audit event, cached bundle, and encrypted secret passes through one datastore. A single persistence layer makes backup, migration, and sharing atomic operations rather than ad-hoc scripts that chase files across scattered directories.
Source-of-truth definitions -- behavior files, workflow files, vault configs --
stay in git-tracked directories (models/, workflows/, vaults/) so that
version control and code review apply to them. The datastore holds only runtime
artifacts: the outputs those definitions produce when executed. See
How Swamp Works for the broader execution
model.
Why three backend options
The default backend stores data in .swamp/ inside the repository -- no
configuration, no network, no external dependencies. This is the right choice
for a single operative working on one machine.
An external filesystem path (such as an NFS mount) enables multiple machines to share state without cloud dependencies. The tradeoff is operational: shared filesystems introduce network latency and require mount management.
An S3-compatible backend enables geo-distributed access with object versioning.
It is itself a Swamp extension (@swamp/s3-datastore), not a built-in -- the
datastore system is extensible by design. Extension-provided backends require
Swamp authentication because they cross the boundary from single-machine state
into shared infrastructure -- the local filesystem backend affects only one
machine, but an S3 bucket affects everyone who shares it, so authentication
provides the accountability boundary. See the
datastore configuration reference
for backend setup and the
API key scoping explanation
for the design rationale.
The catalog database
On top of whatever backend stores the raw files, the datastore maintains a
SQLite catalog that indexes all artifacts. The catalog exists because scanning a
filesystem (or making S3 LIST calls) for every data.query() or data.latest()
would be prohibitively slow. Indexed lookup through SQLite turns those
operations into millisecond queries regardless of how many artifacts exist.
The catalog is a cache, not a source of truth. If deleted or corrupted, the datastore rebuilds it from the underlying storage. This self-healing property means the catalog is never synced to remote backends -- each machine builds its own from whatever data it has locally.
Why lazy hydration is the default
When using an S3 backend, the datastore must decide what to download. Eager hydration (downloading everything on startup) guarantees all data is available locally, but most commands touch a small fraction of the total data. Downloading gigabytes of historical behavior outputs to run one method wastes bandwidth and startup time.
Lazy hydration downloads metadata for catalog visibility but defers content
retrieval until something actually reads the artifact. The tradeoff is explicit:
queries filtering on attributes content may see null for un-hydrated data.
This is a documented limitation -- the alternative (eager sync) trades
correctness for performance in the opposite direction.
Per-behavior locking
When multiple machines share a datastore, concurrent writes to the same behavior must be serialized. The locking scope is per-behavior rather than global because a workflow running a deploy behavior should not block a separate scan behavior from executing concurrently on another machine. Per-behavior locks keep contention proportional to actual conflicts.
The mechanism is a file lock with a TTL of approximately 30 seconds. The executing process writes a lock file and renews it via a heartbeat for the duration of the method run. If the process crashes or is killed without releasing the lock, stale-lock detection compares the lock's PID against the process table. Once the TTL expires and the owning PID is confirmed dead, the lock is reclaimed automatically by the next run that needs it.
When a run cannot acquire the lock, the CLI exits with code 75 (lock_timeout)
so the caller can retry with backoff. For stuck locks that stale-detection
cannot clear -- the owning process is on another host, or the PID has been
recycled -- the breakglass commands swamp datastore lock status and
swamp datastore lock release --force provide manual intervention. See the
operational commands reference
for usage.
Global locks exist for structural operations -- garbage collection, behavior deletion, datastore migration -- where cross-behavior consistency matters. These are rare and short-lived by design. For why fan-out methods reduce lock contention within a single behavior, see How Swamp Works.
Edit conflicts under managed config
With managed config,
behavior definitions, workflows, vault configs and the extension lockfile are
datastore records that every checkout shares. Locks serialize writes, but they
do not stop two members from editing one definition: each command holds the lock
only while it runs, and the conflict lies in what each member started from. If
two members load deploy, A saves an edit, and B then saves an edit of the copy
B loaded, B's save replaces A's change without B ever seeing it.
Checking the file at save time cannot catch this. Every command first catches the checkout up with the datastore, which writes A's version over B's local file, so by the time B's edit saves, B's file already matches the datastore.
A v3 datastore checks against what the checkout last loaded instead. When a
catch-up overwrites or removes a managed-config record with another writer's
version, the checkout remembers the version it held before. That version stays
the starting point of the next edit until the member loads the definition again
(get, or opening it in the editor) or commits an edit of it. An edit or delete
then expects the datastore still to hold that version. If it does not, the
command is refused with config_edit_conflict, naming who changed the
definition and when, and asks the member to load the current version and
re-apply the change. The check runs before the command changes anything, so a
refused delete leaves a behavior's outputs and data and a workflow's runs in
place. The refusal also lasts: running the same command again is refused until
the member loads the definition, because the remembered version has not moved.
The extension lockfile works differently. A lockfile change merges entries into the current lockfile, so an edit there does not need to start from a particular version. Only a change another checkout lands while an extension command runs is refused; the refused change is kept and replayed by the next extension command.
An edit sent with --server runs in serve's own handler, and serve's copy is
always the latest one, so serve can only refuse a change that lands while it
applies the edit.
A v2 datastore has no such check: under managed config the last writer wins. For
the exact error and its --json form, see
Edit conflicts on a v3 datastore.
Namespace partitioning
Namespaces partition a shared datastore by repository. Each namespace gets its own data, definitions, and workflow runs while sharing the underlying storage backend. This is how Giga-Swamp repositories enable multiple teams to work against a single datastore without data collisions. See Giga-Swamp for the full namespace design and the isolation guarantees it provides.
Secrets follow a different path entirely -- local_encryption vault secrets
(.swamp/secrets/) are excluded from the datastore's managed directory set and
are never synced, migrated, or relocated, regardless of the backend. They stay
local to the machine, which is what enables the shared-workflow-per-user-
credentials pattern: the vault definition is git-tracked, but each operative's
secret values remain on their own machine. See the
vaults reference
for the full exclusion rule and when to migrate to a cloud backend instead.
Namespace contamination
When multiple repositories share a remote datastore (S3 or GCS), each
repository's namespace scopes its data into a distinct prefix. A push from
repository infra writes objects under infra/, a push from platform writes
under platform/, and so on. The namespace prefix is the boundary that keeps
each team's data in its own subtree.
Before the namespace isolation fix, pushes did not always enforce this scoping.
An unscoped push could write objects from one namespace into another namespace's
prefix. The result is contamination: the infra prefix contains objects that
actually belong to platform (or vice versa). The foreign objects are not
dangerous -- they cannot corrupt the owning namespace's data -- but they create
two problems.
First, they inflate the namespace's object count and storage usage. Garbage collection treats them as local data and will not remove them, because it has no way to know they are foreign.
Second, and more disruptive, they can shadow legitimate objects in the catalog
index. When the catalog rebuilds from the remote prefix, it ingests the foreign
objects alongside the real ones. This can make workflow runs invisible -- the
catalog lists a foreign object where the real run's output should be. Suspended
approval runs that depend on those outputs become unreachable via
workflow approvals, because the catalog points to the wrong data.
The swamp doctor datastores subcommand detects contamination by scanning the
namespace prefix for objects whose namespace metadata does not match. The
--repair flag runs a five-step cleanup: delete foreign objects from the
prefix, rebuild the remote index so it reflects only scoped data, wipe the local
catalog cache (which may have indexed foreign objects), re-pull scoped data to
rebuild a clean catalog, and invalidate workflow run and catalog indexes so
previously shadowed runs become visible again. The preview-then-execute pattern
(--repair to preview, --repair -y to execute) lets you review the scope of
the cleanup before anything is modified.
Contamination only affects custom (remote) datastores with a namespace
configured. The default filesystem backend stores data locally in .swamp/ with
no namespace prefix, so there is no path for foreign objects to appear.
See Troubleshoot Your Swamp Repo for the step-by-step repair procedure and Doctor for the full flag and output reference.