Skip to main content

DATASTORE ARCHITECTURE

Every behavior method execution, workflow run, audit event, cached bundle, and encrypted secret passes through one datastore. A single persistence layer makes backup, migration, and sharing atomic operations rather than ad-hoc scripts that chase files across scattered directories.

Source-of-truth definitions -- behavior files, workflow files, vault configs -- stay in git-tracked directories (models/, workflows/, vaults/) so that version control and code review apply to them. The datastore holds only runtime artifacts: the outputs those definitions produce when executed. See How Swamp Works for the broader execution model.

Why three backend options

The default backend stores data in .swamp/ inside the repository -- no configuration, no network, no external dependencies. This is the right choice for a single operative working on one machine.

An external filesystem path (such as an NFS mount) enables multiple machines to share state without cloud dependencies. The tradeoff is operational: shared filesystems introduce network latency and require mount management.

An S3-compatible backend enables geo-distributed access with object versioning. It is itself a Swamp extension (@swamp/s3-datastore), not a built-in -- the datastore system is extensible by design. Extension-provided backends require Swamp authentication because they cross the boundary from single-machine state into shared infrastructure -- the local filesystem backend affects only one machine, but an S3 bucket affects everyone who shares it, so authentication provides the accountability boundary. See the datastore configuration reference for backend setup and the API key scoping explanation for the design rationale.

The catalog database

On top of whatever backend stores the raw files, the datastore maintains a SQLite catalog that indexes all artifacts. The catalog exists because scanning a filesystem (or making S3 LIST calls) for every data.query() or data.latest() would be prohibitively slow. Indexed lookup through SQLite turns those operations into millisecond queries regardless of how many artifacts exist.

The catalog is a cache, not a source of truth. If deleted or corrupted, the datastore rebuilds it from the underlying storage. This self-healing property means the catalog is never synced to remote backends -- each machine builds its own from whatever data it has locally.

Why lazy hydration is the default

When using an S3 backend, the datastore must decide what to download. Eager hydration (downloading everything on startup) guarantees all data is available locally, but most commands touch a small fraction of the total data. Downloading gigabytes of historical behavior outputs to run one method wastes bandwidth and startup time.

Lazy hydration downloads metadata for catalog visibility but defers content retrieval until something actually reads the artifact. The tradeoff is explicit: queries filtering on attributes content may see null for un-hydrated data. This is a documented limitation -- the alternative (eager sync) trades correctness for performance in the opposite direction.

Per-behavior locking

When multiple machines share a datastore, concurrent writes to the same behavior must be serialized. The locking scope is per-behavior rather than global because a workflow running a deploy behavior should not block a separate scan behavior from executing concurrently on another machine. Per-behavior locks keep contention proportional to actual conflicts.

The mechanism is a file lock with a TTL of approximately 30 seconds. The executing process writes a lock file and renews it via a heartbeat for the duration of the method run. If the process crashes or is killed without releasing the lock, stale-lock detection compares the lock's PID against the process table. Once the TTL expires and the owning PID is confirmed dead, the lock is reclaimed automatically by the next run that needs it.

When a run cannot acquire the lock, the CLI exits with code 75 (lock_timeout) so the caller can retry with backoff. For stuck locks that stale-detection cannot clear -- the owning process is on another host, or the PID has been recycled -- the breakglass commands swamp datastore lock status and swamp datastore lock release --force provide manual intervention. See the operational commands reference for usage.

Global locks exist for structural operations -- garbage collection, behavior deletion, datastore migration -- where cross-behavior consistency matters. These are rare and short-lived by design. For why fan-out methods reduce lock contention within a single behavior, see How Swamp Works.

Edit conflicts under managed config

With managed config, behavior definitions, workflows, vault configs and the extension lockfile are datastore records that every checkout shares. Locks serialize writes, but they do not stop two members from editing one definition: each command holds the lock only while it runs, and the conflict lies in what each member started from. If two members load deploy, A saves an edit, and B then saves an edit of the copy B loaded, B's save replaces A's change without B ever seeing it.

Checking the file at save time cannot catch this. Every command first catches the checkout up with the datastore, which writes A's version over B's local file, so by the time B's edit saves, B's file already matches the datastore.

A v3 datastore checks against what the checkout last loaded instead. When a catch-up overwrites or removes a managed-config record with another writer's version, the checkout remembers the version it held before. That version stays the starting point of the next edit until the member loads the definition again (get, or opening it in the editor) or commits an edit of it. An edit or delete then expects the datastore still to hold that version. If it does not, the command is refused with config_edit_conflict, naming who changed the definition and when, and asks the member to load the current version and re-apply the change. The check runs before the command changes anything, so a refused delete leaves a behavior's outputs and data and a workflow's runs in place. The refusal also lasts: running the same command again is refused until the member loads the definition, because the remembered version has not moved.

The extension lockfile works differently. A lockfile change merges entries into the current lockfile, so an edit there does not need to start from a particular version. Only a change another checkout lands while an extension command runs is refused; the refused change is kept and replayed by the next extension command.

An edit sent with --server runs in serve's own handler, and serve's copy is always the latest one, so serve can only refuse a change that lands while it applies the edit.

A v2 datastore has no such check: under managed config the last writer wins. For the exact error and its --json form, see Edit conflicts on a v3 datastore.

Namespace partitioning

Namespaces partition a shared datastore by repository. Each namespace gets its own data, definitions, and workflow runs while sharing the underlying storage backend. This is how Giga-Swamp repositories enable multiple teams to work against a single datastore without data collisions. See Giga-Swamp for the full namespace design and the isolation guarantees it provides.

Secrets follow a different path entirely -- local_encryption vault secrets (.swamp/secrets/) are excluded from the datastore's managed directory set and are never synced, migrated, or relocated, regardless of the backend. They stay local to the machine, which is what enables the shared-workflow-per-user- credentials pattern: the vault definition is git-tracked, but each operative's secret values remain on their own machine. See the vaults reference for the full exclusion rule and when to migrate to a cloud backend instead.

Namespace contamination

When multiple repositories share a remote datastore (S3 or GCS), each repository's namespace scopes its data into a distinct prefix. A push from repository infra writes objects under infra/, a push from platform writes under platform/, and so on. The namespace prefix is the boundary that keeps each team's data in its own subtree.

Before the namespace isolation fix, pushes did not always enforce this scoping. An unscoped push could write objects from one namespace into another namespace's prefix. The result is contamination: the infra prefix contains objects that actually belong to platform (or vice versa). The foreign objects are not dangerous -- they cannot corrupt the owning namespace's data -- but they create two problems.

First, they inflate the namespace's object count and storage usage. Garbage collection treats them as local data and will not remove them, because it has no way to know they are foreign.

Second, and more disruptive, they can shadow legitimate objects in the catalog index. When the catalog rebuilds from the remote prefix, it ingests the foreign objects alongside the real ones. This can make workflow runs invisible -- the catalog lists a foreign object where the real run's output should be. Suspended approval runs that depend on those outputs become unreachable via workflow approvals, because the catalog points to the wrong data.

The swamp doctor datastores subcommand detects contamination by scanning the namespace prefix for objects whose namespace metadata does not match. The --repair flag runs a five-step cleanup: delete foreign objects from the prefix, rebuild the remote index so it reflects only scoped data, wipe the local catalog cache (which may have indexed foreign objects), re-pull scoped data to rebuild a clean catalog, and invalidate workflow run and catalog indexes so previously shadowed runs become visible again. The preview-then-execute pattern (--repair to preview, --repair -y to execute) lets you review the scope of the cleanup before anything is modified.

Contamination only affects custom (remote) datastores with a namespace configured. The default filesystem backend stores data locally in .swamp/ with no namespace prefix, so there is no path for foreign objects to appear.

See Troubleshoot Your Swamp Repo for the step-by-step repair procedure and Doctor for the full flag and output reference.