DATASTORE ARCHITECTURE

Every model method execution, workflow run, audit event, cached bundle, and encrypted secret passes through one datastore. A single persistence layer makes backup, migration, and sharing atomic operations rather than ad-hoc scripts that chase files across scattered directories.

Source-of-truth definitions -- model files, workflow files, vault configs -- stay in git-tracked directories (models/, workflows/, vaults/) so that version control and code review apply to them. The datastore holds only runtime artifacts: the outputs those definitions produce when executed. See How Swamp Works for the broader execution model.

Why three backend options

The default backend stores data in .swamp/ inside the repository -- no configuration, no network, no external dependencies. This is the right choice for a single operative working on one machine.

An external filesystem path (such as an NFS mount) enables multiple machines to share state without cloud dependencies. The tradeoff is operational: shared filesystems introduce network latency and require mount management.

An S3-compatible backend enables geo-distributed access with object versioning. It is itself a Swamp extension (@swamp/s3-datastore), not a built-in -- the datastore system is extensible by design. Extension-provided backends require Swamp authentication because they cross the boundary from single-machine state into shared infrastructure -- the local filesystem backend affects only one machine, but an S3 bucket affects everyone who shares it, so authentication provides the accountability boundary. See the datastore configuration reference for backend setup and the API key scoping explanation for the design rationale.

The catalog database

On top of whatever backend stores the raw files, the datastore maintains a SQLite catalog that indexes all artifacts. The catalog exists because scanning a filesystem (or making S3 LIST calls) for every data.query() or data.latest() would be prohibitively slow. Indexed lookup through SQLite turns those operations into millisecond queries regardless of how many artifacts exist.

The catalog is a cache, not a source of truth. If deleted or corrupted, the datastore rebuilds it from the underlying storage. This self-healing property means the catalog is never synced to remote backends -- each machine builds its own from whatever data it has locally.

Why lazy hydration is the default

When using an S3 backend, the datastore must decide what to download. Eager hydration (downloading everything on startup) guarantees all data is available locally, but most commands touch a small fraction of the total data. Downloading gigabytes of historical model outputs to run one method wastes bandwidth and startup time.

Lazy hydration downloads metadata for catalog visibility but defers content retrieval until something actually reads the artifact. The tradeoff is explicit: queries filtering on attributes content may see null for un-hydrated data. This is a documented limitation -- the alternative (eager sync) trades correctness for performance in the opposite direction.

Per-model locking

When multiple machines share a datastore, concurrent writes to the same model must be serialized. The locking scope is per-model rather than global because a workflow running a deploy model should not block a separate scan model from executing concurrently on another machine. Per-model locks keep contention proportional to actual conflicts.

The mechanism is a file lock with a TTL of approximately 30 seconds. The executing process writes a lock file and renews it via a heartbeat for the duration of the method run. If the process crashes or is killed without releasing the lock, stale-lock detection compares the lock's PID against the process table. Once the TTL expires and the owning PID is confirmed dead, the lock is reclaimed automatically by the next run that needs it.

When a run cannot acquire the lock, the CLI exits with code 75 (lock_timeout) so the caller can retry with backoff. For stuck locks that stale-detection cannot clear -- the owning process is on another host, or the PID has been recycled -- the breakglass commands swamp datastore lock status and swamp datastore lock release --force provide manual intervention. See the operational commands reference for usage.

Global locks exist for structural operations -- garbage collection, model deletion, datastore migration -- where cross-model consistency matters. These are rare and short-lived by design. For why fan-out methods reduce lock contention within a single model, see How Swamp Works.

Namespace partitioning

Namespaces partition a shared datastore by repository. Each namespace gets its own data, definitions, and workflow runs while sharing the underlying storage backend. This is how Giga-Swamp repositories enable multiple teams to work against a single datastore without data collisions. See Giga-Swamp for the full namespace design and the isolation guarantees it provides.

Secrets follow a different path entirely -- they pass through the vault system, which manages encryption and access control independently of the datastore backend. See the vaults reference for how secrets are stored and scoped.

Namespace contamination

When multiple repositories share a remote datastore (S3 or GCS), each repository's namespace scopes its data into a distinct prefix. A push from repository infra writes objects under infra/, a push from platform writes under platform/, and so on. The namespace prefix is the boundary that keeps each team's data in its own subtree.

Before the namespace isolation fix, pushes did not always enforce this scoping. An unscoped push could write objects from one namespace into another namespace's prefix. The result is contamination: the infra prefix contains objects that actually belong to platform (or vice versa). The foreign objects are not dangerous -- they cannot corrupt the owning namespace's data -- but they create two problems.

First, they inflate the namespace's object count and storage usage. Garbage collection treats them as local data and will not remove them, because it has no way to know they are foreign.

Second, and more disruptive, they can shadow legitimate objects in the catalog index. When the catalog rebuilds from the remote prefix, it ingests the foreign objects alongside the real ones. This can make workflow runs invisible -- the catalog lists a foreign object where the real run's output should be. Suspended approval runs that depend on those outputs become unreachable via workflow approvals, because the catalog points to the wrong data.

The swamp doctor datastores subcommand detects contamination by scanning the namespace prefix for objects whose namespace metadata does not match. The --repair flag runs a five-step cleanup: delete foreign objects from the prefix, rebuild the remote index so it reflects only scoped data, wipe the local catalog cache (which may have indexed foreign objects), re-pull scoped data to rebuild a clean catalog, and invalidate workflow run and catalog indexes so previously shadowed runs become visible again. The preview-then-execute pattern (--repair to preview, --repair -y to execute) lets you review the scope of the cleanup before anything is modified.

Contamination only affects custom (remote) datastores with a namespace configured. The default filesystem backend stores data locally in .swamp/ with no namespace prefix, so there is no path for foreign objects to appear.

See Troubleshoot Your Swamp Repo for the step-by-step repair procedure and Doctor for the full flag and output reference.