DATASTORE CONFIGURATION

The datastore controls where Swamp stores runtime data — evaluated definitions, workflow runs, data outputs, audit logs, and telemetry. By default, everything lives in .swamp/ within the repository. An external filesystem path or an extension-provided backend (such as S3) can replace the default location. Vault secrets (.swamp/secrets/) are stored separately and are never included in datastore sync — see the vaults reference.

datastore:
  type: filesystem
  path: /data/swamp-store

Backend Types

filesystem

Stores data at a directory on the local filesystem.

When no datastore field is present in .swamp.yaml, Swamp uses the filesystem backend with path set to {repoDir}/.swamp/.

Extension backends

Extension-provided datastores use a scoped type identifier in @collective/name format (e.g., @swamp/s3-datastore). Extensions implement the Datastore Provider interface and are loaded from extensions/datastores/ within the repository.

The legacy type s3 is automatically remapped to @swamp/s3-datastore.

@swamp/s3-datastore

First-party extension that stores data in an Amazon S3 bucket with local cache synchronization. Distributed locking uses S3 conditional writes (If-None-Match: *). Bidirectional sync transfers files between a local cache directory and S3.

datastore:
  type: "@swamp/s3-datastore"
  config:
    bucket: my-swamp-bucket
    prefix: project-name
    region: us-east-1

Config fields

Authentication

Uses the default AWS credential chain — no credentials in the config object. Provide credentials via one of:

• Environment variables: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY
• AWS profile: ~/.aws/credentials
• SSO profile in ~/.aws/config, selected with AWS_PROFILE
• IAM role attached to the instance or task

AWS_CONFIG_FILE overrides the config file path. Refer to Authenticate to AWS with SSO for the sign-in sequence and session-expiry messages.

On a plain EC2 instance, the instance metadata credential lookup is disabled unless AWS_CONTAINER_CREDENTIALS_RELATIVE_URI, AWS_CONTAINER_CREDENTIALS_FULL_URI, or AWS_EC2_METADATA_DISABLED is set. Set AWS_EC2_METADATA_DISABLED=false to use an attached instance profile.

Required IAM permissions

• s3:HeadBucket
• s3:GetObject
• s3:PutObject
• s3:DeleteObject
• s3:ListBucket
• s3:HeadObject

Setup

$ swamp datastore setup extension @swamp/s3-datastore \
    --config '{"bucket":"my-bucket","region":"us-east-1"}'

With a key prefix and custom endpoint:

$ swamp datastore setup extension @swamp/s3-datastore \
    --config '{"bucket":"my-bucket","prefix":"swamp","endpoint":"https://minio.internal:9000","forcePathStyle":true}'

Environment variable

SWAMP_DATASTORE=@swamp/s3-datastore:{"bucket":"my-bucket","region":"us-east-1"}

The legacy format SWAMP_DATASTORE=s3:my-bucket/prefix is also accepted and auto-remapped. The part before the first / is the bucket; the rest is the prefix.

Sync

The S3 backend supports bidirectional sync via swamp datastore sync. Write commands automatically pull before executing and push after. Change detection compares file size and modification time against a remote index. Transfers run with a concurrency of 10.

The local cache lives at ~/.swamp/repos/{repoId}/ by default.

S3-compatible services

The endpoint and forcePathStyle fields enable use with S3-compatible services such as MinIO, DigitalOcean Spaces, Backblaze B2, and Cloudflare R2. Set forcePathStyle: true when the service does not support virtual-hosted-style bucket addressing.

@swamp/gcs-datastore

First-party extension that stores data in a Google Cloud Storage bucket. Distributed locking uses GCS generation-based preconditions. Bidirectional sync works the same way as the S3 extension.

datastore:
  type: "@swamp/gcs-datastore"
  config:
    bucket: my-gcs-bucket
    prefix: swamp

Authentication uses Google Cloud Application Default Credentials (ADC):

• Environment variable: GOOGLE_APPLICATION_CREDENTIALS pointing to a service account key JSON file
• User credentials: gcloud auth application-default login
• Attached service account on GCE, Cloud Run, or GKE

Required IAM permissions (covered by the roles/storage.objectAdmin predefined role):

• storage.buckets.get
• storage.objects.create
• storage.objects.get
• storage.objects.delete
• storage.objects.list

Configuration Fields

The datastore field in .swamp.yaml accepts the following properties.

type

Backend type identifier.

path

Absolute path for the filesystem datastore directory.

config

Arbitrary key-value configuration passed to extension-provided datastores. The extension's configSchema (if defined) validates this object at setup time. String values in config support expression interpolation.

datastore:
  type: "@swamp/s3-datastore"
  config:
    bucket: my-swamp-bucket
    prefix: project-name
    region: us-east-1

hydrationStrategy

Content download strategy for extension-provided datastores with sync support.

When set to "full", the initial sync downloads all files from the remote backend. When set to "lazy", the initial sync downloads only metadata files (metadata.yaml, latest markers, partition indexes) and skips content files (raw) under data/. Content files are downloaded on demand when first accessed.

Lazy hydration requires the extension to advertise lazyHydration in its SyncCapabilities and implement the hydrateFile method on its sync service. See the Datastore Provider reference.

datastore:
  type: "@swamp/s3-datastore"
  hydrationStrategy: lazy
  config:
    bucket: my-bucket
    region: us-east-1

directories

Which .swamp/ subdirectories are stored in the datastore. When omitted, all default subdirectories are included.

Default subdirectories:

definitions-evaluated    workflows-evaluated    data
outputs                  workflow-runs          audit
telemetry                logs                   files

Vault secrets (.swamp/secrets/) are excluded from this set — they always remain local to the machine regardless of datastore backend. See the vaults reference for details.

datastore:
  type: filesystem
  path: /data/swamp-store
  directories:
    - data
    - outputs
    - workflow-runs

Subdirectories not in this list remain in the local .swamp/ directory.

managedConfig

Opt-in flag that stores configuration in the datastore tier.

When true, model definitions, workflow definitions, vault configs, the extension lockfile, and pulled extension sources are stored in a config subdirectory within the datastore. The datastore becomes the single source of truth for configuration across swamp serve instances.

Set automatically by swamp datastore config migrate. Can also be set manually:

datastore:
  type: "@swamp/s3-datastore"
  managedConfig: true
  config:
    bucket: my-bucket
    region: us-east-1

Requires the datastore extension to support the configRefresh sync capability (see Config Refresh Capability). Without configRefresh, other instances cannot poll for config changes and managedConfig has no effect.

See Enable Managed Config for the step-by-step setup guide.

exclude

Gitignore-style patterns for files to exclude from datastore operations.

datastore:
  type: filesystem
  path: /data/swamp-store
  exclude:
    - "telemetry/**"

namespace

Namespace identifier for this repository within a shared datastore. When set, data is scoped under .swamp/<namespace>/ instead of .swamp/ directly.

datastore:
  type: "@swamp/s3-datastore"
  namespace: infra
  config:
    bucket: shared-bucket
    region: us-east-1

For new shared datastores, set the namespace during setup with swamp datastore setup extension --namespace <slug> — this assigns the namespace and migrates data in one step. For existing solo repositories transitioning to a namespace, use swamp datastore namespace set <slug> followed by swamp datastore namespace migrate --yes. See the Namespace Commands section below.

Legacy S3 shorthand fields

These fields are accepted in .swamp.yaml for backwards compatibility with the legacy s3 type. They are equivalent to placing the values inside config for the @swamp/s3-datastore extension.

Expression Interpolation in Config Values

String values in the datastore config object support expression interpolation using the ${{ }} delimiter syntax. Expressions are resolved during config resolution, before schema validation and createProvider(). Two expression namespaces are supported.

env

Resolves to the value of the named environment variable. Throws a startup error if the variable is not set or empty.

datastore:
  type: "@swamp/s3-datastore"
  config:
    bucket: "${{ env.SWAMP_S3_BUCKET }}"
    region: "${{ env.AWS_REGION }}"
    endpoint: "https://${{ env.S3_HOST }}:9000"

Expressions can appear as the entire value or embedded within surrounding text (as shown by endpoint above). When the entire value is a single expression, the resolved result preserves its type. When mixed with surrounding text, the result is coerced to a string.

vault.get

Resolves to the decrypted value of the named secret from the named vault. The vault service is initialized lazily — only when a vault.get() expression is encountered. All installed vault types are supported: the built-in local_encryption provider and extension providers (@swamp/aws-sm, @swamp/azure-kv, @swamp/1password, and custom vault extensions). Extension vault bundles are loaded from the .swamp/vault-bundles/ cache during early boot, so the extension must be installed before referencing it in datastore config.

datastore:
  type: "@swamp/s3-datastore"
  config:
    bucket: my-swamp-bucket
    region: us-east-1
    endpoint: "${{ vault.get('infra', 'minio-endpoint') }}"

See the Vaults reference for vault types, vault.get() quoting styles, and the CEL Expressions reference for the full expression language.

Limitations

• Only env and vault.get() are supported. The full CEL evaluation context (model, self, data, file) is not available at this bootstrap phase.
• Vault expressions are not supported when managedConfig is true. Managed config stores vault configurations in the datastore tier, which is not available during datastore initialization — a circular dependency. Environment variable expressions still work with managedConfig.
• An unsupported expression (anything other than env.* or vault.get()) produces a startup error naming the supported namespaces.

Resolution Priority

When multiple configuration sources exist, they are resolved in this order (highest priority first):

SWAMP_DATASTORE Environment Variable

Overrides the datastore configuration for a single invocation. Format:

SWAMP_DATASTORE=<type>:<value>

The legacy s3:bucket/prefix format is auto-remapped to @swamp/s3-datastore with the bucket and optional prefix extracted.

Environment variables within the value are expanded (e.g., filesystem:$HOME/swamp-data).

$ SWAMP_DATASTORE="filesystem:/tmp/override" swamp datastore status --json

{
  "type": "filesystem",
  "path": "/tmp/override",
  "healthy": true,
  "message": "Filesystem datastore at /tmp/override is healthy",
  "latencyMs": 0.36,
  "directories": [
    "definitions-evaluated",
    "workflows-evaluated",
    "data",
    "outputs",
    "workflow-runs",
    "audit",
    "telemetry",
    "logs",
    "files"
  ]
}

Managed Config

When managedConfig: true is set in .swamp.yaml, the datastore stores configuration alongside runtime data. The config subdirectory within the datastore holds:

• Model definitions (evaluated)
• Workflow definitions (evaluated)
• Vault configurations
• The extension lockfile
• Pulled extension sources

This makes the datastore the single source of truth for configuration. In a multi-instance swamp serve deployment, all instances read configuration from the shared datastore rather than from local directories, so a model created on one instance is available on every other instance without manual file distribution.

The config subdirectory

The config directory sits alongside the existing datastore subdirectories (data, outputs, workflow-runs, etc.) but is managed separately. It is not included in the directories list and is not subject to exclude patterns — it is governed entirely by the managedConfig flag.

Config Refresh Capability

The configRefresh sync capability is required for managed config to work with extension-provided datastores. It enables the datastore extension to poll for config changes pushed by other instances.

The S3 datastore (@swamp/s3-datastore) supports configRefresh from version 2026.08.27. The GCS datastore (@swamp/gcs-datastore) supports it from version 2026.08.27.

Without configRefresh, a swamp serve instance cannot detect config changes made by other instances. The managedConfig flag is accepted in .swamp.yaml but has no multi-instance effect — each instance sees only its own local writes.

What stays local

The following are never stored in the managed config tier:

• .swamp.yaml — the bootstrap configuration file. It must be distributed manually to each instance.
• Skills (.claude/skills/, etc.) — tool-specific instruction files.
• Extension kind directories (extensions/models/, etc.) — user-authored extension source code.
• Bundles — derived artifacts rebuilt from sources.

Known limitations

• Last-writer-wins on concurrent config writes. Config changes are infrequent and operative-initiated, so write conflicts are rare in practice. There is no merge or conflict resolution.
• ~30s propagation delay between instances. The ConfigPoller's default interval is 30 seconds. See swamp serve — Config management.
• Migration should run only once. Running swamp datastore config migrate on multiple instances can clobber index entries. Run it on a single instance and let other instances pick up the config via the poller.
• .swamp.yaml distribution is manual. Each instance needs its own copy of .swamp.yaml with managedConfig: true and the correct datastore configuration.

Sync Behavior

Sync applies only to extension-provided datastores that implement a sync service. Filesystem datastores do not sync.

Write commands follow this lifecycle:

1. Pull — download changed files from the remote backend to the local cache.
2. Execute — run the command against the local cache.
3. Push — upload changed files from the local cache to the remote backend.

Read-only commands skip sync entirely.

Change detection compares file size and modification time against a remote index. Transfer concurrency is capped at 10 concurrent file operations.

Cache path

Extension backends use a local cache directory for reads and writes. The sync service transfers data between this cache and the remote backend. Filesystem datastores access the configured path directly and have no separate cache.

Manual sync

$ swamp datastore sync

Triggers a full pull-then-push cycle. Only available for sync-capable extension datastores. Filesystem datastores return an error:

Datastore sync is only available for sync-capable custom datastores.
Current datastore type: filesystem

Distributed Locking

Write commands acquire a distributed lock before syncing and executing. The lock prevents concurrent writers from corrupting the datastore.

Lock metadata

Lock parameters

Heartbeat

The lock holder renews the lock every ttlMs / 3 milliseconds (10 seconds at the default TTL). Each renewal writes a fresh acquiredAt timestamp.

Stale lock detection

A lock is considered stale when either:

• The holder process is dead (checked via OS signal).
• The TTL has expired (acquiredAt + ttlMs < now).

Stale locks are automatically reclaimed by the next writer.

Nonce fencing

Each lock acquisition generates a unique UUID nonce. Heartbeat renewals verify the on-disk nonce matches the held nonce. If a mismatch is detected (another process reclaimed the lock), the holder self-revokes.

SIGINT handling

When a process receives SIGINT (Ctrl-C) during a locked operation, a best-effort lock release runs before exit.

Per-model locks

In addition to the global datastore lock, individual model operations acquire per-model locks scoped to the model type and ID. Both lock types use the same mechanism.

CLI Commands

All datastore commands accept the standard global options (--json, --log, --log-level, -q, -v, --no-telemetry, --no-color, --show-properties).

swamp datastore status

Show datastore configuration and health.

$ swamp datastore status

Datastore Status
  Type:    filesystem
  Path:    /home/user/my-repo/.swamp
  Health:  ● healthy (0ms)
  Dirs:    definitions-evaluated, workflows-evaluated, data, outputs, ...

$ swamp datastore status --json

{
  "type": "filesystem",
  "path": "/home/user/my-repo/.swamp",
  "healthy": true,
  "message": "Filesystem datastore at /home/user/my-repo/.swamp is healthy",
  "latencyMs": 0.40,
  "directories": [
    "definitions-evaluated",
    "workflows-evaluated",
    "data",
    "outputs",
    "workflow-runs",
    "audit",
    "telemetry",
    "logs",
    "files"
  ]
}

swamp datastore setup filesystem

Configure a filesystem datastore backend.

$ swamp datastore setup filesystem --path /data/swamp-store --json

{
  "type": "filesystem",
  "path": "/data/swamp-store",
  "filesCopied": 3,
  "bytesCopied": 94576,
  "directoriesMigrated": [
    "definitions-evaluated",
    "workflows-evaluated",
    "data",
    "outputs",
    "workflow-runs",
    "audit",
    "telemetry"
  ],
  "errors": []
}

When --directories is specified, only those subdirectories are moved to the external path. The rest remain in .swamp/.

$ swamp datastore setup filesystem \
    --path /data/swamp-store \
    --directories data,outputs,workflow-runs \
    --json

{
  "type": "filesystem",
  "path": "/data/swamp-store",
  "filesCopied": 3,
  "bytesCopied": 94576,
  "directoriesMigrated": [
    "data",
    "outputs",
    "workflow-runs"
  ],
  "errors": []
}

The resulting .swamp.yaml:

datastore:
  type: filesystem
  path: /data/swamp-store
  directories:
    - data
    - outputs
    - workflow-runs

swamp datastore setup extension

Configure an extension-provided datastore backend.

$ swamp datastore setup extension @swamp/s3-datastore \
    --config '{"bucket":"my-bucket","region":"us-east-1"}'

With a namespace (preferred for shared datastores):

$ swamp datastore setup extension @swamp/s3-datastore \
    --namespace infra \
    --config '{"bucket":"my-bucket","region":"us-east-1"}'

With lazy hydration:

$ swamp datastore setup extension @swamp/s3-datastore \
    --hydration-strategy lazy \
    --config '{"bucket":"my-bucket","region":"us-east-1"}'

swamp datastore sync

Manually sync the local cache with a remote datastore.

Without --pull or --push, runs a full sync (pull then push).

Only available for sync-capable extension datastores. Filesystem datastores return an error.

$ swamp datastore sync --pull
$ swamp datastore sync --push
$ swamp datastore sync

swamp datastore lock status

Show who holds the datastore lock.

$ swamp datastore lock status

Lock Status: no lock held

When a lock is held, the output includes the holder, hostname, PID, acquisition time, TTL, and nonce.

$ swamp datastore lock status --json

Returns the lock metadata object, or null if no lock is held.

swamp datastore lock release

Force-release a stuck datastore lock. This is a breakglass operation for recovering from a process that died without releasing its lock and the automatic stale-lock detection has not yet reclaimed it.

Without --model, releases the global datastore lock. With --model, releases the per-model lock for the specified model.

$ swamp datastore lock release --force --json

{
  "released": false,
  "reason": "no lock held"
}

$ swamp datastore lock release --force --model aws-ec2/my-server

swamp datastore config migrate

Migrate configuration into the datastore. Copies model definitions, workflow definitions, vault configs, the extension lockfile, and pulled extension sources into the datastore's config tier. Sets managedConfig: true in .swamp.yaml if not already set.

The command is idempotent — safe to re-run. A migration sentinel prevents duplicate work: the first run copies config and writes the sentinel; subsequent runs detect the sentinel and skip the copy.

Run this command on one instance only. Running it on multiple instances concurrently can clobber index entries in the datastore. Other instances pick up the migrated config automatically via the ConfigPoller on their next poll cycle.

$ swamp datastore config migrate

Migrated   config into datastore
           models: 3, workflows: 2, vaults: 1, lockfile: yes, extensions: 4
           Set managedConfig: true in .swamp.yaml

$ swamp datastore config migrate --json

{
  "migrated": true,
  "models": 3,
  "workflows": 2,
  "vaults": 1,
  "lockfile": true,
  "extensions": 4,
  "managedConfigSet": true
}

When the migration has already been run (sentinel present):

$ swamp datastore config migrate --json

{
  "migrated": false,
  "reason": "already migrated"
}

Setup Pipeline

When swamp datastore setup runs, it follows this sequence:

1. Validate — verify the target is accessible (writable directory or reachable remote).
2. Migrate — copy existing data from .swamp/ to the new location (unless --skip-migration). See Directory Relocation for which directories move and where they end up.
3. Verify — compare file counts between source and destination.
4. Hydrate — for extension datastores, pull existing remote data into the local cache. Runs unconditionally regardless of --skip-migration.
5. Update — write the datastore field to .swamp.yaml.
6. Clean up — remove migrated subdirectories from .swamp/.

Directory Relocation

When a datastore is configured on a repository that already has data in .swamp/, the setup command relocates runtime directories from .swamp/ to the datastore location. The CLI prints which directories were moved and how many files were copied.

Which directories relocate

The default set of relocatable directories:

definitions-evaluated    workflows-evaluated    data
outputs                  workflow-runs          audit
telemetry                logs                   files

These are the same directories listed under the directories configuration field. If directories is set in .swamp.yaml, only those directories are relocated; the rest stay in .swamp/.

Directories outside this list — such as auto-definitions/, config/, secrets/, and internal state files — always remain in .swamp/ and are never relocated. In particular, local_encryption vault secrets stay local to the machine even when the rest of the repository's state syncs through a shared datastore.

Where directories move to

The destination depends on the backend type:

For filesystem datastores, the directories are moved directly to the configured path. For extension-provided datastores, the directories are moved into the local cache directory; the sync service then pushes them to the remote backend.

Hardcoded path warning

After relocation, the directories no longer exist under .swamp/. Code or scripts that hardcode paths like .swamp/workflow-runs/ or .swamp/data/ will break. Use CLI commands to resolve paths programmatically:

$ swamp workflow run search --json     # find workflow runs
$ swamp data get <model> <output>      # read data outputs
$ swamp datastore status --json        # inspect the active datastore path

Path Resolution

Each file operation resolves to either the local .swamp/ directory or the configured datastore path:

• If the file's parent subdirectory is in the directories list and does not match an exclude pattern, it goes to the datastore path.
• Otherwise, it stays in local .swamp/.

For extension backends with sync support, the "datastore path" is the local cache directory. The sync service handles transfer between the cache and the remote backend.

Related

• Repository Configuration — datastore field in .swamp.yaml
• Namespace Commands — CLI reference for swamp datastore namespace (below)
• Giga-Swamp and Namespaces — why namespaces exist and how they work
• Extension Manifest — packaging datastore extensions
• CEL Expressions — full expression language reference (env, vault, and other context variables)
• Vaults — secrets stored in the secrets datastore subdirectory
• Data — data stored in the data datastore subdirectory
• Enable Managed Config — step-by-step guide for multi-instance config convergence
• swamp serve — Config management — architectural decisions behind ConfigPoller and config propagation

Namespace Commands

The swamp datastore namespace command group manages datastore namespaces. Namespaces scope data within a shared datastore to identify which repository produced it.

All commands accept the standard global options (--json, --log, --log-level, -q, -v, --no-telemetry, --no-color, --show-properties, --repo-dir).

For background on namespaces and shared datastores, see Giga-Swamp and Namespaces.

swamp datastore namespace set

Assign a namespace to this repository.

swamp datastore namespace set <slug>

Writes the namespace field to the datastore section of .swamp.yaml and registers the namespace in the datastore's namespace manifest. Does not move existing data — run namespace migrate --yes after setting to reorganize data into the namespaced layout.

$ swamp datastore namespace set infra

Namespace set to "infra"
  Datastore: /home/user/my-repo/.swamp

Warning: Existing data remains at the old un-namespaced path.
Run 'swamp datastore namespace migrate --yes' to move it
to the namespaced layout.

swamp datastore namespace unset

Remove the namespace from this repository.

swamp datastore namespace unset

Removes the namespace field from .swamp.yaml. Without --migrate, data remains at its namespaced path. With --migrate --yes, data is moved from .swamp/<namespace>/... back to .swamp/....

swamp datastore namespace migrate

Migrate data between solo and namespaced directory layouts.

swamp datastore namespace migrate

Without --yes, displays a preview of the migration: each directory to be moved, source and destination paths, file count, and total size. No data is modified.

With --yes, moves each directory from the solo layout (.swamp/<dir>/) to the namespaced layout (.swamp/<namespace>/<dir>/), or the reverse with --reverse. The catalog is invalidated after migration and rebuilt on the next access.

The directories migrated are: definitions-evaluated, workflows-evaluated, data, outputs, workflow-runs, audit, telemetry.

swamp datastore namespace list

List all namespaces in the datastore.

swamp datastore namespace list

$ swamp datastore namespace list --json

{
  "namespaces": [
    {
      "namespace": "infra",
      "repoId": "94735c06-fc0a-4d77-b712-04fbd801b18d",
      "registeredAt": "2026-05-15T14:30:00.000Z",
      "isCurrent": true
    }
  ],
  "currentNamespace": "infra"
}

swamp datastore catalog pull

Pull catalog metadata from foreign namespaces.

swamp datastore catalog pull --namespaces <namespaces>

Fetches catalog entries (model names, types, data output names, version counts) from the specified foreign namespaces into the local catalog cache. The pulled metadata is read-only and has no automatic refresh or TTL.

$ swamp datastore catalog pull --namespaces infra,security