S3 Datastore

Store data in an Amazon S3 bucket with local cache synchronization. Provides distributed locking via S3 conditional writes with exponential backoff under contention and bidirectional sync between a local cache directory and S3. Emits OpenTelemetry spans for S3 operations, lock acquisition/release, and push/pull sync so datastore activity is visible in traces when a TracerProvider is configured. Features shard-first indexing with per-model/per-workflow partition shards under _index/ as the source of truth (v2 repos skip the monolithic index upload entirely), commitSeq-based zero-diff fast path, compare-and-swap (If-Match) merges for every shard and _meta.json write so concurrent writers never drop each other's index entries, two-phase sync to narrow the global-lock critical section, SHA-256 content hashing, per-path dirty tracking, namespace-scoped sync for multi-repo shared datastores, configurable transfer concurrency, configurable per-request timeout, managed config refresh for pulling config partitions from the datastore, a single-file remote read that writes nothing to the local cache, and a create-only repair that uploads cache files the datastore never received.

Groundwork for the v3 datastore format: a datastore-wide control plane at the root _control/ that holds the v3 writer lease and replication lineage (no namespaced pull or push moves _control/writer/, _control/lineage/ or _control/datastore-format, and the provider declares datastoreWideLease), and storage/1 chunk storage under _blobs/ with 16 MiB single-request puts, a price-tuned has, and a touch that refreshes a chunk by a copy of it onto itself, with no re-upload and no new permissions.

The Litestream replica target for v3: replicaTarget() names the bucket and the datastore root that storage() and the control plane write under, with no credentials (Litestream reads the same default credential chain), and refuses a config it cannot describe. replicaPaths() lists and removes the Litestream lineage paths under <prefix>/_litestream/<fence>/, so swamp can collect a lineage a takeover replaced and stop deferring chunk collection. Swamp's v3 on buckets (Phase 4E) needs this version or later.

Keeps the v2 file sync off a v3 datastore, for swamp releases older than v20261008.201010.0 that check no format marker: _litestream/ is never indexed, pulled or pushed, and the sync service reads the root _control/datastore-format once, alongside the credentials preflight, and refuses format 3 or newer, or a garbled marker, with datastore_format_unsupported before it syncs anything. An unreadable marker does not block that operation, and the next one reads it again.

Authentication

Uses the default AWS credential chain — no credentials in config. Provide credentials via one of:

• Environment variables: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY
• AWS profile: ~/.aws/credentials
• IAM role attached to the instance or task

Required IAM Permissions

• s3:GetObject (authorizes GetObject, HeadObject)
• s3:PutObject
• s3:DeleteObject
• s3:ListBucket (authorizes ListBucket, HeadBucket)

Usage

swamp datastore setup @swamp/s3-datastore \
  --config '{"bucket": "my-bucket", "prefix": "swamp", "region": "us-east-1"}' --json

S3-Compatible Endpoints

Supports S3-compatible services (MinIO, DigitalOcean Spaces, etc.) via the endpoint and forcePathStyle config options. Verification probes conditional writes (If-None-Match and If-Match) under a throwaway _control/ key, so an endpoint that silently ignores those headers — on which distributed locking would be unsafe — fails setup instead of reporting healthy.