S3 Datastore

Store data in an Amazon S3 bucket with local cache synchronization. Provides distributed locking via S3 conditional writes and bidirectional sync between a local cache directory and S3. Emits OpenTelemetry spans for S3 operations, lock acquisition/release, and push/pull sync so datastore activity is visible in traces when a TracerProvider is configured. Features shard-first indexing with per-model/per-workflow partition shards under _index/ as the source of truth (v2 repos skip the monolithic index upload entirely), commitSeq-based zero-diff fast path, two-phase sync to narrow the global-lock critical section, SHA-256 content hashing, per-path dirty tracking, namespace-scoped sync for multi-repo shared datastores, configurable transfer concurrency, and configurable per-request timeout.

Authentication

Uses the default AWS credential chain — no credentials in config. Provide credentials via one of:

• Environment variables: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY
• AWS profile: ~/.aws/credentials
• IAM role attached to the instance or task

Required IAM Permissions

• s3:HeadBucket
• s3:GetObject
• s3:PutObject
• s3:DeleteObject
• s3:ListBucket
• s3:HeadObject

Usage

swamp datastore setup @swamp/s3-datastore \
  --config '{"bucket": "my-bucket", "prefix": "swamp", "region": "us-east-1"}' --json

S3-Compatible Endpoints

Supports S3-compatible services (MinIO, DigitalOcean Spaces, etc.) via the endpoint and forcePathStyle config options.