Gcs Datastore

Store data in a Google Cloud Storage bucket with local cache synchronization. Provides distributed locking via GCS generation-based preconditions with exponential backoff under contention and bidirectional sync between a local cache directory and GCS. Emits OpenTelemetry spans for GCS operations, lock acquisition/release, and push/pull sync so datastore activity is visible in traces when a TracerProvider is configured. Features shard-first indexing with per-model partition shards under _index/ as the source of truth (v2 repos skip the monolithic index upload entirely), commitSeq-based zero-diff fast path, compare-and-swap (ifGenerationMatch) merges for every shard and _meta.json write so concurrent writers never drop each other's index entries, two-phase sync to narrow the global-lock critical section, SHA-256 content hashing, per-path dirty tracking, namespace-scoped sync for multi-repo shared datastores, configurable transfer concurrency, configurable per-request timeout, managed config refresh for pulling config partitions from the datastore, and a single-file remote read that writes nothing to the local cache.

Groundwork for the v3 datastore format: a datastore-wide control plane at the root _control/ that holds the v3 writer lease and replication lineage (no namespaced pull or push moves _control/writer/, _control/lineage/ or _control/datastore-format, and the provider declares datastoreWideLease), and storage/1 chunk storage under _blobs/ with 16 MiB single-request puts, a price-tuned has, and a touch that refreshes a chunk by a rewrite of it onto itself, with no re-upload and no new permissions.

The Litestream replica target for v3: replicaTarget() names the bucket and the datastore root that storage() and the control plane write under, with no credentials (Litestream reads the same default credential chain), and refuses a config it cannot describe, including a custom apiEndpoint, which Litestream cannot use. replicaPaths() lists and removes the Litestream lineage paths under <prefix>/_litestream/<fence>/, so swamp can collect a lineage a takeover replaced and stop deferring chunk collection. Swamp's v3 on buckets (Phase 4E) needs this version or later.

Keeps the v2 file sync off a v3 datastore, for swamp releases older than v20261008.201010.0 that check no format marker: _litestream/ is never indexed, pulled or pushed, and the sync service reads the root _control/datastore-format once, alongside the credentials preflight, and refuses format 3 or newer, or a garbled marker, with datastore_format_unsupported before it syncs anything. An unreadable marker does not block that operation, and the next one reads it again.

Authentication

Uses Google Cloud Application Default Credentials (ADC) — no credentials in config. Provide credentials via one of:

• Environment variable: GOOGLE_APPLICATION_CREDENTIALS pointing to a service account key JSON file
• User credentials: gcloud auth application-default login
• Attached service account on GCE, Cloud Run, or GKE

Required IAM Permissions

• storage.buckets.get
• storage.objects.create
• storage.objects.get
• storage.objects.delete
• storage.objects.list

The predefined role roles/storage.objectAdmin covers all of these.

Usage

swamp datastore setup @swamp/gcs-datastore \
  --config '{"bucket": "my-bucket", "prefix": "swamp"}' --json