Relationships
#3508 Datastore rework Phase 4 G: S3 and GCS extensions implement replicaTarget() for v3 Litestream replication
Opened by stack72 · 10/15/2026· Shipped 10/16/2026
Background
swamp is reworking its datastore layer (tracking: swamp-club#2865). Phase 4E puts the v3 format on S3 and GCS:
- a checkout's
commits.dbis replicated by Litestream to the datastore's bucket, under<prefix>/_litestream/<fence>/; - chunks go through
storage/1; - the writer lease and lineage records live in the datastore-wide control plane.
@swamp/s3-datastore and @swamp/gcs-datastore 2026.10.15.2 (swamp-club#3395, #3492) already provide the control plane and storage/1.
The last piece the extensions owe is telling swamp where Litestream replicates: the core contract replicaTarget(), defined in swamp-club#3507. swamp-extensions has issues turned off, so this is tracked here.
What is needed
replicaTarget()on the S3 provider:{ type: "s3", bucket, prefix, region?, endpoint?, forcePathStyle? }from the datastore's config.prefixis the same datastore root thatstorage()anddatastoreControlPlaneStore()use.- Covers AWS and every S3-compatible endpoint the extension supports (R2, MinIO, DigitalOcean Spaces and others).
replicaTarget()on the GCS provider:{ type: "gcs", bucket, prefix }.- Refuse when
apiEndpointis set, with a clear error: Litestream has no custom GCS endpoint, so it would replicate to real GCS instead of the emulator.
- Refuse when
- No credentials in the descriptor. Both providers resolve credentials from the default chains (AWS SDK chain, Google ADC), and Litestream does the same. Document that in the README: whatever credentials the datastore uses must also be visible to the Litestream sidecar, through the same environment or shared config.
- Run
assertReplicaTargetConformancefrom@swamp-club/swamp-testingonce core publishes it. - Bump both manifests. The README and manifest should say that Phase 4E names this version as the minimum.
Tests
- Unit tests: the descriptor for each config shape (region, endpoint, path style, empty and nested prefixes); GCS's refusal with
apiEndpoint. - The conformance suite.
- An integration check against ministack and fake-gcs-server where
SWAMP_TEST_*_ENDPOINTis set: a Litestream replica configured from the descriptor lands under<prefix>/_litestream/. GCS only without a custom endpoint, so this may be S3-only locally. Say so.
Blocked by: swamp-club#3507.
Shipped
Click a lifecycle step above to view its details.
system commented 10/15/2026, 6:50:52 PM
Classified automatically when this issue was filed.
- Source: Extensions
If you feel this classification is incorrect, add a ripple to tell us so.
stack72 commented 10/15/2026, 10:42:14 PM
Scope addition (swamp-club#3531): please also implement replicaPaths() on both providers in the same release as replicaTarget(). It's an optional capability scoped to <prefix>/_litestream/, with list(signal) (fence directories present) and removePath(fence, signal) (delete every object under <prefix>/_litestream/<fence>/, paging, idempotent, never outside that path; refuse a fence that isn't the 12-digit padded form). Core needs it to collect abandoned lineage paths on S3 and GCS. Without it, chunk GC stays deferred forever on buckets (swamp-club#3491's rule). The core contract and conformance land in #3531; replicaTarget() can start now, since @swamp-club/swamp-testing 0.20261015.56 already ships assertReplicaTargetConformance. Pick up replicaPaths() once #3531 publishes its conformance suite.
Sign in to post a ripple.