Relationships
#4825 S3 and GCS datastores: copy replica files into a new Litestream lineage (the replicaPaths copy method)
Opened by stack72 · 11/16/2026
Why
Team-mode serve takeover is being made flat in database size: the new leader seeds its Litestream lineage by copying the old lineage's files inside the bucket, instead of re-uploading a full snapshot (27.8 s at 558 MB, measured). Swamp core never holds bucket credentials and cannot reach lineage paths through storage, so the copy must come from the datastore provider, the same way replicaPaths() list and removePath do.
What
Core adds an optional method to ReplicaPathStore that copies named files from /_litestream// to /_litestream/<fence f+1>/, plus a conformance case in packages/testing. Implement it in both providers:
- @swamp/s3-datastore: server-side CopyObject per file.
- @swamp/gcs-datastore: server-side copy (rewrite) per file.
Rules, enforced by the conformance case:
- both fences in the 12-digit padded form; nothing outside the two fence paths is read or written;
- only the files named are copied (the caller names the snapshot and the L0 files at or below the published head);
- a failure partway throws, so core falls back to today's full snapshot and never writes the new lineage's start record.
A provider without the method keeps today's behaviour.
Done when
Both providers pass the new conformance case, and the GCS one is exercised by the team-mode takeover VM run on the 558 MB dogfood datastore.
Closed
No activity in this phase yet.
system commented 11/16/2026, 9:53:46 PM
Classified automatically when this issue was filed.
- Source: Extensions
If you feel this classification is incorrect, add a ripple to tell us so.
stack72 commented 11/17/2026, 1:45:38 PM
Closed as not needed. The only user was swamp-club#4824, which stopped under its stop rule: on Litestream 0.5.17 a lineage seeded by server-side copy still uploads a full database image as its first L0 (320,085,046 bytes at 558 MB, measured on serve-a), so seeding saves nothing. If a copy method is ever revived, copy in parallel or in batch: per-object gcloud copies of 118 files (305 MB) took 187-190 s, against 2 s for one whole-tree copy.
Sign in to post a ripple.