#3482 Datastore rework Phase 4 D2: restore, refresh and takeover from the v3 replica, with a checkout-local commits.db (dormant)

Background (read this first)

swamp is reworking its datastore layer (tracking: swamp-club#2865). In the new format, v3, every change is a commit in a single-writer SQLite commits.db. Content is stored as content-addressed chunks, and each checkout rebuilds its files from the commits. v3 is opt-in today for filesystem datastores (swamp-club#3279), and swamp-uat's filesystem-v3 backend tests it in CI.

Phase 4 is "solo mode on buckets":

• the CLI is the single writer of a local commits.db;
• Litestream replicates it to the bucket;
• chunks go to the bucket through storage/1;
• a fenced writer lease makes sure only one machine writes.

Shipped so far:

• A, swamp-club#3357: replication cost, choosing an idle sidecar.
• B, swamp-club#3358: the pinned, hash-checked Litestream binary.
• C, swamp-club#3359: the fenced writer lease (src/domain/datastore_v3/writer_lease.ts, src/infrastructure/datastore_v3/v3_writer_lock.ts). Every CLI write command takes a new fence, serve keeps one for its lifetime, and commits.db keeps a store-wide writer_fence that refuses lower fences.
• D1, swamp-club#3416: a dormant Litestream sidecar (src/infrastructure/datastore_v3/litestream/, src/domain/datastore_v3/litestream_replication.ts and replication_lineage.ts).
  • A write command confirms its commits are replicated (POST /sync with wait, replicated_txid >= txid, then the lease check) before it succeeds.
  • Replication follows a lineage: the replica path is _litestream/<fence>/, named by the fence at which this copy's replication began. Another writer's fence restarts it on a new path. The lineage is recorded in lineage.json beside commits.db, with abandoned ones in previousLineages.
  • Nothing replicates unless the internal SWAMP_V3_REPLICA setting (replica_setting.ts, tests only) names a file or S3 replica.

The decisions are in design/enablers/datastore-litestream.md ("Decision", "Restore, failover and fences (Q6)", "Phase 4D1") and in design/enablers/datastores.md (the "v3 …" sections).

Goal

Phase 4 D2: a v3 copy that isn't the replica's writer can get the datastore's commits from the replica, and a writer that takes over from another machine starts from them.

• Restore onto a fresh machine.
• Refresh a copy that is behind.
• Take over from another writer with the old writer fenced.

Ship it dormant, behind the same internal SWAMP_V3_REPLICA setting as D1. Phase 4E switches it on for S3 and GCS.

What's missing today

1. commits.db lives in the shared datastore directory. On a bucket each machine has its own. With replication on, nothing places a checkout-local commits.db, so two "machines" can't be modelled.
2. The current lineage is known only locally. lineage.json sits beside commits.db, so another machine can't find which _litestream/<fence>/ path holds the datastore's state.
3. Nothing restores. A copy with no commits.db, or a stale one, has no way to fetch the replica's state. A writer that takes the lease after another machine wrote would write on top of a stale database.
4. Old lineage paths are never removed. previousLineages records them for later collection.

Design

1. A checkout-local commits.db when replicating. While SWAMP_V3_REPLICA is set, each checkout keeps its own commits.db (and Litestream's state and lineage.json beside it) in the checkout's .swamp/, not in the shared datastore directory.
   • Chunks, the format marker and the lease stay in the datastore.
   • Two checkouts on one filesystem-v3 datastore then behave like two machines on a bucket, which is what the tests use.
   • Unset, nothing changes.
   • This was listed under E; doing it here makes D2 testable end to end. Update E's plan entry.
2. Publish the lineage in the datastore's control plane, next to the lease (_control/, through ControlPlaneStore; on buckets that binding is swamp-club#3364).
   • Use create-only records, as the lease does, so two writers can never both claim a lineage.
   • Record:
     • that a lineage started: its start fence, holder and path;
     • that its first sync was confirmed.
   • "The current state" is the highest lineage with a confirmed sync.
   • Record the scheme in datastore-litestream.md.
3. Restore when the local copy is missing or behind, before the command's first read or write:
   • missing: restore the current lineage into a temp file, check it (PRAGMA integrity_check, schema version), then rename it into place;
   • behind: Litestream 0.5 has no incremental restore outside follow mode, so a refresh is a full restore with -force into a temp file, then a swap.
   • Decide and record how "behind" is detected cheaply, for example the current lineage record against the local lineage.json, plus a last-confirmed TXID published with the lineage. A read of an up-to-date copy must cost no download.
   • Read-only commands refresh too, so reads on machine B see machine A's confirmed writes. Measure the cost of a refresh check and of a refresh at 1 MB and 100 MB.
4. Takeover. A writer whose lease fence restarts the lineage (another writer took a fence in between) restores the current lineage before it writes, then starts its own lineage at its fence, which needs a full first snapshot.
   • Keep this ordering:
     • the old writer: confirm the sync, then check the lease, then acknowledge;
     • the new writer: take the lease, then restore.
   • So every commit the old writer acknowledged is in what the new writer restores. A zombie's later uploads go to its own old path, which nothing restores from once a newer lineage is confirmed.
5. Garbage-collect old lineage paths after the new lineage's first confirmed sync: the paths in previousLineages and lineages older than the current one. Never touch the current lineage, or one that isn't confirmed yet.
6. The first-clone budget. A small restore costs about 32 sequential round trips (Phase 4A). Report what a fresh machine pays before its first command, and keep the restore off the path when nothing changed.
7. Windows stays unavailable while the pin is below 0.5.18.

Tests

• Integration, against a real Litestream with a file replica and two checkouts on one filesystem-v3 datastore (repeat-safe, no leaked processes):
  • a fresh checkout restores and sees every acknowledged commit;
  • a stale reader refreshes;
  • an up-to-date reader downloads nothing;
  • a takeover restores before writing, starts a new lineage, and keeps every commit the old writer acknowledged;
  • a zombie (paused past its lease, resumed after a takeover) lands nothing in what the new writer or a fresh restore sees;
  • a writer killed after its lineage record but before its first confirmed sync doesn't become "the current state";
  • GC removes only abandoned paths.
• The opt-in S3 case against a local emulator, as D1 has it.
• Property or model-based tests (fast-check) for the lineage records: over random sequences of take, confirm, crash and take-over by two or three writers, the current state is always the latest confirmed lineage, and no acknowledged commit is missing from it.
• Dormancy: with SWAMP_V3_REPLICA unset, nothing changes on filesystem-v3 and nothing is written to the control plane. v2 doesn't change; apply the v2 gate.

Out of scope

• E: turning it on for S3 and GCS, chunks through storage/1, the marker (#3204) and binding the lease at the bucket root (#3364).
• F: the outbox and the durability leftovers.
• Team mode's followers (restore -f warm standby) are Phase 5.

Blocked by: nothing (D1 has shipped). Runs in parallel with F.