#3416 Datastore rework Phase 4 D1: Litestream sidecar for v3 writers, and a write command waits for replication before it reports success (dormant)

Background (read this first)

swamp is reworking its datastore layer (tracking: swamp-club#2865). In the new format, v3, every change is a commit in a single-writer SQLite commits.db. Content is stored as content-addressed chunks, and each checkout rebuilds its files from the commits. v3 is opt-in today for filesystem datastores (swamp-club#3279), and swamp-uat's filesystem-v3 backend tests it in CI.

Phase 4 is "solo mode on buckets":

• the CLI is the single writer of a local commits.db;
• Litestream replicates it to the bucket;
• chunks go to the bucket through storage/1;
• a fenced writer lease makes sure only one machine writes.

Done so far:

• A, swamp-club#3357: replication cost on a bucket model, choosing an idle sidecar. See design/enablers/datastore-litestream.md, section "Phase 4A".
• B, swamp-club#3358: download, verify and cache a pinned Litestream (0.5.17), dormant: src/infrastructure/datastore_v3/litestream/litestream_binary_provider.ts.
• C, swamp-club#3359: the fenced writer lease (src/domain/datastore_v3/writer_lease.ts, src/infrastructure/datastore_v3/v3_writer_lock.ts). Every CLI write command takes a new fence, and serve keeps one for its lifetime. commits.db keeps a store-wide writer_fence.

The decisions are in design/enablers/datastore-litestream.md, under "Decision" and "What changes for Phase 4D and 4E", and in design/enablers/datastores.md, under the "v3 …" sections.

Goal

Run Litestream as a sidecar for a v3 writer, and make a write command wait until its commits are replicated before it reports success. This is Phase 4 D, part 1. Restore and takeover are D2, a separate issue.

Ship it dormant, as #3358 did:

• on a filesystem-v3 datastore, commits.db already lives in the shared datastore directory, so nothing needs replicating;
• Phase 4E turns it on for S3 and GCS;
• here, it runs only when a test or an internal setting gives it a replica.

Test it against a file replica and an S3 emulator.

Design (decided in Phase 4A; see "What changes for Phase 4D and 4E")

1. An idle sidecar, not one per command.
   • A write command reuses a running sidecar for the datastore copy, or starts one.
   • The sidecar exits after a five-minute idle timeout.
   • Read-only commands never start one.
2. Finding the sidecar:
   • a pidfile next to commits.db and a short socket path. Stay under the AF_UNIX path limit; macOS's is about 104 bytes.
   • a probe with a deadline. A frozen sidecar's socket still accepts connections and never answers; 4A saw a command hang there.
   • after the deadline, kill the process the pidfile names, but only once its argv shows it is this database's sidecar, and start a new one. A pidfile can name an unrelated live pid.
   • never let two sidecars share a replica path.
3. An idle exit that can't overlap a command. Each command holds a shared lock while it uses the sidecar. The sidecar exits only once it can take that lock exclusively after the timeout. 4A's prototype exited on the pidfile's age alone and could stop Litestream during a command.
4. Waiting for replication.
   • Before a write command reports success:
     1. it calls POST /sync {"path": …, "wait": true} once;
     2. it checks replicated_txid >= txid (the rule in "Knowing a commit is replicated");
     3. it runs checkV3WriterLease (C) before it acknowledges.
   • One sync covers every commit the command made.
   • If the sync fails or times out, the command fails with a clear error. Its commits stay in the local WAL, and the next start ships them; the error says so.
5. Teardown order: sync, close swamp's stores, then release the sidecar. If swamp holds the last connection when it closes commits.db, SQLite checkpoints and deletes the WAL, and the next sidecar uploads a full snapshot ("Teardown order" in 4A). Check the first restart after an idle timeout with this order.
6. Replica path keyed by lineage, not by fence: <replica root>/_litestream/<fence>/, where <fence> is the fence at which this copy's replication began.
   • The lineage continues while the fence just before a command's own was this copy's: no other writer took the lease in between.
   • Otherwise it restarts at the command's fence.
   • Record the last fence when the lease is taken, so a command that crashed is continued by its successor.
   • Litestream reads its replica path from its config at start, so a sidecar on the old path is stopped and replaced when the lineage changes.
   • Prototype: spike/exp_lineage.ts on spike/3357-litestream-buckets.
7. The binary:
   • call LitestreamBinaryProvider.resolve() right before each spawn, so the hash is checked before every start;
   • Windows: keep it unavailable while the pin is below 0.5.18 (litestreamAvailability). Leave the litestream sync subprocess client for when 0.5.18 is pinned.
8. Serve keeps one sidecar for as long as it holds the lease, and syncs on drain within --shutdown-drain-timeout.
9. Serve also clears the read model's unsynced list (unsynced_keys in read_model_state.ts, from swamp-club#3337). Run a barrier from V3MaintenanceTimer (v3_repo_context.ts) that syncs the listed files once and clears them. Otherwise a long-running serve's first open after a reboot hashes most of its cache.
10. Config: commits.db keeps SQLite's default checkpointing and never forces one (a fitness rule refuses wal_checkpoint in v3 code).

Tests

• Integration, with Litestream as a real subprocess against a file replica. A port-0 S3 emulator is also fine, if the repo's test rules allow it for this adapter; otherwise put it behind an opt-in variable, like #3358's network test. Cover:
  • a command's commits are confirmed before it returns;
  • one sync covers a burst;
  • a sidecar is reused across commands;
  • an idle exit doesn't overlap a command;
  • a frozen sidecar is replaced after the deadline;
  • a stale pidfile naming an unrelated pid is ignored;
  • a lineage continues across commands and restarts after another writer's fence;
  • the teardown order leaves no full snapshot on the next start.
• Unit tests with an injected clock for the idle timer, the probe deadline and lineage decisions. No sleeps.
• v2 doesn't change, and nothing changes on a filesystem-v3 datastore while the sidecar is dormant. Apply the v2 gate: suites unedited, and the route recorder identical.

Out of scope

• D2: restore onto a fresh machine and takeover from another checkout.
• F (durability leftovers): a killed run's staged changes, re-running #3249's chunk check, pruning compacted_txns.
• E: turning v3 on for S3 and GCS, chunk uploads through storage/1, and binding the lease at the bucket root (#3364).

Blocked by: nothing (A, B and C have shipped).