#3489 Datastore rework Phase 4: serve confirms each v3 write request is replicated before it answers (group commit)

Background (read this first)

swamp is reworking its datastore layer (tracking: swamp-club#2865). In the new format, v3, every change is a commit in a single-writer SQLite commits.db. Content is stored as content-addressed chunks, and each checkout rebuilds its files from the commits. v3 is opt-in today for filesystem datastores (swamp-club#3279), and swamp-uat's filesystem-v3 backend tests it in CI.

Phase 4 is "solo mode on buckets":

• the CLI is the single writer of a local commits.db;
• Litestream replicates it to the bucket;
• chunks go to the bucket through storage/1;
• a fenced writer lease makes sure only one machine writes.

Shipped so far:

• A, swamp-club#3357: replication cost, choosing an idle sidecar.
• B, swamp-club#3358: the pinned, hash-checked Litestream binary.
• C, swamp-club#3359: the fenced writer lease (src/domain/datastore_v3/writer_lease.ts, src/infrastructure/datastore_v3/v3_writer_lock.ts). Every CLI write command takes a new fence, serve keeps one for its lifetime, and commits.db keeps a store-wide writer_fence that refuses lower fences.
• D1, swamp-club#3416: a dormant Litestream sidecar (src/infrastructure/datastore_v3/litestream/, src/domain/datastore_v3/litestream_replication.ts and replication_lineage.ts).
  • A write command confirms its commits are replicated (POST /sync with wait, replicated_txid >= txid, then the lease check) before it succeeds.
  • Replication follows a lineage: the replica path is _litestream/<fence>/, named by the fence at which this copy's replication began. Another writer's fence restarts it on a new path. The lineage is recorded in lineage.json beside commits.db, with abandoned ones in previousLineages.
  • Nothing replicates unless the internal SWAMP_V3_REPLICA setting (replica_setting.ts, tests only) names a file or S3 replica.
• D2, swamp-club#3482: restore, refresh and takeover, also dormant.
  • While replicating, each checkout keeps its own commits.db under .swamp/_datastore_v3_copies/.
  • Lineages are published under _control/lineage/<fence>/ (started.json, head-<writer fence>.json, confirmed.json).
  • A restore truncates to the published head's high-water mark.
  • See "Phase 4D2" in datastore-litestream.md.

The decisions are in design/enablers/datastore-litestream.md ("Decision", "Restore, failover and fences (Q6)", "Phase 4D1") and in design/enablers/datastores.md (the "v3 …" sections).

The gap

A serve write request reports success before its commits are replicated, so a takeover after serve dies can drop writes serve already acknowledged.

• A CLI write command confirms its commits before it succeeds. At teardown it syncs, publishes the lineage head, then checks the lease (syncAndPublish in src/infrastructure/datastore_v3/v3_repo_context.ts).
• Serve doesn't, per request. Its commits replicate passively (Litestream's 1–2 s lag). It publishes a head only on its 30 s replication probe, at drains and at shutdown (startV3ReplicationUpkeep).
• D2's restore truncates to the published head's high-water mark. So if serve dies, a takeover drops up to 30 s of commits that serve had acknowledged to its clients, even ones Litestream had already uploaded.

That breaks Phase 4's decision: a write waits for replication before it reports success, so no acknowledged write is lost. It's dormant today, and must be closed before Phase 4E turns replication on.

What to do

Serve confirms each write request's commits before it answers, like a CLI command.

1. When a write request has committed, and before serve sends its success, run the same sync, publish-head and lease-check sequence the CLI uses (syncAndPublish), and wait for it.
   • Find the seam: the place a request's root unit ends and serve answers. That is likely where v2's pushChangedToRemote (src/serve/handlers/shared.ts) runs for sync-capable datastores. Say where you put it.
2. Group commit. Concurrent requests share syncs.
   • A request whose commits landed before a sync started is covered by that sync.
   • Otherwise it waits for the next one.
   • Never one sync per request when several are in flight: one sync in flight, and the next one starts as soon as it finishes, covering everything committed meanwhile.
   • syncAndPublish already serialises per datastore.
3. Failure follows the CLI:
   • an unconfirmed sync fails the request with datastore_v3_replication_unconfirmed, and its commits stay local and ship with the next sync;
   • a fenced lease fails it with the fenced error, using the replicating wording from #3482.
4. Read-only requests never wait. Serve's background writes (bookkeeping GC, boot reconciliation, scheduled runs) confirm at their own end where they acknowledge something to a user. Otherwise the 30 s probe still covers them; list which is which in the PR.
5. Workflow runs through serve: a run's steps commit as they go. Confirm at the points serve reports a step or run as done to a client, not after each internal commit. Say which points those are.
6. Keep the 30 s probe as the backstop for commits nothing waits on.
7. Without SWAMP_V3_REPLICA, nothing changes: no sync, no added latency.

Tests

• Integration, with real Litestream and a file replica (the D1/D2 harness):
  • serve answers a write only after its head is published;
  • kill serve right after it answers, take over from another checkout, and the acknowledged commit is there;
  • concurrent writes share syncs (count the syncs);
  • an unconfirmed sync fails the request;
  • a read doesn't sync.
• Unit tests for the group-commit coordinator, with an injected clock and no sleeps.
• Dormancy: with the setting unset, serve's request path makes no sync call (a fitness rule or a test double).
• Report the latency added per write request on a warm sidecar against a file replica and the local S3 emulator. Phase 4A modelled 24–283 ms.

Blocked by: nothing. Phase 4E waits for it.