#3490 Datastore rework Phase 4 F: a durable outbox for v3 commits, the post-commit chunk check on recovery, and pruning compacted_txns

Background (read this first)

swamp is reworking its datastore layer (tracking: swamp-club#2865). In the new format, v3, every change is a commit in a single-writer SQLite commits.db. Content is stored as content-addressed chunks, and each checkout rebuilds its files from the commits. v3 is opt-in today for filesystem datastores (swamp-club#3279), and swamp-uat's filesystem-v3 backend tests it in CI.

Phase 4 is "solo mode on buckets":

• the CLI is the single writer of a local commits.db;
• Litestream replicates it to the bucket;
• chunks go to the bucket through storage/1;
• a fenced writer lease makes sure only one machine writes.

Shipped so far:

• A, swamp-club#3357: replication cost, choosing an idle sidecar.
• B, swamp-club#3358: the pinned, hash-checked Litestream binary.
• C, swamp-club#3359: the fenced writer lease (src/domain/datastore_v3/writer_lease.ts, src/infrastructure/datastore_v3/v3_writer_lock.ts). Every CLI write command takes a new fence, serve keeps one for its lifetime, and commits.db keeps a store-wide writer_fence that refuses lower fences.
• D1, swamp-club#3416: a dormant Litestream sidecar (src/infrastructure/datastore_v3/litestream/, src/domain/datastore_v3/litestream_replication.ts and replication_lineage.ts).
  • A write command confirms its commits are replicated (POST /sync with wait, replicated_txid >= txid, then the lease check) before it succeeds.
  • Replication follows a lineage: the replica path is _litestream/<fence>/, named by the fence at which this copy's replication began. Another writer's fence restarts it on a new path. The lineage is recorded in lineage.json beside commits.db, with abandoned ones in previousLineages.
  • Nothing replicates unless the internal SWAMP_V3_REPLICA setting (replica_setting.ts, tests only) names a file or S3 replica.
• D2, swamp-club#3482: restore, refresh and takeover, also dormant.
  • While replicating, each checkout keeps its own commits.db under .swamp/_datastore_v3_copies/.
  • Lineages are published under _control/lineage/<fence>/ (started.json, head-<writer fence>.json, confirmed.json).
  • A restore truncates to the published head's high-water mark.
  • See "Phase 4D2" in datastore-litestream.md.

The decisions are in design/enablers/datastore-litestream.md ("Decision", "Restore, failover and fences (Q6)", "Phase 4D1") and in design/enablers/datastores.md (the "v3 …" sections).

Goal

Phase 4 F: make a commit that a v3 command staged durable across a crash, and close the gaps Phase 3 left for "the Phase 4 outbox". It's three items, recorded in design/enablers/datastores.md.

The gaps

1. A killed run's staged changes are lost.
   • The commit-log unit (src/infrastructure/datastore_v3/commit_log_unit_of_work.ts) appends through a CommitSink (append, head). Phase 3 passes the local CommitStore.
   • A partition whose commit failed stays pending with its txnId, and the next checkpoint or commit retries it. But the pending groups live in memory, and commit ends the unit whether or not it succeeds. So what's pending when a process dies (SIGKILL, OOM, power) is lost.
   • Phase 3's answer is only detect-or-reconcile: swamp-club#3299's interrupted-writer recovery (interrupted_writer_recovery.ts). It removes the files no commit names, and commits a killed run as cancelled.
   • datastores.md says "the Phase 4 outbox makes an interrupted commit durable instead".
2. #3249's post-commit chunk check can be skipped. A writer that dies after its append lands and before its has() check runs, while a chunk sweep deletes one of that commit's chunks, leaves a stored commit naming a missing chunk.
   • "Phase 4's outbox, which knows which commits it sent, is where a recovery would re-run the check."
   • And: an outbox entry that is sent later, possibly after a sweep, must keep its content sources and run the check after it lands, or store its content again before sending.
3. compacted_txns is never pruned (about 120 bytes per compacted commit, swamp-club#3258). It can be pruned once the outbox bounds how old a retry can be.

Design

1. An outbox behind the CommitSink seam. Pending commit groups are written durably on this machine, beside the local commits.db, before the unit reports them as committed or pending. They are removed once the append lands.
   • A recovery on the next writer start (the same place #3299's recovery runs) appends what is left, with the same txnId. A group that had landed reports duplicate and lands once.
   • Locate it from the commit store's own path (v3CommitsDbPath), never from the datastore path. While replicating, commits.db is the checkout's copy (D2), and the outbox must follow it.
   • Decide and record the outbox's store. A table in commits.db keeps it in the same file Litestream replicates, but an unsent group should not count as a commit. A separate local SQLite file avoids that. Say which, and why.
2. Fences. A recovered entry is appended under the current writer's lease.
   • If the lease changed hands since the entry was written (another writer took a fence in between), the entry is dropped and reported, never appended. This matches the lease's rule that a writer that lost it lands nothing; the store's writer_fence refuses it anyway.
   • Say what the user sees: the run was already reported failed or cancelled by #3299's recovery.
3. Content and the chunk check. An outbox entry keeps its content sources until it lands. After it lands, recovery re-runs the writer's post-commit check (#3249), or stores the content again before sending, as datastores.md requires.
   • Recovery also re-runs the check for commits it knows were sent but not checked, which closes gap 2.
4. Bound retry age and prune compacted_txns. Entries older than a bound (decide it; days, not hours) are dropped and reported, and compacted_txns rows older than the bound are pruned during compaction. Record the bound in datastores.md.
5. What #3299 keeps doing: its recovery still removes files no commit names, and still commits a killed run as cancelled when nothing of it can be recovered. Say in datastores.md what each mechanism now covers.
6. Replication: an appended entry is replicated like any commit (D1). A command's success still waits for the confirmed sync. Nothing changes there.

Tests

• Crash tests: a run killed (a Worker or a subprocess in an integration test, as #3299's tests do) after staging and before its append lands. The next writer start appends it once; a duplicate is reported as one.
• A killed run, then another writer takes the lease before recovery: the entry is dropped and reported, and nothing lands.
• Gap 2: a writer killed between its append and its has() check, with a sweep deleting the chunk in between. Recovery finds and repairs it. The test fails without the fix.
• Retry bound and pruning: an old entry is dropped; compacted_txns rows older than the bound are pruned, and newer ones are kept.
• Property or model-based test (fast-check): over random interleavings of stage, kill, recover, take over and compact, a group lands at most once, never under a stale fence, and no stored commit names a missing chunk.
• v2 doesn't change; apply the v2 gate. Existing v3 crash-safety suites pass unedited.

Out of scope

• E: turning v3 on for S3 and GCS.
• Phase 5's team-mode outbox (forwarding to the serve leader): keep the seam general enough for it, but don't build it.

Blocked by: nothing. A takeover restore (D2) replaces or truncates the local commits.db; the outbox's fence rule (design 2) covers entries left from before it. If the outbox is a table in commits.db, say what a restore does to it.