#3586 v3 outbox: a reported failure can land, a commit can name a deleted chunk, and a fenced writer is told nothing landed when it did (bug hunt group 3)

Background

swamp is reworking its datastore layer (tracking: swamp-club#2865). On a v3 datastore every unit of work prepares its commit in the commit outbox (commits-outbox.db, swamp-club#3490) before appending it to commits.db. A later writer's outbox recovery lands the prepared commits of writers that died, settles the ones that landed, and drops the ones that can't land. The outbox's rule (commit_outbox.ts, datastores.md):

• a commit whose failure was reported never lands;
• a killed writer's commit lands exactly once;
• no stored commit names a missing chunk.

A v3 bug hunt (2026-10-16, verified on main d3b76490c, still present on 83a7777a3) found that the outbox breaks these rules in several ways. They matter most once takeovers between machines are routine, which v3 on S3 and GCS (swamp-club#3534) now makes them. This issue fixes them together, since they share commit_log_unit_of_work.ts, commit_outbox.ts, commit_outbox_recovery.ts and v3_datastore.ts.

Since swamp-club#3564 (writer liveness, outbox schema 2), an entry a recovery left for a live writer survives the floor (left_at_fence), so a later recovery can land it once its writer is gone. Such an entry, whose writer then resumes and is refused, is kept from landing only by that writer's discard. A failed discard (bug 1) therefore reaches left entries too, which the floor used to stop. Cover them.

The bugs

1. A reported failure lands when the discard also fails (V3-08, S2). discardEntries (commit_log_unit_of_work.ts ~:1083) swallows a failed discard, and end() still rethrows the original error. retryOwedRemovals (outbox_commit_sink.ts) retries once, and V3Datastore.close only warns. The entry carries no "unit failed" mark, so the next writer's recoverEntry lands it.

• Reproduced: the append fails with database or disk is full, then the discard and the close retry fail, and the command exits 1. After space is freed, the next command logs "Landed 1 v3 commits", and a re-run of the command applies the change twice.
• Trigger: a disk that fills between prepare and append and stays full, a read-only remount, or persistent EIO.

2. A crash after an append that raced a chunk sweep leaves a commit naming a deleted chunk (V3-11, S2). The unit's pre-append restoreMissingChunks check (:1027) runs once, while the sweep (chunk_collector.ts) may still delete a chunk it judged unreferenced before the prepare. The only repairs, the sweep's post-delete re-check and the unit's post-append check (:1045), run in the same process. One crash skips both, and recovery of a landed entry only warns.

• Reproduced: the sweep deletes chunk X, the append lands, then SIGKILL. Recovery returns {landed: 0, settled: 1}, and a fresh checkout's catch-up fails with "Chunk … is missing", so every other checkout is wedged.
• Trigger: swamp serve writing content identical to an old released chunk while its maintenance sweep runs, then OOM, kill -9 or a power cut in the window.

3. A fenced writer is told "nothing was committed" though its successor landed it (V3-12, S3 after verification). Recovery lands an entry whose writer it can't judge, as designed: the recovery runs under a newer lease. When the paused writer resumes, outbox_commit_sink.ts's assertWriterCurrent throws the cached loss before the store's duplicate check. end() discards the landed entry and rethrows, and nothing checks commitStore.landedSeq(txnId) before reporting.

• Reproduced: the writer gets datastore_v3_writer_fenced saying "nothing more from this command was committed. Run the command again." while commits.db holds its txnId. A re-run duplicates the change.

4. The fence raise and the outbox chain record are separate transactions (V3-30, S3). V3Datastore.raiseWriterFence (v3_datastore.ts ~:556-558) runs commitStore.raiseWriterFence(fence) then outbox.recordFence(fence). A writer start that dies or fails between them (or after takeLease consumed the fence) leaves a gap. decideFenceChain restarts the chain on any gap, and dropReason drops a killed writer's prepared commit as stale-fence, with the misleading message "another writer took the datastore after it was prepared".

• Reproduced both ways: killed between the two writes, and SQLITE_FULL on the raise. The control lands the commit.
• Trigger: two faults, a killed writer and then the next writer's start killed or failing.

5. A storage error in the post-commit chunk check reports a landed commit as failed (V3-29, S3). restoreMissingChunks runs after the append inside appendGroup (~:1045). The group leaves pending only after appendGroup returns, so end() drops the landed work and discards its entry. Reproduced: unit.commit() rejects with an EIO from stat while the partition holds the commit. warnMissingChunks has the same gap.

6. Dropped-work recovery sweeps a root that is still flushing (V3-35, S4). end() calls tracker.ended() (~:620) before the root's final flush, and before await flushing on abandon. recoverDroppedWork checks only units.open, so a teardown recovery sweeps the root's file and its commit lands afterwards. It heals at the next catch-up. Reproduced.

7. A landed entry is dropped and reported "not in the datastore" (V3-41, S4). planOutboxRecovery's drop reasons (expired, stale-fence) run before recoverEntry's landedSeq check. A landed entry whose settle was lost (power cut) is dropped with a false data-loss warning when the next recovery is more than 7 days later or after a chain restart (bug 4). Data is intact. Reproduced.

8. openUnit() roots ignore a running dropped-work recovery (V3-42, S4). engineFor().openUnit() (v3_datastore.ts ~:616) never checks units.recovering, unlike runRoot. Separately, dropped() keeps min(), and the units.droppedSince === since equality clears a drop recorded during the recovery. Reproduced at API level. No production caller found (serve uses runRoot).

9. A pending retry reuses its stored chunks without refreshing them (V3-52, S4, latent). The retry path (~:958-980) reuses the stored op with no touch and skips the pre-append check, contrary to datastores.md's "a pending retry stores and refreshes its content again before each append". Past the 24 h grace a sweep could delete the chunks. No production path keeps a pre-prepare group pending that long today.

Note (S4, V3-28): a read-only command's interrupted-writer recovery removes a killed writer's file as "uncommitted", then the next writer's outbox recovery lands the commit and the file reappears on catch-up. The end state is correct, but the two recoveries disagree in between. Fix it if it falls out of the work; otherwise document it.

What to do

1. A reported failure never lands (1). Pick one and say which:

   • (Recommended) a durable abandoned mark that recovery honours, written in the same outbox write as the discard attempt's first step. A unit that fails marks its entries abandoned before it reports, and recovery drops abandoned entries whatever their writer's state. If even the mark can't be written, the command must say its change may land later, not that it failed.
   • or fail closed: keep retrying the discard, and don't release the lease until it succeeds.

   This must cover entries left by #3564 (left_at_fence set), and fix datastores.md's "only a process that dies leaves entries".

2. No stored commit names a missing chunk (2). Make the sweep's delete crash-safe (quarantine by rename, or a durable deleting-batch row that recovery re-checks), or make an overlapping writer wait for the sweep's in-flight batch. Also let recovery of a landed entry restore missing chunks from the checkout's file when it has them, rather than only warn.

3. Report what actually happened (3, 5, 7):

   • after any append failure following a prepare, check commitStore.landedSeq(txnId) and report success, or "may have landed" where it can't be read, before telling the user to re-run;
   • once sink.append resolves, the group has landed: post-append check failures become warnings;
   • in recovery, check landedSeq before the plan-level drop reasons, allowing for txnIds pruned after 8 days.

4. One transaction for the fence (4): record the fence in the outbox before, or atomically with, the store raise. Or never judge stale-fence on a gap in shared mode, if you can show that's safe. Fix the drop message.

5. Recovery waits for roots (6, 8):

   • call tracker.ended() in a finally after the final flush and discard (and after await flushing);
   • openUnit() roots respect units.recovering;
   • replace the droppedSince timestamp equality with a generation counter.

6. Retries refresh their chunks (9): touch or re-check reused chunks before preparing a retry, or correct datastores.md.

7. Schema: a new outbox column or mark means a schema bump (2 → 3) with the same one-way, "upgrade swamp" barrier as #3564. If a commit body or commits.db changes, that's a commits.db bump. Say which, and why it's needed.

Tests

Each must fail on main first. Use the outbox fixture and real stores; child processes for the crash cases go in integration/.

• 1: append fails, discard fails, close retry fails, command reports failure; after the fault clears, the next recovery lands nothing. Repeat with a left entry (#3564): left at recovery R1, writer resumes and is refused, its discard fails; a later recovery after the writer is gone lands nothing.
• 2: the sweep deletes a chunk between the pre-check and the append, then the writer is killed; recovery leaves no stored commit naming a missing chunk, and a fresh checkout catches up.
• 3: a writer paused between prepare and append is taken over and its entry landed; on resume it reports committed (or "may have landed"), not "nothing was committed".
• 4: killed or failing between raise and record; the next writer lands the dead writer's commit.
• 5: a post-append check that throws EIO reports the commit as committed, with a warning.
• 6, 8: recovery during a root's final flush returns busy; an openUnit root opened during recovery waits or is refused; a drop recorded during recovery stays owed.
• 7: a landed entry whose settle was lost is settled, not dropped, after 7+ days and after a chain restart.
• 9: a retry past the grace re-stores or touches its chunks.
• swamp-uat passes unchanged.

Severity: two S2s (a reported failure lands; a commit names a missing chunk and wedges other checkouts), plus S3/S4 wrong reports and races.

Blocked by: nothing; swamp-club#3564 has shipped. It runs in parallel with the crashed-workflow-runs issue (bug hunt group 6), which also touches interrupted-writer recovery. Expect a small rebase. It must land before swamp-uat's s3-v3/gcs-v3 suites (datastore rework Phase 4 H).