Skip to main content
← Back to list
01Issue
BugShippedSwamp CLIPublic
Assigneesstack72

Relationships

#4581 Team mode: a member's recovery after an interrupted command replays the whole history (minutes at a day of history); recover only the affected data names (starts after #4579)

Opened by stack72 · 11/10/2026· Shipped 11/10/2026

Starts after swamp-club#4579 (member recovery failing with "Cannot apply commit N"), which changes the same code. Don't run the two at once. It isn't a dogfood run 7 gate: run 7 judges failover on rounds where no member command was interrupted mid-write.

What happens

When a team member's command is interrupted (a failover, a kill, a crash), or loses its lease so its commits are dropped, the checkout is recovered with a full read-model replay of the whole history: by the next command for an interrupted writer, and in the same command's teardown for dropped work. In team-mode dogfood run 5 (swamp-club#4374), that was the 186–404 s member commands after failover rounds ("pattern B" in #4559's analysis), and the cost grows with history. #4559 fixed the same problem on serve (a follower recovers its killed predecessor at boot) but not in member checkouts.

  • Run 4's F37: four full member recoveries of 504–622 s, about 4 GB written each.
  • A full recovery of a day-old member datastore: 621.6 s and 4.09 GB (#4512's baseline).

Why the obvious shortcut doesn't work

#4512 tried a targeted recovery (catch the catalog up from the cursor, check only files changed since the dead writer started) and withdrew it. #4559 tried it again and reverted it (its diff is step10-withdrawn.diff in #4559's scratchpad). The reason is in design/enablers/datastores.md, "Every recovery is a full rebuild":

  • a data name's catalog rows depend on its whole op history (a data.gc expiry drops rows);
  • the dead writer may have left catalog rows for uncommitted data.

The gate test is sqlite_read_model_test.ts's ignored data.gc case: the targeted result must equal a full rebuild, row for row.

Design to build (design A from #4512)

  • Recover by replaying each touched data name's committed history (the compacted base plus later commits), not the whole datastore.
  • Find the touched names from what the dead writer could have changed: files changed since its marker started, owned files gone missing, and catalog rows written but uncommitted.
  • Re-project their catalog rows exactly, including the effects of data.gc expiry.
  • This needs a new commit-store read (a partition's or name's history from the base onward) and a replay for each one.
  • The full rebuild stays as the fallback for anything the targeted path can't verify. Keep #4512's stoppable, resumable recovery.

Tests

  • The ignored data.gc gate test: enabled and passing.
  • Row-for-row equality: targeted against full rebuild, across a property-based mix of saves, appends, renames, gc expiries, deletes and an interrupted writer.
  • A work-count test: an interrupted writer after a long history recovers by touching only the affected names (work independent of total history).

Done when

The tests pass. A member's recovery after an interrupted command at a day of history takes seconds, not minutes, measured on the next dogfood.

Which recovery run 5's slow commands took (span scan, swamp-club#4579 session with #4374)

Every member command of 150 s or more in run 5's member spans (198 commands, release 20261108.111402.0-sha.37c27525):

Recovery entry / path Commands
dropped_work / full 152
interrupted_writer / full 31
interrupted_writer / resumed 11
view_change / full, or both entries in one command 2
none (a harness token fault) 2
  • Where a recover span exists it takes 93–100% of the command (median 99%), replaying about 2,500–3,000 commits. readmodel.load_base takes 36–131 s of it.
  • Not request timeouts: 17 of 68,800 serve requests timed out; the slowest single request was 0.3–1.3 s.
  • dropped_work runs in the same command's teardown (closeV3Datastores → recoverDroppedWork) after a unit's commits are dropped, which is why many slow commands end in datastore_v3_commit_lease_lost.

Scope (agreed)

  • Both entries. The per-name replay serves dropped_work first (77% of the cost; the dropped directories are already known in process) and interrupted_writer (the touched names found as in "Design to build").
  • The base load is part of it. Load only the touched names' compacted-base rows, not the whole base.
  • Done when also covers dropped work: a command that loses its lease at a day of history ends in seconds.

Also in this issue, first: F43 (#4374 run 6, part E)

In c06-default round 1 (SIGSTOP), the frozen old leader serve-a resumed after serve-b took the writer lease (fence 66 against 65). At 11:15:47 it still landed commit 3312 to ops-writer's partition ("landed but could not be confirmed as replicated (datastore_v3_writer_fenced)"). serve-b, restored from the lineage, wrote a different commit at seq 3312. member-1's copy held serve-a's 3312 and was refused with serve_upstream_diverged for the rest of the run, until someone moved it aside by hand. No acknowledged write was missing.

Cause, from the code:

  • serve appends a commit to its local store before confirmReplicated, which is where writer_fenced surfaces.
  • The unconfirmed-leader gate is checked at request entry, so a request accepted before the freeze appends after it.
  • The feed and partition reads serve locally landed commits whether or not they are replicated.

Work, in order:

  1. Characterise on current main with two-serve SIGSTOP tests: a leader frozen past its lease, and a variant with a commit in flight when it freezes. Record what #4559's unconfirmed-leader refusal already covers.
  2. Fence at the append (a). Re-check the writer fence under the commit lock in the same step as the local append. A commit whose confirm fails with writer_fenced is never made visible.
  3. Replicated watermark (b). The feed and partition reads serve only commits up to the replicated watermark.
  4. May-land settlement (c). A member never applies a 503 may-land commit locally. It settles the commit by reading from the current leader; a re-send under the same txnId is deduplicated.
  5. Divergence repair (d). A diverged copy (other-commit or behind, from a confirmed leader) re-sends its outbox. It then drops unacknowledged commits the leader doesn't hold, so nobody has to move it aside by hand. Commits this checkout landed, and other-log divergence, stay manual.

Then the targeted recovery above. Only a landed outbox entry clears its dirty names.

Done when (added): run 7's failover rounds show no member copy stuck in serve_upstream_diverged. This gates run 7.

02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 7 MOREREVIEW+ 16 MOREPR_LINKED+ 2 MORESESSION_SUMMARIZED

Shipped

11/10/2026, 11:47:00 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack7211/10/2026, 5:23:27 PM

Sign in to post a ripple.