Relationships
#4581 Team mode: a member's recovery after an interrupted command replays the whole history (minutes at a day of history); recover only the affected data names (starts after #4579)
Opened by stack72 · 11/10/2026· Shipped 11/10/2026
Starts after swamp-club#4579 (member recovery failing with "Cannot apply commit N"), which changes the same code. Don't run the two at once. It isn't a dogfood run 7 gate: run 7 judges failover on rounds where no member command was interrupted mid-write.
What happens
When a team member's command is interrupted (a failover, a kill, a crash), or loses its lease so its commits are dropped, the checkout is recovered with a full read-model replay of the whole history: by the next command for an interrupted writer, and in the same command's teardown for dropped work. In team-mode dogfood run 5 (swamp-club#4374), that was the 186–404 s member commands after failover rounds ("pattern B" in #4559's analysis), and the cost grows with history. #4559 fixed the same problem on serve (a follower recovers its killed predecessor at boot) but not in member checkouts.
- Run 4's F37: four full member recoveries of 504–622 s, about 4 GB written each.
- A full recovery of a day-old member datastore: 621.6 s and 4.09 GB (#4512's baseline).
Why the obvious shortcut doesn't work
#4512 tried a targeted recovery (catch the catalog up from the cursor, check only files changed since the dead writer started) and withdrew it. #4559 tried it again and reverted it (its diff is step10-withdrawn.diff in #4559's scratchpad). The reason is in design/enablers/datastores.md, "Every recovery is a full rebuild":
- a data name's catalog rows depend on its whole op history (a
data.gcexpiry drops rows); - the dead writer may have left catalog rows for uncommitted data.
The gate test is sqlite_read_model_test.ts's ignored data.gc case: the targeted result must equal a full rebuild, row for row.
Design to build (design A from #4512)
- Recover by replaying each touched data name's committed history (the compacted base plus later commits), not the whole datastore.
- Find the touched names from what the dead writer could have changed: files changed since its marker started, owned files gone missing, and catalog rows written but uncommitted.
- Re-project their catalog rows exactly, including the effects of
data.gcexpiry. - This needs a new commit-store read (a partition's or name's history from the base onward) and a replay for each one.
- The full rebuild stays as the fallback for anything the targeted path can't verify. Keep #4512's stoppable, resumable recovery.
Tests
- The ignored
data.gcgate test: enabled and passing. - Row-for-row equality: targeted against full rebuild, across a property-based mix of saves, appends, renames, gc expiries, deletes and an interrupted writer.
- A work-count test: an interrupted writer after a long history recovers by touching only the affected names (work independent of total history).
Done when
The tests pass. A member's recovery after an interrupted command at a day of history takes seconds, not minutes, measured on the next dogfood.
Which recovery run 5's slow commands took (span scan, swamp-club#4579 session with #4374)
Every member command of 150 s or more in run 5's member spans (198 commands, release 20261108.111402.0-sha.37c27525):
| Recovery entry / path | Commands |
|---|---|
dropped_work / full |
152 |
interrupted_writer / full |
31 |
interrupted_writer / resumed |
11 |
view_change / full, or both entries in one command |
2 |
| none (a harness token fault) | 2 |
- Where a recover span exists it takes 93–100% of the command (median 99%), replaying about 2,500–3,000 commits.
readmodel.load_basetakes 36–131 s of it. - Not request timeouts: 17 of 68,800 serve requests timed out; the slowest single request was 0.3–1.3 s.
dropped_workruns in the same command's teardown (closeV3Datastores→recoverDroppedWork) after a unit's commits are dropped, which is why many slow commands end indatastore_v3_commit_lease_lost.
Scope (agreed)
- Both entries. The per-name replay serves
dropped_workfirst (77% of the cost; the dropped directories are already known in process) andinterrupted_writer(the touched names found as in "Design to build"). - The base load is part of it. Load only the touched names' compacted-base rows, not the whole base.
- Done when also covers dropped work: a command that loses its lease at a day of history ends in seconds.
Also in this issue, first: F43 (#4374 run 6, part E)
In c06-default round 1 (SIGSTOP), the frozen old leader serve-a resumed after serve-b took the writer lease (fence 66 against 65). At 11:15:47 it still landed commit 3312 to ops-writer's partition ("landed but could not be confirmed as replicated (datastore_v3_writer_fenced)"). serve-b, restored from the lineage, wrote a different commit at seq 3312. member-1's copy held serve-a's 3312 and was refused with serve_upstream_diverged for the rest of the run, until someone moved it aside by hand. No acknowledged write was missing.
Cause, from the code:
- serve appends a commit to its local store before
confirmReplicated, which is wherewriter_fencedsurfaces. - The unconfirmed-leader gate is checked at request entry, so a request accepted before the freeze appends after it.
- The feed and partition reads serve locally landed commits whether or not they are replicated.
Work, in order:
- Characterise on current main with two-serve SIGSTOP tests: a leader frozen past its lease, and a variant with a commit in flight when it freezes. Record what #4559's unconfirmed-leader refusal already covers.
- Fence at the append (a). Re-check the writer fence under the commit lock in the same step as the local append. A commit whose confirm fails with
writer_fencedis never made visible. - Replicated watermark (b). The feed and partition reads serve only commits up to the replicated watermark.
- May-land settlement (c). A member never applies a 503 may-land commit locally. It settles the commit by reading from the current leader; a re-send under the same txnId is deduplicated.
- Divergence repair (d). A diverged copy (other-commit or behind, from a confirmed leader) re-sends its outbox. It then drops unacknowledged commits the leader doesn't hold, so nobody has to move it aside by hand. Commits this checkout landed, and other-log divergence, stay manual.
Then the targeted recovery above. Only a landed outbox entry clears its dirty names.
Done when (added): run 7's failover rounds show no member copy stuck in serve_upstream_diverged. This gates run 7.
Shipped
Click a lifecycle step above to view its details.
Sign in to post a ripple.