Skip to main content
← Back to list
01Issue
BugShippedSwamp CLIPublic
Assigneesstack72

Relationships

#4677 Team mode: recoveries still take tens of seconds on members and 6-16 minutes on a serve after a kill (targeted path slow; the leader's dirty set forces full rebuilds)

Opened by stack72 · 11/12/2026· Shipped 11/13/2026

Found in team-mode dogfood run 7 (swamp-club#4374, release 20261110.234711.0-sha.3432b140, GCS, 12-hour soak plus 45 failover rounds). Follows swamp-club#4581, whose targeted recovery measured 0.65 s against 8.4 s in a local benchmark. On the VMs, recoveries are still tens of seconds to minutes for members and 6–16 minutes for a serve restarting after a kill. Correctness held (no failed recovery, no lost write, no diverged copy); this is cost. It is the first fix after run 7 because it likely causes part of run 7's failover stalls too (see "Link to failover").

Numbers (run 7, readmodel.recover spans)

Members (203 spans over the soak):

Member Spans p50 Max > 60 s Paths
member-1 46 70.5 s 536 s 27 interrupted_writer/targeted 24, dropped_work/targeted 15, view_change/full 3, full via scope_too_large 4
member-2 119 59.9 s 271 s 59 dropped_work/targeted 101, interrupted_writer/targeted 11, full via scope_too_large 7
member-far 38 22.1 s 142 s 8 dropped_work/targeted 33, interrupted_writer/targeted 2, view_change/full 2, full via scope_too_large 1
  • Targeted recoveries take 36–63 s even with dirty_names 0–10. So the cost is not the per-name replay #4581 measured, but something around it (candidates below).
  • scope_too_large forced 12 member recoveries to full (40–142 s).

Serve boot recovery of the killed predecessor:

Node Spans p50 Max Paths
serve-a 14 342 s 886 s full via scope_too_large 12, targeted 2
serve-b 19 530 s 982 s full via scope_too_large 17, full via dirty_tracking_recent 1, targeted 1
  • scope_too_large tripped in 29 of 33 serve boot recoveries after failover kills. This is #4581's open risk 2: on a long-running leader, committed names stay dirty until a clean teardown, so the dirty set grows with activity until the quarter-share cap sends the recovery to a full rebuild.
  • At the deliberate restarts (6 h and 12 h kills) the cap did not trip, yet targeted still took 36.4 s (dirty 0, touched 14) and 82.9 s (dirty 10, touched 10).

Login reset (F54): after a revoked token is replaced, the copy resets with a view_change/full rebuild of 536 s.

Run 7's failover rounds show one member stalling 40–392 s in most rounds (F24), and takeovers of 303 s and 458 s. Those windows overlap member recoveries (an interrupted command's next command recovers) and the new or restarting serve's boot recovery. Part of the failover stall is probably this issue. A separate failover issue will follow and will attribute its stalls first; anything that is recovery lands here.

Work

  1. Break down a slow targeted recovery from run 7's spans first (recover.scan, recover.history, recover.restore, recover.reproject, scan_ms, files_checked/listed), before changing code. Candidates:
    • the scan walking the whole checkout (#scanSince, every listed key, lstat per missing key) — cost proportional to files, not to touched names;
    • catching up from the cursor before the scope is built;
    • reading partition history from the base for touched partitions with long histories;
    • the read-model catch-up that follows.
  2. Keep a long-running leader's dirty set small: clear a committed name's dirty mark once its commit is durable (confirmed replicated), not only at clean teardown. Keep ADV-7 (an unlanded outbox entry's names stay pinned). Then scope_too_large should be rare on serve.
  3. Make the full fallback itself cheaper, or bounded, where it must run (serve boot after a kill).
  4. Login reset (F54): a view_change caused only by a credential change for the same datastore should not need a full rebuild, if the copy's lineage still matches. If it must rebuild, it should be bounded the same way.

Tests

  • In-process count tests: a targeted recovery with N dirty names does work bounded by N and their partitions (files touched, partitions read), not by checkout size or history length.
  • A long simulated leader run: the dirty set stays bounded by unlanded work, and a kill-then-boot recovery takes the targeted path.
  • #4581's gate test, property test and mutation checks keep passing; targeted equals full, row for row.

Done when

Measured on the dogfood VMs (GCS), not only locally:

  • member recoveries after an interrupted command: p50 under 5 s, max under 30 s, over a soak with failover rounds;
  • serve boot recovery of a killed predecessor: under 60 s, with scope_too_large rare;
  • a login reset of the same datastore: under 30 s.

Must not regress

  • No failed recovery, no acknowledged write lost, no diverged copy (run 7: all zero).
  • #4512's stoppable, resumable recovery; #4579's refused-read handling; F43's repair.
02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 8 MOREREVIEW+ 11 MOREPR_LINKED+ 2 MORESESSION_SUMMARIZED

Shipped

11/13/2026, 12:23:12 AM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack7211/12/2026, 4:04:59 PM

Sign in to post a ripple.