Relationships
#4679 Team mode failover: one member stalls for minutes in most rounds, takeover runs past 60 s, and a serve outage exits slowly with exit 1 (run 7)
Opened by stack72 · 11/12/2026· Shipped 11/16/2026
Found in team-mode dogfood run 7 (swamp-club#4374, release 20261110.234711.0-sha.3432b140, GCS, 45 failover rounds plus a 12-hour soak). Follows swamp-club#4559. Correctness held (0 acknowledged writes lost, 0 diverged copies across 51 rounds; F43 fixed), but failover availability does not: in most rounds one member is stuck for minutes, takeovers run past 60 s, and a serve outage now exits slowly with the wrong code.
Runs alongside swamp-club#4677 (recovery cost), which likely causes part of these stalls. Step 1 below is analysis only, no code: attribute every stall, hand the recovery part to #4677, and fix only the rest. Don't change serve's takeover or boot-recovery path (v3_repo_context.ts, interrupted-writer recovery) until #4677's changes there have landed, or coordinate with that issue first.
Numbers (run 7, part D and F)
| Set | Takeover ≤ 60 s | Rounds with both member pauses ≤ 15 s | Worst pause |
|---|---|---|---|
c05 kill, KillMode=mixed (10) |
3 / 10 (51–66 s) | 8 / 10 | 100 s |
c06 SIGSTOP 90 s, mixed (10) |
— | 3 / 10 | 182 s |
c06 with a write in flight, mixed (5) |
— | 4 / 5 | 39 s |
c05 kill, default (10) |
1 / 10 (63–73 s, plus 303 s and 458 s) | 2 / 10 | 392 s |
c06 SIGSTOP, default (10) |
— | 3 / 10 | 286 s |
- F24: the pattern is one member stalling 40–392 s while the other recovers in 4–13 s.
- F51: takeover over 60 s in most kill rounds.
- F50 (regression): the c09 outage check took 106 s and exited 1 ("could not commit the start of step s2 … this command's lease on …"); run 5 was about 12 s and exit 75.
- F53: serve-driven (
--server)nightly-reportsucceeded only 78.8% over the soak:serve_not_leader9,workflow_execution_failed8, 72 with no code, around failover rounds. - F44 / F22: serve-a's graceful stops fail or time out at resets; 8 "could not confirm" lines at a restart.
- F12: 2 follower-wedge lines per node in the serve scan.
- F14: follower
AbortError31 and 30 over the soak. - One far K2a on the release got
serve_not_leaderwithinFlighttrue and no leader change. - Hourly soak success dips below 93% on every member, lining up with failover rounds.
From PR #3741's failover rounds (its shared-connection change, Fix B, was dropped and moves here): requests pinned to a frozen old leader wait the full 20 s client timeout; follower forwards to a frozen leader take 13–17 s per attempt (503 "forwarded to a leader that changed"); far "follower behind" retries take 15–34 s over 7–17 attempts; the first request after a takeover takes 3–5.7 s.
Evidence
- The #4374 session's run 7 final report and #3741 failover-round write-up (ask it for
4374-run7-final-report.mdand4374-run7-3741-failover-rounds.md). - Controller VM:
~/dogfood/results/e/<round>/(51 round directories: result, divergence, journals, member and servess, member and serve spans),results.jsonl,orchestrator.log,f4631-analysis.json. - Members:
~/load/ops.jsonl,~/load/fail/,/var/lib/otel/spans.jsonl. Serves: journals and spans.
Work
- Attribute first, no code. For every member pause over 15 s and every takeover over 60 s in run 7, break the time into: member or serve recovery (
readmodel.recoverspans), waiting on a dead or pinned connection (client timeout), follower forwarding and retries, lock waits, and anything else. Report the split per round. Recovery time goes to #4677. - Fix what's left, which may include:
- members leaving a dead or frozen leader's connection after one timeout, not one per client or per attempt; reconnecting through the load balancer to the new leader quickly; and the far connection reuse that #3741's Fix B attempted, without a dead shared connection costing every client a timeout;
- follower forwarding to a frozen leader: fail fast and redirect rather than 13–17 s per attempt;
- takeover time: where the time goes between the lease lapse and "This serve node leads";
- F50: an outage mid-step exits within 45 s with exit 75 and releases the writer, as in run 5;
- F53: serve-driven runs ride out a failover the way member runs do;
- F44 / F22: a graceful stop of either node completes cleanly.
- Fencing and correctness unchanged: F43's fence at the append, #4559's unconfirmed-leader refusal, lease exclusion.
Tests
- Each fix red on current main where it can be reproduced in-process (the two-serve harness from #4581,
integration/serve_v3_team_fenced_leader_test.ts, and the failover repro from #4559). - A test that several clients in one command on a frozen connection pay at most one timeout between them.
- F50: the outage path exits 75 within the deadline.
Done when
Measured on the dogfood VMs with a branch build, in a short failover set (at least 3 c05 and 3 c06 rounds under each KillMode), not only locally:
- no member pause over 15 s;
- takeover ≤ 60 s;
- c09 outage exits within 45 s with exit 75;
- serve-driven runs succeed through the rounds;
- graceful stops of both nodes are clean;
- 0 acknowledged writes lost, 0 diverged copies.
Must not regress
Run 7's correctness results (0 lost, 0 diverged, F43 holds), and the speed gains from #4574 and #4631.
Shipped
Click a lifecycle step above to view its details.
Sign in to post a ripple.