Skip to main content
← Back to list
01Issue
BugShippedSwamp CLIPublic
Assigneesstack72

Relationships

#4679 Team mode failover: one member stalls for minutes in most rounds, takeover runs past 60 s, and a serve outage exits slowly with exit 1 (run 7)

Opened by stack72 · 11/12/2026· Shipped 11/16/2026

Found in team-mode dogfood run 7 (swamp-club#4374, release 20261110.234711.0-sha.3432b140, GCS, 45 failover rounds plus a 12-hour soak). Follows swamp-club#4559. Correctness held (0 acknowledged writes lost, 0 diverged copies across 51 rounds; F43 fixed), but failover availability does not: in most rounds one member is stuck for minutes, takeovers run past 60 s, and a serve outage now exits slowly with the wrong code.

Runs alongside swamp-club#4677 (recovery cost), which likely causes part of these stalls. Step 1 below is analysis only, no code: attribute every stall, hand the recovery part to #4677, and fix only the rest. Don't change serve's takeover or boot-recovery path (v3_repo_context.ts, interrupted-writer recovery) until #4677's changes there have landed, or coordinate with that issue first.

Numbers (run 7, part D and F)

Set Takeover ≤ 60 s Rounds with both member pauses ≤ 15 s Worst pause
c05 kill, KillMode=mixed (10) 3 / 10 (51–66 s) 8 / 10 100 s
c06 SIGSTOP 90 s, mixed (10) — 3 / 10 182 s
c06 with a write in flight, mixed (5) — 4 / 5 39 s
c05 kill, default (10) 1 / 10 (63–73 s, plus 303 s and 458 s) 2 / 10 392 s
c06 SIGSTOP, default (10) — 3 / 10 286 s
  • F24: the pattern is one member stalling 40–392 s while the other recovers in 4–13 s.
  • F51: takeover over 60 s in most kill rounds.
  • F50 (regression): the c09 outage check took 106 s and exited 1 ("could not commit the start of step s2 … this command's lease on …"); run 5 was about 12 s and exit 75.
  • F53: serve-driven (--server) nightly-report succeeded only 78.8% over the soak: serve_not_leader 9, workflow_execution_failed 8, 72 with no code, around failover rounds.
  • F44 / F22: serve-a's graceful stops fail or time out at resets; 8 "could not confirm" lines at a restart.
  • F12: 2 follower-wedge lines per node in the serve scan.
  • F14: follower AbortError 31 and 30 over the soak.
  • One far K2a on the release got serve_not_leader with inFlight true and no leader change.
  • Hourly soak success dips below 93% on every member, lining up with failover rounds.

From PR #3741's failover rounds (its shared-connection change, Fix B, was dropped and moves here): requests pinned to a frozen old leader wait the full 20 s client timeout; follower forwards to a frozen leader take 13–17 s per attempt (503 "forwarded to a leader that changed"); far "follower behind" retries take 15–34 s over 7–17 attempts; the first request after a takeover takes 3–5.7 s.

Evidence

  • The #4374 session's run 7 final report and #3741 failover-round write-up (ask it for 4374-run7-final-report.md and 4374-run7-3741-failover-rounds.md).
  • Controller VM: ~/dogfood/results/e/<round>/ (51 round directories: result, divergence, journals, member and serve ss, member and serve spans), results.jsonl, orchestrator.log, f4631-analysis.json.
  • Members: ~/load/ops.jsonl, ~/load/fail/, /var/lib/otel/spans.jsonl. Serves: journals and spans.

Work

  1. Attribute first, no code. For every member pause over 15 s and every takeover over 60 s in run 7, break the time into: member or serve recovery (readmodel.recover spans), waiting on a dead or pinned connection (client timeout), follower forwarding and retries, lock waits, and anything else. Report the split per round. Recovery time goes to #4677.
  2. Fix what's left, which may include:
    • members leaving a dead or frozen leader's connection after one timeout, not one per client or per attempt; reconnecting through the load balancer to the new leader quickly; and the far connection reuse that #3741's Fix B attempted, without a dead shared connection costing every client a timeout;
    • follower forwarding to a frozen leader: fail fast and redirect rather than 13–17 s per attempt;
    • takeover time: where the time goes between the lease lapse and "This serve node leads";
    • F50: an outage mid-step exits within 45 s with exit 75 and releases the writer, as in run 5;
    • F53: serve-driven runs ride out a failover the way member runs do;
    • F44 / F22: a graceful stop of either node completes cleanly.
  3. Fencing and correctness unchanged: F43's fence at the append, #4559's unconfirmed-leader refusal, lease exclusion.

Tests

  • Each fix red on current main where it can be reproduced in-process (the two-serve harness from #4581, integration/serve_v3_team_fenced_leader_test.ts, and the failover repro from #4559).
  • A test that several clients in one command on a frozen connection pay at most one timeout between them.
  • F50: the outage path exits 75 within the deadline.

Done when

Measured on the dogfood VMs with a branch build, in a short failover set (at least 3 c05 and 3 c06 rounds under each KillMode), not only locally:

  • no member pause over 15 s;
  • takeover ≤ 60 s;
  • c09 outage exits within 45 s with exit 75;
  • serve-driven runs succeed through the rounds;
  • graceful stops of both nodes are clean;
  • 0 acknowledged writes lost, 0 diverged copies.

Must not regress

Run 7's correctness results (0 lost, 0 diverged, F43 holds), and the speed gains from #4574 and #4631.

02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 15 MOREFINDINGS+ 62 MOREPR_LINKED+ 2 MORESESSION_SUMMARIZED

Shipped

11/16/2026, 3:52:07 AM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack7211/12/2026, 5:43:24 PM

Sign in to post a ripple.