Relationships
#4559 Team mode: after a serve failover a member's writes stall 1-7 minutes, and takeover takes ~8 minutes under the default KillMode (F24, F14 from #4374)
Opened by stack72 · 11/9/2026· Shipped 11/10/2026
Found by the team-mode dogfood on real GCS (swamp-club#4374), recurring in every run since run 2. A ship blocker: after a serve failover, a member's writes stall for minutes, and under systemd's default KillMode the follower takes about 8 minutes to take over. No acknowledged write has been lost; the cost is availability.
Setup
- Two serve nodes (serve-a, serve-b) in different zones, behind an internal TCP load balancer, with TLS from a private CA, token auth,
managedConfigon, and the v3 datastore on GCS claimed by serve. - Members write through the load balancer's address.
- c05: SIGKILL the leader and hold it down until the follower leads.
- c06: SIGSTOP the leader for 90 s.
- Two write probes per round measure each member's write pause.
What happens
| Run | Release | Result |
|---|---|---|
| 2 | 3ff9a703 |
one member 190 s once (pause round) |
| 3 | 20261105.030003.0 |
131–199 s, 3 times |
| 4 | 20261106.094050.0-sha.fcdd7e60 |
7 of 18 rounds: kill 60–65 s (once both members), pause 143–223 s; the other member resumed in about 5 s |
| 5 | 20261108.111402.0-sha.37c27525 |
member-1 stalled 145–407 s in 13 of 20 rounds; under the default KillMode, the follower took 467–492 s to take over |
- The normal failover pause is 4–7 s, the same as on the emulator, so this is the outlier, but a frequent one.
- Worse after SIGSTOP than after SIGKILL.
- Run 5 caveat: some of member-1's default-
KillModekill rounds are void because of a dogfood harness bug (a rotated-out token). The serve-side takeover time of 467–492 s is measured on serve and stands.
F14, folded in here: followers logged AbortError on forwarded requests, 19 times on serve-a and 29 on serve-b in run 4. Nobody has lined them up against failover rounds or member stalls.
Evidence:
- the dogfood controller's
~/dogfood/results/(per-round, per-member pauses with timestamps); - both serves' journals;
- members'
~/load/ops.jsonl; - run 4's handoff:
~/Desktop/4374-run4-architecture-handoff.md, section 5.2.
Hypotheses (unverified)
- Members wait on connections to the frozen or dead leader.
- A member reuses keep-alive connections through the TCP load balancer; since #4525, locks use one kept client per process.
- A frozen node keeps its connections open but never answers, so a request on such a connection waits out the full per-request timeout (30 s). Retries may reuse it, or be routed back to the same node until the load balancer's health check drops it.
- Two or three timeouts gives 60–90 s, more with a lease wait on top. It fits "worse after SIGSTOP than SIGKILL".
- A slow takeover under the default
KillMode.- Killing serve's process (not
systemctl stop) may leave its detached Litestream supervisor and Litestream running (#4432's analysis: the supervisor has its own process group). - The follower may then wait on something the orphaned sidecar holds or keeps fresh, until an idle timeout.
- The difference between
KillModesettings points at process-tree state, not the network.
- Killing serve's process (not
- Lease adoption after a takeover: the new leader adopting or expiring leases the old leader held may add its own wait.
Work
Find the cause from the round data first. For each stalled round, line up: the member's request spans (which request waited, on which connection or node, with which timeout), the follower's takeover timeline, the old leader's sidecar processes, and F14's aborts.
Fix what it shows. If hypothesis 1 holds:
- member requests to serve get a short connect and first-response deadline;
- a timed-out request drops its pooled connection, so the retry opens a fresh one;
- the kept lock client does the same.
If hypothesis 2 holds: the follower's takeover doesn't depend on the old leader's sidecar exiting, or a dead leader's sidecar exits promptly whatever the
KillMode.Classify F14: each abort is either expected (the leader killed mid-forward) or fixed.
Tests
- A frozen leader: in a local two-node setup with a TCP proxy standing in for the load balancer, SIGSTOP the leader. Every member's next write lands within the failover bar, and none waits out a full request timeout. Red on current main if hypothesis 1 holds.
- A killed leader under the default
KillMode: the follower leads within about 10 s, in a systemd container or an equivalent process-tree setup.
Must not regress
- No acknowledged write lost, and 0
serve_upstream_diverged. - The usual failover write pause of 4–7 s.
- #4431's outage behaviour: an unreachable serve ends the command within seconds, with exit 75.
Done when
The local tests pass, and the next dogfood run's failover rounds show no member write pause over 30 s and takeover within about 10 s: 10 rounds each of kill and pause, under each KillMode.
Shipped
Click a lifecycle step above to view its details.
Sign in to post a ripple.