Relationships
↔ sibling #4577#4900 Docs: team serve rolling restart runbook, leader takeover time, and how clients treat an inFlight refusal (#4811, #4824, #4830, #4839, #4853)
Opened by stack72 · 11/19/2026· Shipped 11/19/2026
Three shipped swamp serve changes from the #4439 reload and restart follow-ups have no docs issue, and the manual under content/manual/ (swamp-club repo) does not cover them. Docs only. The open docs issues #4577, #4794 and #4829 touch the same restart pages, so these fit in the same docs PR.
1. Restarting a team serve one host at a time (#4811)
The manual never mentions a rolling restart. #4811 asked for an operator runbook:
- Restart followers first, one at a time. A follower owns no runs, so nothing is parked; its connections move to the other hosts behind the load balancer.
- Restart the leader last. It parks its runs at a step boundary (#4737) and gives up the writer lease; a follower is promoted and continues the parked runs under the same run ids.
- What to watch between hosts: wait for the restarted host's
/readybefore moving on, and use/ready, not/health, as the load balancer check (see #4577).
Pages: how-to/swamp-serve/run-multi-instance-deployment.md (a rolling restart section), linked from how-to/swamp-serve.md. #4325 (team mode manual pages) covers the same pages more broadly.
2. How long a leader takeover takes (#4824)
The writer lease now has a 15 s TTL, so after a leader dies a follower takes over in about 25 s, down from about 40 s. A planned leader restart hands the lease over without waiting for it to lapse. The multi-instance page explains takeover but gives no time; say how long writes and run starts are refused while a dead leader's lease lapses, and that a member's CLI rides it out with its read retries (about 15 s) and run-start retries (30 s).
Page: how-to/swamp-serve/run-multi-instance-deployment.md.
3. What a client does with an inFlight refusal (#4830, #4839, #4853)
The multi-instance page explains a follower's serve_not_leader with inFlight: true (the request may already have run). Add:
swamp ... --serverexits 1 for aninFlightrefusal, not 75: exit 75 means safe to retry, and this request may have run. A refusal withoutinFlightstill exits 75.- An HTTP action (
build-an-integration.md) answers aninFlightrefusal asoutcome_unknown(503), never as a retryableserve_not_leader.@swamp-club/swamp-libnever resends it; read the run's status before retrying. - A leader marks a not-leader refusal raised after a request started as
inFlighttoo, with a message that it lost the lead while handling the request. - A follower that names no leader (the data plane) still says the request may already have run for an
inFlightrefusal, never try again.
Pages: how-to/swamp-serve/run-multi-instance-deployment.md, how-to/swamp-serve/build-an-integration.md (the error table), and wherever the manual lists exit 75.
Shipped
Click a lifecycle step above to view its details.
system commented 11/19/2026, 4:57:59 PM
Classified automatically when this issue was filed.
- Source: Swamp Club
If you feel this classification is incorrect, add a ripple to tell us so.
Sign in to post a ripple.