← Back to list
01Issue
FeatureShippedSwamp CLIPublic
Assigneesstack72
Relationships
#4566 swamp serve: restarts without failed requests (close the listener early, overlap restarts, clients and workers ride out a restart)
Opened by stack72 · 11/9/2026· Shipped 11/10/2026
Part of swamp-club#4439 (the rolling-restart half, phase 1). It is filed separately because #4439 also covers a full config reload and runs surviving restarts, which this does not.
Problem
Restarting swamp serve (to upgrade it, or for a setting that needs a new process) causes failed requests, even behind a load balancer:
- The listener stays open through the whole drain. On SIGTERM,
/readyturns 503, but theDeno.servelistener only closes at the very end of shutdown (src/cli/commands/serve.ts,ac.abort()after the drain). New connections keep arriving at a process that is going away. /and/healthkeep answering 200 "ok" while draining. They ignore the shutdown state, so a load balancer that checks them keeps routing to the draining node.- Open WebSockets get no close frame. They drop when the process exits.
- Nothing listens while the new process starts. The port only opens after lease, migrations, hydration (up to 60 s) and stale-run reaping, so callers get "connection refused" for that whole window. There is no way to start the new process before the old one stops.
- Startup serves requests before it is ready. Remote reconciliation, pending-run replay, the heartbeat and the GC services start after the port opens, while
/readystill says 503.design/primitives/serve.mdsays startup finishes before traffic is accepted. - Clients give up quickly.
swamp ... --serverdoes not retry a refused connection or adrainingreply before it has a run id, and its reattach budget is about 15 s (src/cli/remote_run.ts).packages/clienthas no retry at all. - Workers give up quickly. A worker exits after 3 failed connects before enrolment (
MAX_PRE_ENROLL_FAILURESinsrc/worker/connect.ts), roughly 3 s of serve downtime, and waits for its service manager to restart it.
Proposed solution
- Close the listener early. Shortly after
/readyturns 503 (a short grace period so load balancers notice), stop accepting new connections with the server's graceful shutdown, then drain./and/healthreport draining. Open WebSocket sessions get a 1001 "going away" close. - Finish startup before serving run requests. Requests that start or resume runs answer 503 with
Retry-Afteruntil boot completes, matchingserve.md. - Overlapping restarts. An opt-in
reusePortsetting on the listener (Linux; documented as not load-balanced on macOS), so a new process can start and become ready before the old one is stopped. Off by default. - Clients ride out a restart. The CLI's
--serverpath andpackages/clientretry a refused connection, a 503 and adrainingreply with backoff and jitter, for a bounded total time, before a run id exists; the reattach budget grows to cover a typical restart. Adrainingreply carriesretryAfter. - Workers ride out a restart. Pre-enrolment connection failures use a time budget (for example a few minutes) instead of a count, so a worker waits for serve rather than exiting.
Acceptance
- Two serve processes behind a local proxy, restarted one at a time while clients and webhook senders keep sending: no failed requests and no "connection refused", where the current release shows failures.
- A single serve restarted with
reusePort(Linux): no failed requests. - A worker connected during a serve restart reconnects without exiting.
Out of scope
- Reloading other settings without a restart (#4439, unified reload).
- Resuming runs interrupted by the restart, worker dispatches surviving it, protocol version ranges (#4439, later part).
- Rolling upgrades on a v3 datastore (needs datastore Phase 5 team mode).
- Socket handover without a proxy (no file-descriptor adoption API in Deno).
Notes
packages/clientchanged recently (delegated refresh, how runs settle); the retry work should build on that behaviour.
02Bog Flow
Shipped
Click a lifecycle step above to view its details.
03Sludge Pulse
Sign in to post a ripple.