Skip to main content
← Back to list
01Issue
FeatureShippedSwamp CLIPublic
Assigneesstack72

Relationships

#4566 swamp serve: restarts without failed requests (close the listener early, overlap restarts, clients and workers ride out a restart)

Opened by stack72 · 11/9/2026· Shipped 11/10/2026

Part of swamp-club#4439 (the rolling-restart half, phase 1). It is filed separately because #4439 also covers a full config reload and runs surviving restarts, which this does not.

Problem

Restarting swamp serve (to upgrade it, or for a setting that needs a new process) causes failed requests, even behind a load balancer:

  • The listener stays open through the whole drain. On SIGTERM, /ready turns 503, but the Deno.serve listener only closes at the very end of shutdown (src/cli/commands/serve.ts, ac.abort() after the drain). New connections keep arriving at a process that is going away.
  • / and /health keep answering 200 "ok" while draining. They ignore the shutdown state, so a load balancer that checks them keeps routing to the draining node.
  • Open WebSockets get no close frame. They drop when the process exits.
  • Nothing listens while the new process starts. The port only opens after lease, migrations, hydration (up to 60 s) and stale-run reaping, so callers get "connection refused" for that whole window. There is no way to start the new process before the old one stops.
  • Startup serves requests before it is ready. Remote reconciliation, pending-run replay, the heartbeat and the GC services start after the port opens, while /ready still says 503. design/primitives/serve.md says startup finishes before traffic is accepted.
  • Clients give up quickly. swamp ... --server does not retry a refused connection or a draining reply before it has a run id, and its reattach budget is about 15 s (src/cli/remote_run.ts). packages/client has no retry at all.
  • Workers give up quickly. A worker exits after 3 failed connects before enrolment (MAX_PRE_ENROLL_FAILURES in src/worker/connect.ts), roughly 3 s of serve downtime, and waits for its service manager to restart it.

Proposed solution

  1. Close the listener early. Shortly after /ready turns 503 (a short grace period so load balancers notice), stop accepting new connections with the server's graceful shutdown, then drain. / and /health report draining. Open WebSocket sessions get a 1001 "going away" close.
  2. Finish startup before serving run requests. Requests that start or resume runs answer 503 with Retry-After until boot completes, matching serve.md.
  3. Overlapping restarts. An opt-in reusePort setting on the listener (Linux; documented as not load-balanced on macOS), so a new process can start and become ready before the old one is stopped. Off by default.
  4. Clients ride out a restart. The CLI's --server path and packages/client retry a refused connection, a 503 and a draining reply with backoff and jitter, for a bounded total time, before a run id exists; the reattach budget grows to cover a typical restart. A draining reply carries retryAfter.
  5. Workers ride out a restart. Pre-enrolment connection failures use a time budget (for example a few minutes) instead of a count, so a worker waits for serve rather than exiting.

Acceptance

  • Two serve processes behind a local proxy, restarted one at a time while clients and webhook senders keep sending: no failed requests and no "connection refused", where the current release shows failures.
  • A single serve restarted with reusePort (Linux): no failed requests.
  • A worker connected during a serve restart reconnects without exiting.

Out of scope

  • Reloading other settings without a restart (#4439, unified reload).
  • Resuming runs interrupted by the restart, worker dispatches surviving it, protocol version ranges (#4439, later part).
  • Rolling upgrades on a v3 datastore (needs datastore Phase 5 team mode).
  • Socket handover without a proxy (no file-descriptor adoption API in Deno).

Notes

  • packages/client changed recently (delegated refresh, how runs settle); the retry work should build on that behaviour.
02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 5 MOREREVIEW+ 18 MOREPR_LINKED+ 2 MORESESSION_SUMMARIZED

Shipped

11/10/2026, 7:23:21 AM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack7211/9/2026, 11:00:25 PM

Sign in to post a ripple.