Skip to main content
← Back to list
01Issue
BugOpenSwamp CLIPublic
AssigneesNone

Relationships

#3865 serve shutdown does not wait for worker status writes, and its listener accepts workers after the gateway is disposed

Opened by zack · 10/23/2026

Description

During swamp serve shutdown, the worker gateway's queued transitions (worker status writes such as set_status: disconnected, token expiry records, enrollment verify results) are not awaited. src/cli/commands/serve.ts calls workerGateway.dispose() (synchronous, it only stops session timers) and then stops pollers, v3 replication and the audit flush, and only then calls ac.abort() to stop the HTTP server. Worker sockets stay open until that abort, so:

  • A worker that disconnects during shutdown has its disconnected record queued on the gateway's TransitionTail with nothing awaiting it; the write can land after v3 replication stopped, or be lost at Deno.exit.
  • Grace and token-expiry timers stay armed after dispose.
  • A worker can enroll (redeem its token, record its status) while serve shuts down.

Boot reconciliation (src/serve/boot_reconciliation.ts) repairs stale worker records on the next start, so the impact is stale swamp worker list output until then, and records written outside the closing sync.

What was tried in swamp-club#3854 and why it was backed out

#3854 (test temp-dir leaks) tried a WorkerGateway.close() that awaited the transition tail with a deadline. Three review rounds found a new gap each time, all from the same cause: the listener stays open after the gateway closes.

  1. Awaiting in-flight enrollment probes could stall shutdown up to 60s each.
  2. close() ran before worker sockets closed, so their disconnects were still unawaited.
  3. Force-disconnecting workers in close() made them reconnect (1s backoff) and re-enroll while the datastore stopped. Guarding enrollment with a closing flag still left windows after each await inside the enroll transition (after redeem, after the token expiry read), where a worker could be recorded idle after shutdown or arm a stray expiry timer.

Expected

Serve shutdown stops accepting worker connections first, then closes the gateway (clears its timers, refuses enrollment, awaits queued transitions with a deadline), then stops the datastore. The enroll transition should record nothing once closing has begun, including after each of its awaits.

Notes

#3854 keeps WorkerGateway.settled() and TransitionTail.settled(), which await the queue; tests use them after closing the server.

02Bog Flow
◉OPEN○TRIAGED○IN PROGRESS○SHIPPED

Open

10/23/2026, 8:43:55 PM

No activity in this phase yet.

03Sludge Pulse

Sign in to post a ripple.