Relationships
#4629 serve shutdown: /ready reports shutting_down while the webhook route still queues new deliveries
Opened by zack · 11/11/2026· Shipped 11/11/2026
Summary
During swamp serve shutdown, GET /ready can return 503 shutting_down while the webhook route is still accepting deliveries. A delivery in that window gets 200 {"status":"queued"}. This instance never runs it: the drain drops it from the queue, and it stays pending until the next swamp serve on the repo replays it at boot. The sender got a 200, so it won't retry against the replacement.
Seen once in swamp-uat CI, run 42928364344 (job "Serve / Serve: No Auth"), build 20261110.235929.0-sha.7617a016. The failing test is swamp serve on SIGTERM drains an in-flight webhook run and turns new deliveries away in tests/cli/serve/none/lifecycle_test.ts. It expected 503 with Retry-After for a delivery sent after /ready reported shutting_down, and got 200 queued.
Root cause
/ready and the webhook route read two different flags, and shutdown sets them at different times (line numbers at main 27a966d84):
/readyreadsshuttingDown(src/cli/commands/serve.ts:9332). The shutdown handler sets it with no await before it (serve.ts:9431).- The webhook route reads
WebhookService.draining(src/serve/webhook.ts:875and:1013). Onlydrain()sets it (webhook.ts:1221), and that is called fromrunShutdownDrain(serve.ts:9475).
Between the two, shutdown awaits twice:
await removePidFileIfOurs(...)(serve.ts:9435): three async file operations. This runs whenever stdin is not a terminal, which is always the case in UAT.await stages.run("pending replay sweep", ...)(serve.ts:9474): at least one microtask hop, and the whole pass if a replay pass (every 5s) is running.
Interleaving: SIGTERM, then shuttingDown = true, then shutdown yields at the pid-file await. /ready returns 503. The delivery passes the entry check, reads the body and verifies the signature, and passes the second check. It is recorded as pending and queued behind the in-flight run, and the response is 200. Later drain() sets draining and drops it from the queue.
The gap came from b1f3b765b (#3706), which added the sweep-stop await before the drain, and 7671fc362 (#3716), which added the pid-file await. Before b1f3b765b, drain() ran in the same turn as shuttingDown = true.
Other routes
- WebSocket run start and resume check
admission.refusal(). Since 7671fc362,admission.beginDraining()comes after the pid-file await, so these have the same gap, but shorter. A run accepted there is still waited on by the drain, so it isn't lost. ScheduledExecution.drainingis also only set indrain(). A cron fire in the gap starts a run, which the drain then waits for. Nothing outside can observe this.
Proposed fix
- Call
admission.beginDraining()right aftershuttingDown = true, before the pid-file await. - Add a
shuttingDown?: () => booleandependency toWebhookService, wired as() => admission.phase === "draining". - Make both webhook checks refuse when
this.draining || this.deps.shuttingDown?.() === true.
/ready, WebSocket admission and webhook intake then all switch in the same turn. drain(), the queue, replay and the shutdown step order are unchanged.
Reproducing unit tests (src/serve/webhook_test.ts)
- One run is in flight. Call
admission.beginDraining()without callingdrain(), as serve does during the awaits. A signed delivery must get 503 withretry-after: 5and record no pending entry. Today it gets 200. - The
shuttingDowngate returns false on the entry check and true after verification. The second check must refuse the delivery. Today it gets 200.
Both fail on main and pass with the fix. The fix and tests are written in a local worktree but not committed.
Shipped
Click a lifecycle step above to view its details.
Sign in to post a ripple.