Relationships
↔ sibling #4660#4656 serve: a webhook or cron run aborted by shutdown ends cancelled instead of interrupted, so it cannot be recovered
Opened by stack72 · 11/11/2026· Shipped 11/12/2026
Found while designing part c of #4439 (runs survive a restart).
What happens
When serve shuts down, runShutdownDrain (src/serve/shutdown_drain.ts) aborts every run still going after the drain timeout plus the abort grace. The runs then settle in two different ways:
- API workflow runs:
interruptShutdownRuns(src/serve/shutdown_interrupt.ts) marks theminterrupted("server_shutdown"). Their in-flight steps becomeunknown, soswamp workflow recoverandswamp workflow resume --runcan continue them. - Webhook and cron runs: they are aborted by their own services (
WebhookService.stop,ScheduledExecutionService.stop). Shutdown never interrupts them (it handles onlyworkflow-runandworkflow-resumekinds), so they endcancelled.design/primitives/serve.md, Rolling restart, documents this.
A cancelled run cannot be recovered. So a webhook delivery or a scheduled run that a restart cuts off is lost, while an API run cut off the same way is recoverable. For an operator this depends on how the run started, which they should not need to know. A rolling restart that hits a webhook-triggered deploy leaves nothing to continue.
Expected
A webhook or cron run aborted by shutdown settles the way an API run does: interrupted("server_shutdown"), with abort-stopped steps unknown, its tracker row settled interrupted, and the write fence applied. It is then recoverable, and part c can continue it automatically. A run cancelled on purpose (a user cancel, or a webhook cancel request) stays cancelled.
Update the serve design doc's Rolling restart section to match.
Test
A webhook run and a cron run each stopped mid-step by shutdown end interrupted with reason server_shutdown. swamp workflow recover and resume --run then continue each one. A run cancelled by a user before shutdown is still cancelled.
Shipped
Click a lifecycle step above to view its details.