Skip to main content
← Back to list
01Issue
BugShippedSwamp CLIPublic
Assigneesstack72

Relationships

↔ sibling #4660

#4656 serve: a webhook or cron run aborted by shutdown ends cancelled instead of interrupted, so it cannot be recovered

Opened by stack72 · 11/11/2026· Shipped 11/12/2026

Found while designing part c of #4439 (runs survive a restart).

What happens

When serve shuts down, runShutdownDrain (src/serve/shutdown_drain.ts) aborts every run still going after the drain timeout plus the abort grace. The runs then settle in two different ways:

  • API workflow runs: interruptShutdownRuns (src/serve/shutdown_interrupt.ts) marks them interrupted("server_shutdown"). Their in-flight steps become unknown, so swamp workflow recover and swamp workflow resume --run can continue them.
  • Webhook and cron runs: they are aborted by their own services (WebhookService.stop, ScheduledExecutionService.stop). Shutdown never interrupts them (it handles only workflow-run and workflow-resume kinds), so they end cancelled. design/primitives/serve.md, Rolling restart, documents this.

A cancelled run cannot be recovered. So a webhook delivery or a scheduled run that a restart cuts off is lost, while an API run cut off the same way is recoverable. For an operator this depends on how the run started, which they should not need to know. A rolling restart that hits a webhook-triggered deploy leaves nothing to continue.

Expected

A webhook or cron run aborted by shutdown settles the way an API run does: interrupted("server_shutdown"), with abort-stopped steps unknown, its tracker row settled interrupted, and the write fence applied. It is then recoverable, and part c can continue it automatically. A run cancelled on purpose (a user cancel, or a webhook cancel request) stays cancelled.

Update the serve design doc's Rolling restart section to match.

Test

A webhook run and a cron run each stopped mid-step by shutdown end interrupted with reason server_shutdown. swamp workflow recover and resume --run then continue each one. A run cancelled by a user before shutdown is still cancelled.

02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 6 MOREREVIEW+ 10 MOREPR_LINKED+ 2 MORESESSION_SUMMARIZED

Shipped

11/12/2026, 9:16:09 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack7211/11/2026, 6:45:49 PM
stack72 linked sibling of #466011/11/2026, 7:11:32 PM

Sign in to post a ripple.