Skip to main content
← Back to list
01Issue
BugShippedSwamp CLIPublic
Assigneeszack

Relationships

#4883 serve SIGTERM skips the failed-gated cleanup job of a run the user cancelled before the shutdown

Opened by zack · 11/18/2026· Shipped 11/18/2026

Description

When a user cancels a run and serve then gets SIGTERM while the run's cleanup is still going, serve skips the run's failed-gated cleanup job. The run still ends cancelled as if its cleanup had finished, and serve logs no cut-off.

The shutdown drain is meant to settle a run the user cancelled first, never interrupt it, and wait for its cleanup within the cleanup grace (src/serve/shutdown_drain.ts:49, 163-166). The same cancel with no SIGTERM runs the cleanup job.

Steps to reproduce

  1. Use a workflow where job A has a long work step and an in-job always cleanup step that takes 8 s. Job B dependsOn: A with condition: failed and writes a marker.
  2. Start serve: swamp serve --no-schedule --shutdown-drain-timeout 3s.
  3. Run swamp workflow run <wf> --server <url> --detach.
  4. While work runs, cancel it with swamp workflow cancel <run> --server <url>. The reply is cancellation_requested.
  5. About 4 s into A's cleanup, send serve SIGTERM.

Expected

A's cleanup finishes, then job B starts and finishes, the same as with no SIGTERM. The run ends cancelled with B succeeded. If the cleanup grace ran out, serve would log Saved cancelled workflow run …: shutdown cut off its cleanup.

Actual

  • A's cleanup finishes and serve exits 0 about 3.4 s after SIGTERM, which is as soon as A's cleanup ends.
  • Job B never starts: there is no started marker.
  • History shows B failed and B/on-failure failed: cancelled. The run ends cancelled.
  • The roughly 20 s cleanup grace goes unused, and no cut-off is logged.

Control, with the same cancel and no SIGTERM: B starts and succeeds.

Root cause (from source)

The SIGTERM handover added by swamp-club#4737 also reaches the cleanup jobs of a run the user cancelled before the shutdown.

  • src/domain/workflows/execution_service.ts around line 8718: cleanupStepOpts = { ...stepOpts, signal: cleanupSignal, cleanupJobLevel: true } keeps stepOpts.handover.
  • markStarted (around lines 3388-3398) returns without starting the job when handedOver(handover).
  • runShutdownDrain correctly treats the run as cancelled-before-shutdown and would wait stopGraceFor, but the run has already settled by then.

A user-cancelled run's cleanup jobs should ignore the shutdown handover and run under the cleanup-grace signal, as they do on a cancel with no shutdown.

Environment

  • swamp built from main 8b8e14c6, Linux.
  • Found by the swamp-uat test tests/cli/serve/none/shutdown_cleanup_test.ts: "swamp serve SIGTERM lets a run cancelled before the shutdown finish its cleanup, and leaves it cancelled, not interrupted" (swamp-uat#976).
02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 5 MOREREVIEW+ 8 MOREPR_LINKED+ 2 MORESESSION_SUMMARIZED

Shipped

11/18/2026, 9:26:27 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
zack assigned zack11/18/2026, 8:53:05 PM

Sign in to post a ripple.