Skip to main content
← Back to list
01Issue
FeatureShippedSwamp CLIPublic
Assigneesstack72

Relationships

#4737 serve: a planned restart parks runs at a step boundary and the new process continues them

Opened by stack72 · 11/14/2026· Shipped 11/16/2026

Part c of #4439, phase 1, step A. The design page "Runs Across Restarts" describes this as option A. It builds on #4693 (swamp PR #3794: safe auto-continue, autoContinue, the continuation claim) and on #4655 and #4656.

Problem

On a planned restart, a rolling one included, serve drains in-flight runs for up to --shutdown-drain-timeout (30 s by default). It then aborts what is left and settles those runs interrupted("server_shutdown"). Any step still running becomes unknown.

With #4693 such a run is continued automatically only when every cut-off step has a guard. Anything else waits for an operator. So a rolling restart in the middle of a deploy workflow still stops the deploy, although the restart was planned and could have waited for a step to finish.

Proposal

Hand runs over at a step boundary.

On SIGTERM

  • Serve keeps refusing new runs, as it does today, and also stops starting new steps in runs already going. This covers local steps, worker dispatches, fan-out lanes and new forEach iterations.
  • Steps already running finish within the drain timeout, as they do now.
  • Each run is then parked for the restart rather than aborted. Its record holds no unknown step, and a reason such as restart says why it stopped.
  • This applies to API, webhook and cron runs alike.
  • A run already suspended on a gate or a signal is left as it is.

After the restart

  • The new process continues parked runs through the continuation path #4693 added, with its claim. It acts only once the old process has gone, the same rule queued-run replay follows. Two processes never run one run.
  • Parked runs are always continued: they have no unknown step, so nothing is re-run.
  • autoContinue: false still keeps a run parked until an operator resumes it.

Drain runs out

  • A step still running when the drain timeout ends is interrupted as today, and named.
  • #4693 then continues the run when that step has a guard, and otherwise leaves it for swamp workflow recover.

Attached clients

  • swamp workflow run --server already reattaches for up to 120 s after a restart. Today the new process answers not_found.
  • The new process should answer with the continued run, so the client keeps streaming it and sees it finish.
  • The dashboard shows the run as parked, then continuing, never as interrupted.

Nested and fan-out runs

  • A parent waiting on a nested run is parked with its child, and both are continued.
  • A fan-out job keeps its frozen plan, so lanes that finished are kept.

Team mode

  • A follower forwards; only the instance that owns the run parks it.
  • On a leader change the new leader continues parked runs through the path #4693 uses on promotion.

Out of scope

  • Worker steps surviving a restart, meaning a step still running past the drain (option C).
  • Any change to step semantics: no step is re-run that would not be re-run today.

Tests

  • Rolling restart (new process started first): a run mid-workflow is parked when the old process drains, and is continued by the new process once the old one exits. No step runs twice, and the run id is unchanged.
  • Stop then start: the same, with the old process gone before the new one starts.
  • Step outlasts the drain: that step is interrupted and named; a guarded step is continued by #4693, an unguarded one is left interrupted.
  • Webhook and cron runs: parked and continued like API runs.
  • Attached client: a client streaming a run sees it continue after the restart and gets the final status.
  • Nested runs: a parent with a running child is parked and continued.
  • Fan-out: a run in a fan-out job is parked and continued, and finished lanes are not re-run.
  • autoContinue: false: the run stays parked until resumed by hand.
  • Team mode: a parked run is continued by the new leader.
02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 5 MOREREVIEW+ 31 MOREPR_LINKED+ 2 MORESESSION_SUMMARIZED

Shipped

11/16/2026, 7:22:10 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack7211/16/2026, 1:29:27 PM

Sign in to post a ripple.