Relationships
#4737 serve: a planned restart parks runs at a step boundary and the new process continues them
Opened by stack72 · 11/14/2026· Shipped 11/16/2026
Part c of #4439, phase 1, step A. The design page "Runs Across Restarts" describes this as option A. It builds on #4693 (swamp PR #3794: safe auto-continue, autoContinue, the continuation claim) and on #4655 and #4656.
Problem
On a planned restart, a rolling one included, serve drains in-flight runs for up to --shutdown-drain-timeout (30 s by default). It then aborts what is left and settles those runs interrupted("server_shutdown"). Any step still running becomes unknown.
With #4693 such a run is continued automatically only when every cut-off step has a guard. Anything else waits for an operator. So a rolling restart in the middle of a deploy workflow still stops the deploy, although the restart was planned and could have waited for a step to finish.
Proposal
Hand runs over at a step boundary.
On SIGTERM
- Serve keeps refusing new runs, as it does today, and also stops starting new steps in runs already going. This covers local steps, worker dispatches, fan-out lanes and new forEach iterations.
- Steps already running finish within the drain timeout, as they do now.
- Each run is then parked for the restart rather than aborted. Its record holds no
unknownstep, and a reason such asrestartsays why it stopped. - This applies to API, webhook and cron runs alike.
- A run already suspended on a gate or a signal is left as it is.
After the restart
- The new process continues parked runs through the continuation path #4693 added, with its claim. It acts only once the old process has gone, the same rule queued-run replay follows. Two processes never run one run.
- Parked runs are always continued: they have no
unknownstep, so nothing is re-run. autoContinue: falsestill keeps a run parked until an operator resumes it.
Drain runs out
- A step still running when the drain timeout ends is interrupted as today, and named.
- #4693 then continues the run when that step has a guard, and otherwise leaves it for
swamp workflow recover.
Attached clients
swamp workflow run --serveralready reattaches for up to 120 s after a restart. Today the new process answersnot_found.- The new process should answer with the continued run, so the client keeps streaming it and sees it finish.
- The dashboard shows the run as parked, then continuing, never as interrupted.
Nested and fan-out runs
- A parent waiting on a nested run is parked with its child, and both are continued.
- A fan-out job keeps its frozen plan, so lanes that finished are kept.
Team mode
- A follower forwards; only the instance that owns the run parks it.
- On a leader change the new leader continues parked runs through the path #4693 uses on promotion.
Out of scope
- Worker steps surviving a restart, meaning a step still running past the drain (option C).
- Any change to step semantics: no step is re-run that would not be re-run today.
Tests
- Rolling restart (new process started first): a run mid-workflow is parked when the old process drains, and is continued by the new process once the old one exits. No step runs twice, and the run id is unchanged.
- Stop then start: the same, with the old process gone before the new one starts.
- Step outlasts the drain: that step is interrupted and named; a guarded step is continued by #4693, an unguarded one is left interrupted.
- Webhook and cron runs: parked and continued like API runs.
- Attached client: a client streaming a run sees it continue after the restart and gets the final status.
- Nested runs: a parent with a running child is parked and continued.
- Fan-out: a run in a fan-out job is parked and continued, and finished lanes are not re-run.
autoContinue: false: the run stays parked until resumed by hand.- Team mode: a parked run is continued by the new leader.
Shipped
Click a lifecycle step above to view its details.
Sign in to post a ripple.