Skip to main content
← Back to list
01Issue
FeatureShippedSwamp CLIPublic
Assigneesstack72

Relationships

↔ sibling #4703

#4693 serve: continue an interrupted run automatically when every step the restart cut off is safe to run again

Opened by stack72 · 11/12/2026· Shipped 11/14/2026

Part c of #4439, phase 1, step B. Design page: "Runs Across Restarts" (option B). Groundwork shipped in #4655 and #4656.

Problem

When serve stops mid-run (a crash, kill -9, a container restart, or a shutdown whose drain runs out), every running run ends interrupted:

  • server_crash from the boot reaper;
  • remote_instance_dead from reconciliation;
  • owner_process_dead;
  • server_shutdown.

Since #4656 this includes webhook and cron runs. Nothing continues them. The boot log tells the operator to run swamp workflow recover, then swamp workflow resume --run. Serve deliberately never continues a recovered run (awaitsResumeAfterRecovery). WorkflowExecutionService.recover(), which recovers and resumes in one call, exists but has no production caller.

Proposal

Serve continues an interrupted run on its own when that is safe.

When it is safe. Every step the interruption cut off (status unknown) has a guard, or the run has no unknown step at all because it stopped at a step boundary. This is the same rule swamp workflow recover uses to decide whether --acknowledge-unknown is needed (recovery_assessment.ts).

When it runs.

  • At boot, after the reaper settles the previous process's runs.
  • In the reconciliation loop, after it settles a dead instance's runs.
  • On a team leader change, for the old leader's runs, through the same path.

What it does. It recovers the run (resetUnknownStepsForRecovery) and resumes it under the same run id, through the existing recover-and-resume code. Guards decide whether each reset step runs again, and finished steps keep their outputs.

What it refuses, leaving the run interrupted as today:

  • the definition has changed since the run started (fingerprint);
  • a fan-out plan or selector no longer matches;
  • a nested run is unsettled;
  • the workflow opts out (below);
  • an unknown step has no guard.

When it refuses, the boot or reconciliation log, and the run, name the step or reason, so the operator knows what to decide with swamp workflow recover.

Who continues it. Exactly one process. Claim the run before acting, the way continuation claims and pending-run replay already do, so two instances, or an old and a new process during a rolling restart, never both continue it.

Opt-out. A per-workflow setting turns automatic continuation off; the name is settled in the plan. The default is on, because only safe runs are ever continued.

Visibility. The run records that it was continued automatically, and why it stopped. swamp run history and the dashboard show it. An audit event records each automatic continuation, and each refusal with its reason.

Out of scope

  • Parking runs at a step boundary on a planned shutdown (option A, #4737).
  • Worker steps surviving a restart (option C).
  • An attached client following the continued run into the new process (part of A).

Decisions taken from the design page

  • Continue on its own only when it is safe, with the per-workflow opt-out.
  • A step with no guard is never re-run without an operator.
  • Only one process continues a given run.

Tests

  • Stopped at a step boundary: the run is continued at the next boot, under the same run id, and finished steps are not re-run.
  • Guarded step cut off: an unknown step with a guard is continued; the guard skips it when its work is already done, and runs it otherwise.
  • Unguarded step cut off: an unknown step without a guard is left interrupted, and the log and the run name that step.
  • Every refusal: opt-out, changed definition, changed fan-out plan, unsettled nested run.
  • Webhook and cron runs: a webhook run and a cron run interrupted by shutdown are continued like API runs.
  • One continuer: two instances, or two processes, find the same interrupted run; exactly one continues it.
  • Team mode: a new leader continues the old leader's safe runs.
02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 5 MOREREVIEW+ 54 MOREPR_LINKED+ 2 MORESESSION_SUMMARIZED

Shipped

11/14/2026, 10:40:38 AM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack7211/12/2026, 9:49:18 PM
stack72 linked sibling of #470311/13/2026, 1:38:26 PM

Sign in to post a ripple.