Skip to main content
← Back to list
01Issue
FeatureShippedSwamp CLIPublic
Assigneesstack72

Relationships

#4516 feat(workflows): fan-out runs resume, retry and recover on their frozen plan, re-running only the lanes that did not succeed, and show the plan in log and JSON output

Opened by stack72 · 11/8/2026· Shipped 11/8/2026

Context

This is the last item of Phase 2 of the fleet fan-out work. #4506 (PR #3666) made a fanOut job run one pinned lane per matching worker: lane steps are ordinary job steps named step[worker], and the plan is frozen in the run record by judgeRunRecord. To keep that release safe, it refused three things this item lifts or keeps (see src/domain/workflows/fan_out_job.ts):

  1. refuseFanOutResume: resume, recover and CLI recover of any run whose workflow or record holds fan-out. It is called from execution_service.ts, suspended_run_resolver.ts (resolveResumableRun) and cli/commands/workflow_recover.ts.
  2. refuseFanOutRun: a fan-out workflow that holds any step that can suspend the run (manual_approval, wait_for_signal, workflow), because the suspended run could never be resumed.
  3. refuseFanOutRun: rollout on a fan-out job. This one stays; Phase 3 lifts it.

This item makes fan-out runs resumable, and adds the output that shows the plan.

Links:

What changes

1. Resume, retry and recover finish the planned set

A fan-out job never re-plans. Resume reuses the frozen plan in the run record:

  • Retry (workflow resume --run <id>): the reset set is the failed or unfinished lanes' steps (and their lane-local dependents), plus downstream jobs as today. Lanes that succeeded keep their state and outputs, and are never dispatched again.
  • --from <step> inside a fan-out job resets that step and its lane-local dependents only in lanes that did not succeed. --from a later job leaves every lane alone. To re-run passed lanes, start a new run. Say so in the refusal or help text.
  • Recover (an interrupted run): lanes that were in flight follow today's lost-worker and interrupted-step rules, lane by lane. Lanes that finished keep their result.
  • Never add members. A worker that enrolled after the plan was made is not given a lane. A member that has since left keeps its lane, which waits or fails under the plan's offline policy and queueTimeout.
  • Lanes stay pinned by worker name. A real worker gets a new instanceUuid on every restart, so a resumed lane must dispatch to the restarted process of the same worker.
  • The plan stays frozen. Resume changes lane step state only, never the plan, which judgeRunRecord would refuse anyway.

2. What resume refuses

Each of these keeps the run record unchanged, and they are checked in resolveResumableRun as well as in the execution service, since the CLI and serve call the resolver first:

  • A started fan-out job whose plan is missing (fanOutPlanMissing, from #4379). An older swamp dropped the plan when it saved the run. Message: start a new run, never re-plan silently.
  • A workflow edited so the fan-out job changed structurally: fanOut removed or added, or the job's labels or platform changed. Add these to the existing resume structure check (planFailedRunResume in resume_reset.ts) so they are refused with the other structural changes. A job gaining fanOut after the run began is refused the same way.

3. Lift the suspend refusal

Once resume works, a fan-out workflow may hold manual_approval, wait_for_signal and nested workflow steps in other jobs. Validation from #4379 still refuses them inside the fan-out job itself. Approve-then-resume and autoResume of such a run continue at the fan-out job with the frozen plan. Remove the suspending-step half of refuseFanOutRun. Keep its --server and rollout refusals.

4. Output

In log and JSON modes, for run and resume alike (method_execution_service.ts events, presentation/renderers/workflow_run.ts):

  • job_fanout_planned, emitted once the plan is recorded. It carries the selector, the policy, plannedLanes, and every member with its disposition and reason.
  • A waiting event per waiting member, naming the worker and the queueTimeout it has.
  • An end-of-job summary: matching, lanes succeeded, lanes failed, excluded and skipped (each with its reason).
  • On resume, which lanes are re-run and which keep their earlier result.

The JSON events follow the json-mode conformance rules. The log renderer follows the terminal-output conventions.

Blocked by

  • #4506 (shipped)

Blocks

  • Phase 3 of the fan-out work (swamp-uat fan-out journeys, rollout with batches and the error budget)

Done when

Fleet harness tests, with dispatch counts:

  • One lane fails, the cause is fixed, and a retry dispatches only that lane. Passed lanes get 0 new dispatches.
  • --from a lane step re-runs that step only in lanes that did not succeed.
  • A member offline at plan time and never connected is waiting on a retry, and runs when its worker connects as a new process (new instanceUuid).
  • A worker that enrolled after the plan gets no lane on resume.
  • A run whose plan was stripped (simulate an older binary's save) is refused, and its record is byte-identical before and after.
  • Removing fanOut, adding it, or changing a fan-out job's labels between run and resume is refused, with the record unchanged.
  • A workflow with an approval in a job before the fan-out job suspends, is approved, auto-resumes, and runs every lane once.
  • job_fanout_planned, the waiting event and the summary appear in both log and JSON output, with renderer tests.
  • The --server and rollout refusals are unchanged.
  • forEach and plain-job resume rows from #4271 stay green, and the existing resume suites pass unedited.

Live check before verifying. Run the compiled branch binary against a native swamp serve with real swamp worker connect processes, as #4506 did:

  • kill one worker mid-lane, restart it, and resume
  • kill -9 serve, restart it, and recover
  • approve-then-resume of a workflow with an earlier approval job

Take a sha256 of the run record before and after each refusal. #4506's live check found two bugs the in-process harness hides.

Red checks. Every new behaviour test fails on main, where it hits the deferred-resume refusal.

02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 14 MOREREVIEW+ 8 MOREPR_LINKED+ 2 MORESESSION_SUMMARIZED

Shipped

11/8/2026, 2:57:37 AM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack7211/8/2026, 1:21:33 AM

Sign in to post a ripple.