Relationships
#4516 feat(workflows): fan-out runs resume, retry and recover on their frozen plan, re-running only the lanes that did not succeed, and show the plan in log and JSON output
Opened by stack72 · 11/8/2026· Shipped 11/8/2026
Context
This is the last item of Phase 2 of the fleet fan-out work. #4506 (PR #3666) made a fanOut job run one pinned lane per matching worker: lane steps are ordinary job steps named step[worker], and the plan is frozen in the run record by judgeRunRecord. To keep that release safe, it refused three things this item lifts or keeps (see src/domain/workflows/fan_out_job.ts):
refuseFanOutResume: resume, recover and CLI recover of any run whose workflow or record holds fan-out. It is called fromexecution_service.ts,suspended_run_resolver.ts(resolveResumableRun) andcli/commands/workflow_recover.ts.refuseFanOutRun: a fan-out workflow that holds any step that can suspend the run (manual_approval,wait_for_signal,workflow), because the suspended run could never be resumed.refuseFanOutRun:rollouton a fan-out job. This one stays; Phase 3 lifts it.
This item makes fan-out runs resumable, and adds the output that shows the plan.
Links:
- Design: https://claude.ai/artifact/WbnKGvQcuJdo6jQBbezDxz
- Work plan (Phase 2, "Resume and output"): https://claude.ai/artifact/CTfkUbRmw6JxccPyyVhY9H#w-resume
- Related: #3824
What changes
1. Resume, retry and recover finish the planned set
A fan-out job never re-plans. Resume reuses the frozen plan in the run record:
- Retry (
workflow resume --run <id>): the reset set is the failed or unfinished lanes' steps (and their lane-local dependents), plus downstream jobs as today. Lanes that succeeded keep their state and outputs, and are never dispatched again. --from <step>inside a fan-out job resets that step and its lane-local dependents only in lanes that did not succeed.--froma later job leaves every lane alone. To re-run passed lanes, start a new run. Say so in the refusal or help text.- Recover (an interrupted run): lanes that were in flight follow today's lost-worker and interrupted-step rules, lane by lane. Lanes that finished keep their result.
- Never add members. A worker that enrolled after the plan was made is not given a lane. A member that has since left keeps its lane, which waits or fails under the plan's
offlinepolicy andqueueTimeout. - Lanes stay pinned by worker name. A real worker gets a new
instanceUuidon every restart, so a resumed lane must dispatch to the restarted process of the same worker. - The plan stays frozen. Resume changes lane step state only, never the plan, which
judgeRunRecordwould refuse anyway.
2. What resume refuses
Each of these keeps the run record unchanged, and they are checked in resolveResumableRun as well as in the execution service, since the CLI and serve call the resolver first:
- A started fan-out job whose plan is missing (
fanOutPlanMissing, from #4379). An older swamp dropped the plan when it saved the run. Message: start a new run, never re-plan silently. - A workflow edited so the fan-out job changed structurally:
fanOutremoved or added, or the job'slabelsorplatformchanged. Add these to the existing resume structure check (planFailedRunResumeinresume_reset.ts) so they are refused with the other structural changes. A job gainingfanOutafter the run began is refused the same way.
3. Lift the suspend refusal
Once resume works, a fan-out workflow may hold manual_approval, wait_for_signal and nested workflow steps in other jobs. Validation from #4379 still refuses them inside the fan-out job itself. Approve-then-resume and autoResume of such a run continue at the fan-out job with the frozen plan. Remove the suspending-step half of refuseFanOutRun. Keep its --server and rollout refusals.
4. Output
In log and JSON modes, for run and resume alike (method_execution_service.ts events, presentation/renderers/workflow_run.ts):
job_fanout_planned, emitted once the plan is recorded. It carries the selector, the policy,plannedLanes, and every member with its disposition and reason.- A waiting event per
waitingmember, naming the worker and thequeueTimeoutit has. - An end-of-job summary: matching, lanes succeeded, lanes failed, excluded and skipped (each with its reason).
- On resume, which lanes are re-run and which keep their earlier result.
The JSON events follow the json-mode conformance rules. The log renderer follows the terminal-output conventions.
Blocked by
- #4506 (shipped)
Blocks
- Phase 3 of the fan-out work (swamp-uat fan-out journeys, rollout with batches and the error budget)
Done when
Fleet harness tests, with dispatch counts:
- One lane fails, the cause is fixed, and a retry dispatches only that lane. Passed lanes get 0 new dispatches.
--froma lane step re-runs that step only in lanes that did not succeed.- A member offline at plan time and never connected is waiting on a retry, and runs when its worker connects as a new process (new
instanceUuid). - A worker that enrolled after the plan gets no lane on resume.
- A run whose plan was stripped (simulate an older binary's save) is refused, and its record is byte-identical before and after.
- Removing
fanOut, adding it, or changing a fan-out job's labels between run and resume is refused, with the record unchanged. - A workflow with an approval in a job before the fan-out job suspends, is approved, auto-resumes, and runs every lane once.
job_fanout_planned, the waiting event and the summary appear in both log and JSON output, with renderer tests.- The
--serverandrolloutrefusals are unchanged. forEachand plain-job resume rows from #4271 stay green, and the existing resume suites pass unedited.
Live check before verifying. Run the compiled branch binary against a native swamp serve with real swamp worker connect processes, as #4506 did:
- kill one worker mid-lane, restart it, and resume
kill -9serve, restart it, and recover- approve-then-resume of a workflow with an earlier approval job
Take a sha256 of the run record before and after each refusal. #4506's live check found two bugs the in-process harness hides.
Red checks. Every new behaviour test fails on main, where it hits the deferred-resume refusal.
Shipped
Click a lifecycle step above to view its details.
Sign in to post a ripple.