Relationships
#4531 feat(workflows): fan-out rollout: lanes start in canary batches under an error budget (maxFailures as a count or a percentage), hold the rest on a halt, and resume continues the rollout
Opened by stack72 · 11/8/2026· Shipped 11/9/2026
Context
This is the rollout card of Phase 3 of the fleet fan-out work: batching and an error budget for a fan-out job. Phase 2 has shipped:
- #4379: the
fanOutfield - #4506: the lane engine
- #4516: resume on the frozen plan
- #4527: the snapshot freeze
Today every lane starts at once, and a fanOut job that sets rollout is refused at run start (refuseFanOutRun in src/domain/workflows/fan_out_job.ts).
The pieces exist but are unused:
RolloutPolicySchema(batches,maxFailures) insrc/domain/remote/fan_out.tsplanBatchesandnextBatchinsrc/domain/remote/rollout.ts(#4279, property-tested).nextBatchis meant to be the one place the budget is decided.- The plan stored on each job run already records
batchesand the resolvedmaxFailures. Today it is always one batch with no budget, becauseexecution_service.tscallsplanBatches(plan.members, undefined).
This item makes rollout real.
fanOut:
offline: wait
rollout:
batches: [1, 2, 100%] # canary 1 host, then 2, then the rest
maxFailures: 10% # or a count, e.g. 3Links:
- Design: https://claude.ai/artifact/WbnKGvQcuJdo6jQBbezDxz
- Work plan (Phase 3, "Rollout: batches and the error budget"): https://claude.ai/artifact/CTfkUbRmw6JxccPyyVhY9H#w-rollout
- Related: #3824
Semantics (decided)
- Batches settle. A batch starts only when every lane of the batches before it has finished, succeeded or failed, assertions included. Inside a batch, the job's
concurrencycaps lanes in flight, andSWAMP_MAX_CONCURRENT_STEPSstays the shared host ceiling. Lanes are ordered asplanBatchesorders them: connected hosts by name, thenwaitinghosts last, so the canary is a host that is up. Batch percentages round up, and the last batch is100%. A rolling window is out of scope. - The budget is a fixed number made at plan time.
maxFailuresis a count, or a percentage of planned lanes rounded down, asresolveMaxFailuresalready does. It is not a live rate. WithoutmaxFailuresthere is no budget. - What counts. A failed lane counts: a failed step after
allowFailure, a failed assertion, aqueueTimeout, orworker_ineligible. Plan-timeexcluded,skippedandfailedmembers never had a lane and never count. - Halt.
nextBatchis consulted before every lane start, not once per batch. Once failed lanes exceedmaxFailures, no lane starts, even in the middle of a batch. Lanes already running finish, so a patch is never cut off halfway. Every lane that never started is held. The job fails, and downstream jobs that needsucceededare skipped.maxFailures: 0stops on the first failure. - Resume continues the rollout. Failed lanes run first, then held lanes, in the remaining batch sizes from the batch the halt stopped in. The budget is fresh: only failures in the resumed attempt count. Lanes that succeeded are never dispatched again, as #4516 already guarantees. A cancel holds no lanes; it settles them as #4506 does.
- No pause or bake time between batches. A slow failure is caught by a re-check at the end of the lane. Approval after the canary is a separate card (
rollout.approve).
What changes
1. Lift the refusal
Remove the rollout half of refuseFanOutRun. Keep its --server refusal.
2. Batch-gated lane scheduler
execution_service.ts builds the batches with planBatches(plan.members, policy.rollout) and records them through recordFanOutPlan, as it already does. It then feeds lanes to the scheduler as nextBatch allows, instead of handing every lane stream to mergeWithConcurrency up front.
Keep the deciding logic pure, in rollout.ts, with nextBatch the single place the budget is checked. If its shape needs changing (for example, telling "halt, but lanes are still running" apart from "halt, nothing running"), change it there with its property tests.
3. Held lanes and the halt record
The plan is frozen by the run-record judge (fanOutPlanRefusal, content_bindings.ts), so the held state and the halt cannot be written into it.
- Held lanes: derive them from lane state. A lane whose steps never started when the job halted is held. Their step records use an existing status: skip reasons cannot grow without older binaries refusing the record.
- The halt: record it once on the job run, in a new optional field (for example
fanOutHalt: { failedLanes, maxFailures, batch }) beside the plan. Make the judge accept it only when the job is ending, and refuse a later change to it. Older binaries drop the field on save; resume must still work then, treating held lanes as unfinished. - The batch a lane belongs to: read it from the frozen plan's
batches. Never store it twice.
4. Resume
Extend #4516's reset selection (resume_reset.ts): failed lanes first, then held lanes, and walk the remaining batches from the halted one with a fresh budget. --from keeps its #4516 meaning, applied only to lanes that did not succeed.
5. Output
Two new event kinds go into FAN_OUT_EVENT_KINDS, so serve sends them only to clients that list them:
job_fanout_batch_started: batch index and its lanesjob_fanout_halted: failed lanes, budget, held lanes
job_fanout_planned gains the batches and the resolved budget. job_fanout_summary gains held and budget: { used, allowed }. These appear in log and JSON modes and follow the json-mode conformance rules. On resume, resumed says which lanes are re-run, held or kept.
Blocked by
- Phase 2 (#4379, #4506, #4516, #4527), all shipped.
The swamp-uat fan-out journeys card runs in parallel. It covers Phase 2 behaviour and touches no files this item changes.
Blocks
- Approval gates (
rollout.approve: the plan and the canary) - The swamp-uat rollout journeys
Done when
Fleet harness tests, with dispatch counts, each red on main (where rollout is refused):
[1, 2, 100%]over 5 lanes: batch 2 starts only after the canary settles, and batch 3 only after batch 2.- A failed assertion in the canary under
maxFailures: 0: dispatch count 1, four lanes held, job failed, downstreamsucceededjob skipped. maxFailures: 25%over 8 lanes (budget 2) halts on the third failure, mid-batch if that is where it lands. Running lanes finish, and the rest are held.- No
maxFailures: failures never halt the rollout. - A
waitinglane is in the last batch. ItsqueueTimeoutfailure counts against the budget, and anexcludedmember does not. - Halt, fix, resume: the failed canary runs first, then held lanes in the remaining batch sizes. Passed lanes get 0 new dispatches. The resumed attempt has a fresh budget.
- A cancel mid-rollout holds nothing and sends nothing new.
Judge tests:
- The halt record is accepted once as the job ends, and refused if added earlier, changed or dropped.
- The plan-freeze and snapshot-freeze tests pass unedited.
Output:
- Renderer tests for the new events and fields in log and JSON.
- An older CLI against the new serve gets no new events and does not crash.
Unchanged behaviour:
nextBatchandplanBatchesproperty tests stay green, and any change to their shape has property coverage.fanOutjobs withoutrollout,forEachand plain jobs are unchanged: the #4271 baseline, the #4506 and #4516 fleet rows, and the resume suites pass unedited.
Live check before verifying. Compiled branch binary, native swamp serve, at least three real swamp worker connect processes:
- break the canary so its assertion fails: the rest are held and nothing else is touched
- fix it and resume: the rollout finishes, and passed lanes are not re-run
- a
waitinghost in the last batch that connects during the rollout kill -9serve mid-rollout, restart it and recover
Take a sha256 of the run record before and after each refusal.
Shipped
Click a lifecycle step above to view its details.
Sign in to post a ripple.