Skip to main content
← Back to list
01Issue
FeatureShippedSwamp CLIPublic
Assigneesstack72

Relationships

#4531 feat(workflows): fan-out rollout: lanes start in canary batches under an error budget (maxFailures as a count or a percentage), hold the rest on a halt, and resume continues the rollout

Opened by stack72 · 11/8/2026· Shipped 11/9/2026

Context

This is the rollout card of Phase 3 of the fleet fan-out work: batching and an error budget for a fan-out job. Phase 2 has shipped:

  • #4379: the fanOut field
  • #4506: the lane engine
  • #4516: resume on the frozen plan
  • #4527: the snapshot freeze

Today every lane starts at once, and a fanOut job that sets rollout is refused at run start (refuseFanOutRun in src/domain/workflows/fan_out_job.ts).

The pieces exist but are unused:

  • RolloutPolicySchema (batches, maxFailures) in src/domain/remote/fan_out.ts
  • planBatches and nextBatch in src/domain/remote/rollout.ts (#4279, property-tested). nextBatch is meant to be the one place the budget is decided.
  • The plan stored on each job run already records batches and the resolved maxFailures. Today it is always one batch with no budget, because execution_service.ts calls planBatches(plan.members, undefined).

This item makes rollout real.

fanOut:
  offline: wait
  rollout:
    batches: [1, 2, 100%]   # canary 1 host, then 2, then the rest
    maxFailures: 10%        # or a count, e.g. 3

Links:

Semantics (decided)

  1. Batches settle. A batch starts only when every lane of the batches before it has finished, succeeded or failed, assertions included. Inside a batch, the job's concurrency caps lanes in flight, and SWAMP_MAX_CONCURRENT_STEPS stays the shared host ceiling. Lanes are ordered as planBatches orders them: connected hosts by name, then waiting hosts last, so the canary is a host that is up. Batch percentages round up, and the last batch is 100%. A rolling window is out of scope.
  2. The budget is a fixed number made at plan time. maxFailures is a count, or a percentage of planned lanes rounded down, as resolveMaxFailures already does. It is not a live rate. Without maxFailures there is no budget.
  3. What counts. A failed lane counts: a failed step after allowFailure, a failed assertion, a queueTimeout, or worker_ineligible. Plan-time excluded, skipped and failed members never had a lane and never count.
  4. Halt. nextBatch is consulted before every lane start, not once per batch. Once failed lanes exceed maxFailures, no lane starts, even in the middle of a batch. Lanes already running finish, so a patch is never cut off halfway. Every lane that never started is held. The job fails, and downstream jobs that need succeeded are skipped. maxFailures: 0 stops on the first failure.
  5. Resume continues the rollout. Failed lanes run first, then held lanes, in the remaining batch sizes from the batch the halt stopped in. The budget is fresh: only failures in the resumed attempt count. Lanes that succeeded are never dispatched again, as #4516 already guarantees. A cancel holds no lanes; it settles them as #4506 does.
  6. No pause or bake time between batches. A slow failure is caught by a re-check at the end of the lane. Approval after the canary is a separate card (rollout.approve).

What changes

1. Lift the refusal

Remove the rollout half of refuseFanOutRun. Keep its --server refusal.

2. Batch-gated lane scheduler

execution_service.ts builds the batches with planBatches(plan.members, policy.rollout) and records them through recordFanOutPlan, as it already does. It then feeds lanes to the scheduler as nextBatch allows, instead of handing every lane stream to mergeWithConcurrency up front.

Keep the deciding logic pure, in rollout.ts, with nextBatch the single place the budget is checked. If its shape needs changing (for example, telling "halt, but lanes are still running" apart from "halt, nothing running"), change it there with its property tests.

3. Held lanes and the halt record

The plan is frozen by the run-record judge (fanOutPlanRefusal, content_bindings.ts), so the held state and the halt cannot be written into it.

  • Held lanes: derive them from lane state. A lane whose steps never started when the job halted is held. Their step records use an existing status: skip reasons cannot grow without older binaries refusing the record.
  • The halt: record it once on the job run, in a new optional field (for example fanOutHalt: { failedLanes, maxFailures, batch }) beside the plan. Make the judge accept it only when the job is ending, and refuse a later change to it. Older binaries drop the field on save; resume must still work then, treating held lanes as unfinished.
  • The batch a lane belongs to: read it from the frozen plan's batches. Never store it twice.

4. Resume

Extend #4516's reset selection (resume_reset.ts): failed lanes first, then held lanes, and walk the remaining batches from the halted one with a fresh budget. --from keeps its #4516 meaning, applied only to lanes that did not succeed.

5. Output

Two new event kinds go into FAN_OUT_EVENT_KINDS, so serve sends them only to clients that list them:

  • job_fanout_batch_started: batch index and its lanes
  • job_fanout_halted: failed lanes, budget, held lanes

job_fanout_planned gains the batches and the resolved budget. job_fanout_summary gains held and budget: { used, allowed }. These appear in log and JSON modes and follow the json-mode conformance rules. On resume, resumed says which lanes are re-run, held or kept.

Blocked by

  • Phase 2 (#4379, #4506, #4516, #4527), all shipped.

The swamp-uat fan-out journeys card runs in parallel. It covers Phase 2 behaviour and touches no files this item changes.

Blocks

  • Approval gates (rollout.approve: the plan and the canary)
  • The swamp-uat rollout journeys

Done when

Fleet harness tests, with dispatch counts, each red on main (where rollout is refused):

  • [1, 2, 100%] over 5 lanes: batch 2 starts only after the canary settles, and batch 3 only after batch 2.
  • A failed assertion in the canary under maxFailures: 0: dispatch count 1, four lanes held, job failed, downstream succeeded job skipped.
  • maxFailures: 25% over 8 lanes (budget 2) halts on the third failure, mid-batch if that is where it lands. Running lanes finish, and the rest are held.
  • No maxFailures: failures never halt the rollout.
  • A waiting lane is in the last batch. Its queueTimeout failure counts against the budget, and an excluded member does not.
  • Halt, fix, resume: the failed canary runs first, then held lanes in the remaining batch sizes. Passed lanes get 0 new dispatches. The resumed attempt has a fresh budget.
  • A cancel mid-rollout holds nothing and sends nothing new.

Judge tests:

  • The halt record is accepted once as the job ends, and refused if added earlier, changed or dropped.
  • The plan-freeze and snapshot-freeze tests pass unedited.

Output:

  • Renderer tests for the new events and fields in log and JSON.
  • An older CLI against the new serve gets no new events and does not crash.

Unchanged behaviour:

  • nextBatch and planBatches property tests stay green, and any change to their shape has property coverage.
  • fanOut jobs without rollout, forEach and plain jobs are unchanged: the #4271 baseline, the #4506 and #4516 fleet rows, and the resume suites pass unedited.

Live check before verifying. Compiled branch binary, native swamp serve, at least three real swamp worker connect processes:

  • break the canary so its assertion fails: the rest are held and nothing else is touched
  • fix it and resume: the rollout finishes, and passed lanes are not re-run
  • a waiting host in the last batch that connects during the rollout
  • kill -9 serve mid-rollout, restart it and recover

Take a sha256 of the run record before and after each refusal.

02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 5 MOREREVIEW+ 14 MOREPR_LINKED+ 2 MORESESSION_SUMMARIZED

Shipped

11/9/2026, 10:34:10 AM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack7211/8/2026, 11:16:27 PM

Sign in to post a ripple.