Skip to main content
← Back to list
01Issue
FeatureShippedSwamp CLIPublic
Assigneesstack72

Relationships

#4541 feat(workflows): fan-out approval gates: rollout.approve suspends a fan-out job before dispatch with a prompt built from the frozen plan, and after the canary batch

Opened by stack72 · 11/9/2026· Shipped 11/9/2026

Context

This is the approval card of Phase 3 of the fleet fan-out work. On main today:

  • fanOut jobs plan against serve's pool and run one pinned lane per matching worker (#4506)
  • runs resume on the frozen plan (#4516)
  • runs roll out in canary batches under an error budget (#4531, PR #3683)

What is missing for the operator is a human decision at the two points that matter in a fleet change:

  1. Before anything is sent, with the actual hosts in front of them.
  2. After the canary, once its lanes and their assertions have passed.

A manual_approval step cannot do either today:

  • A prompt is built when the run starts, so it cannot name the hosts a plan picks later. The fleet patching lab hit this.
  • Validation (#4379) refuses manual_approval inside a fan-out job, and a gate in an earlier job only sees what existed before the fan-out job planned.

Links:

Proposal

fanOut:
  offline: wait
  rollout:
    batches: [1, 2, 100%]
    maxFailures: 0
    approve: [plan, canary]   # either or both
    approveTimeout: 86400     # optional, seconds; as manual_approval's timeout
  • plan suspends the job once the plan is recorded and before any lane is dispatched.
  • canary suspends the job once batch 1 has settled and the error budget still allows the rollout to go on, before batch 2 starts. It is refused when there is only one batch: a canary needs a batch after it.

The gates are ordinary approval gates

They reuse manual_approval's machinery end to end, so nothing about who may approve, how a decision is stored or judged, or how a run resumes is new:

  • A gate step record in the fan-out job. Its name is derived from the job and unique in the workflow, for example patch[approve-plan] and patch[approve-canary], following the step[worker] lane form. Settle the spelling in the PR. Validation refuses an authored step whose name collides.
  • The record goes through waiting_approval and the run suspends. swamp workflow approve / reject (CLI and serve) decide it by that step name, and approvalDecision is stored on it as on any gate.
  • The run-record judge accepts the decision only when serve holds it (decisionRefusal, content_bindings.ts), unchanged. The gate record itself must be accepted only where the plan says it belongs: a plan gate once the plan exists and before any lane step starts, a canary gate once batch 1 has settled. A forwarded record cannot add, move or drop one.
  • The same grants and settings apply: run grants, approveRequiresExplicitGrant, autoResume, timeouts (approval_timeout.ts), and approvals never being reused silently.

Prompts are built from the frozen plan

These prompts are written by swamp, not authored CEL, and are built when the gate opens, never at run start.

  • Plan gate:
    • the selector
    • every member, with its disposition and reason (who gets a lane, who waits, who is excluded and why)
    • the batches
    • the resolved budget
  • Canary gate:
    • the canary lanes' results, including failed or warning assertions
    • the budget used out of allowed
    • the lanes the next batch will start

Log and JSON modes both carry the prompt as data (members, batches, budget, canary) as well as text, so serve clients and the dashboard can render it.

Decisions

  • Approve the plan: batch 1 starts.
  • Approve the canary: the next batch starts, and the rollout continues under the budget.
  • Reject either gate: no further lane starts. Every lane that never started is held, as in a #4531 halt, and recorded with the halt reason rejected. The job fails and downstream succeeded jobs are skipped.
  • Timeout: behaves as a rejection, as manual_approval does.

Resume

  • A gate already approved stays approved and is not asked again, unless --from resets it.
  • A rejected gate is refused on retry, as for manual_approval: start a new run.
  • A resume after a later halt walks the remaining batches without re-asking the canary gate.

Blocked by

  • #4531 (shipped)

Blocks

  • The swamp-uat rollout journeys (Phase 4's unlock test), which include the plan approval naming the hosts.

Done when

Fleet harness tests with dispatch counts, red on main (where approve is an unknown key):

  • approve: [plan]: the run suspends with 0 dispatches. The prompt data names every member, its disposition and the batches. Approve, and every lane runs.
  • Reject at the plan gate: 0 dispatches, every lane held with reason rejected, job failed, downstream job skipped.
  • approve: [canary]: the canary runs, the run suspends with exactly the canary's dispatches, and the prompt shows its results. Approve, and the rest roll out in batch order.
  • Reject at the canary gate: the canary result stands and the rest are held.
  • A canary failure that spends the budget halts as in #4531. The canary gate never opens.
  • autoResume under serve continues the run after a decision, without a manual resume.
  • A gate timeout acts as a rejection.
  • Approving through serve with and without approveRequiresExplicitGrant matches a plain manual_approval gate.
  • Validation refuses:
    • canary with a single batch
    • approve without rollout.batches
    • an authored step named like a gate
    • an unknown approve value

Also:

  • Judge tests: a gate record added out of place (before the plan, after a lane started, or a canary gate before batch 1 settled), changed or dropped is refused. A decision serve does not hold is refused, as today.
  • Fan-out jobs without approve behave identically: the #4531 rollout rows and the #4516 and #4506 fleet rows pass unedited.
  • Live check before verifying: the compiled branch binary, native serve and real workers. Run plan approve, canary approve, canary reject, and approve-then-kill -9-serve followed by recover. Take a sha256 of the run record before and after each refusal.
02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 9 MOREREVIEW+ 22 MOREPR_LINKED+ 2 MORESESSION_SUMMARIZED

Shipped

11/9/2026, 9:33:18 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack7211/9/2026, 1:56:30 PM

Sign in to post a ripple.