Relationships
#4541 feat(workflows): fan-out approval gates: rollout.approve suspends a fan-out job before dispatch with a prompt built from the frozen plan, and after the canary batch
Opened by stack72 · 11/9/2026· Shipped 11/9/2026
Context
This is the approval card of Phase 3 of the fleet fan-out work. On main today:
fanOutjobs plan against serve's pool and run one pinned lane per matching worker (#4506)- runs resume on the frozen plan (#4516)
- runs roll out in canary batches under an error budget (#4531, PR #3683)
What is missing for the operator is a human decision at the two points that matter in a fleet change:
- Before anything is sent, with the actual hosts in front of them.
- After the canary, once its lanes and their assertions have passed.
A manual_approval step cannot do either today:
- A prompt is built when the run starts, so it cannot name the hosts a plan picks later. The fleet patching lab hit this.
- Validation (#4379) refuses
manual_approvalinside a fan-out job, and a gate in an earlier job only sees what existed before the fan-out job planned.
Links:
- Design: https://claude.ai/artifact/WbnKGvQcuJdo6jQBbezDxz
- Work plan (Phase 3, "Approval gates: the plan and the canary"): https://claude.ai/artifact/CTfkUbRmw6JxccPyyVhY9H#w-approve
- Related: #3824
Proposal
fanOut:
offline: wait
rollout:
batches: [1, 2, 100%]
maxFailures: 0
approve: [plan, canary] # either or both
approveTimeout: 86400 # optional, seconds; as manual_approval's timeoutplansuspends the job once the plan is recorded and before any lane is dispatched.canarysuspends the job once batch 1 has settled and the error budget still allows the rollout to go on, before batch 2 starts. It is refused when there is only one batch: a canary needs a batch after it.
The gates are ordinary approval gates
They reuse manual_approval's machinery end to end, so nothing about who may approve, how a decision is stored or judged, or how a run resumes is new:
- A gate step record in the fan-out job. Its name is derived from the job and unique in the workflow, for example
patch[approve-plan]andpatch[approve-canary], following thestep[worker]lane form. Settle the spelling in the PR. Validation refuses an authored step whose name collides. - The record goes through
waiting_approvaland the run suspends.swamp workflow approve/reject(CLI and serve) decide it by that step name, andapprovalDecisionis stored on it as on any gate. - The run-record judge accepts the decision only when serve holds it (
decisionRefusal,content_bindings.ts), unchanged. The gate record itself must be accepted only where the plan says it belongs: a plan gate once the plan exists and before any lane step starts, a canary gate once batch 1 has settled. A forwarded record cannot add, move or drop one. - The same grants and settings apply:
rungrants,approveRequiresExplicitGrant,autoResume, timeouts (approval_timeout.ts), and approvals never being reused silently.
Prompts are built from the frozen plan
These prompts are written by swamp, not authored CEL, and are built when the gate opens, never at run start.
- Plan gate:
- the selector
- every member, with its disposition and reason (who gets a lane, who waits, who is excluded and why)
- the batches
- the resolved budget
- Canary gate:
- the canary lanes' results, including failed or warning assertions
- the budget used out of allowed
- the lanes the next batch will start
Log and JSON modes both carry the prompt as data (members, batches, budget, canary) as well as text, so serve clients and the dashboard can render it.
Decisions
- Approve the plan: batch 1 starts.
- Approve the canary: the next batch starts, and the rollout continues under the budget.
- Reject either gate: no further lane starts. Every lane that never started is held, as in a #4531 halt, and recorded with the halt reason
rejected. The job fails and downstreamsucceededjobs are skipped. - Timeout: behaves as a rejection, as
manual_approvaldoes.
Resume
- A gate already approved stays approved and is not asked again, unless
--fromresets it. - A rejected gate is refused on retry, as for
manual_approval: start a new run. - A resume after a later halt walks the remaining batches without re-asking the canary gate.
Blocked by
- #4531 (shipped)
Blocks
- The swamp-uat rollout journeys (Phase 4's unlock test), which include the plan approval naming the hosts.
Done when
Fleet harness tests with dispatch counts, red on main (where approve is an unknown key):
approve: [plan]: the run suspends with 0 dispatches. The prompt data names every member, its disposition and the batches. Approve, and every lane runs.- Reject at the plan gate: 0 dispatches, every lane held with reason
rejected, job failed, downstream job skipped. approve: [canary]: the canary runs, the run suspends with exactly the canary's dispatches, and the prompt shows its results. Approve, and the rest roll out in batch order.- Reject at the canary gate: the canary result stands and the rest are held.
- A canary failure that spends the budget halts as in #4531. The canary gate never opens.
autoResumeunder serve continues the run after a decision, without a manual resume.- A gate timeout acts as a rejection.
- Approving through serve with and without
approveRequiresExplicitGrantmatches a plainmanual_approvalgate. - Validation refuses:
canarywith a single batchapprovewithoutrollout.batches- an authored step named like a gate
- an unknown
approvevalue
Also:
- Judge tests: a gate record added out of place (before the plan, after a lane started, or a canary gate before batch 1 settled), changed or dropped is refused. A decision serve does not hold is refused, as today.
- Fan-out jobs without
approvebehave identically: the #4531 rollout rows and the #4516 and #4506 fleet rows pass unedited. - Live check before verifying: the compiled branch binary, native serve and real workers. Run plan approve, canary approve, canary reject, and approve-then-
kill -9-serve followed by recover. Take a sha256 of the run record before and after each refusal.
Shipped
Click a lifecycle step above to view its details.
Sign in to post a ripple.