Relationships
#4506 feat(workflows): the fan-out lane engine: a fanOut job plans against serve's worker pool and runs one pinned lane per matching worker
Opened by stack72 · 11/7/2026· Shipped 11/8/2026
Context
This is the lane engine, the second item of Phase 2 of the fleet fan-out work. It is the change that makes fan-out real: a job with fanOut runs once per matching worker, as a lane pinned to that worker. Already shipped:
- #4271: baseline tests and
integration/fleet_harness.ts, the red checks this item flips - #4279:
WorkerPoolReader,planFanOut,planBatches/nextBatch(src/domain/remote/) - #4375: the pool reader's mint-clock fix
- #4379: the job-level
fanOutfield, its refusals,JobRun.recordFanOutPlan,fanOutPlanMissing, and the run guardrefuseFanOutJobsthat this item removes
Links:
- Design: https://claude.ai/artifact/WbnKGvQcuJdo6jQBbezDxz
- Work plan (Phase 2, "Lane engine"): https://claude.ai/artifact/CTfkUbRmw6JxccPyyVhY9H#w-lanes
- Related: #3824
Resume and output (the plan event, log and JSON rendering, the resume rules) are the next item. This one runs lanes for a fresh run. Rollout batches and the error budget come in Phase 3: here every lane goes in one batch, and rollout is stored but not acted on.
What changes
1. Pool at job start
Serve registers its WorkerPoolReader (createServeWorkerPoolReader, src/serve/worker_pool_reader.ts) next to setRemoteStepDispatcher in src/cli/commands/serve.ts, through a domain-level setter and getter in src/domain/remote/ that follow the dispatcher's pattern, and clears it on shutdown.
When a fan-out job starts and no reader is registered (a local swamp workflow run without --server), the run fails before any step with a UserError: "fanOut needs the worker pool of swamp serve; run it with --server".
2. Plan and record
At job start, before any dispatch:
- read the pool
- call
planFanOut(selector, job.fanOutPolicy, pool), where the selector is the job's labels and platform after workflow inheritance - call
planBatches - call
JobRun.recordFanOutPlan, and save the run
A refused plan (fewer live members than minWorkers) fails the job with a message that names the selector and the live count, and nothing is dispatched.
3. Lanes
Each dispatch or waiting member gets a lane (job[worker]) that runs the job's step DAG on its own:
- Pinned to its worker. An internal target override, with an affinity key per lane (
runId:job:worker) bound at plan time. This is not today's "bind on the first completed step" rule: a lane must never move to another worker. Release the keys when the job ends, as job-level affinity does today. self.worker(name,labels,platform,arch) is in scope for every step's inputs, global arguments, name and guard.- Per-worker instances.
behaviorNamegets a-<worker>suffix unless it already referencesself.worker. Per-model locks and data names are then per host, and lanes run in parallel. - Assert steps evaluate on the orchestrator, as today, against the data their lane's host steps wrote.
- Concurrency. The job's
concurrencycaps lanes in flight. Steps inside a lane follow their owndependsOn. - Lane result. A lane fails when one of its steps fails, after
allowFailure. The job succeeds only if every lane does. Theexcluded,skippedand plan-timefailedmembers are reported in the plan;skippedandexcludeddon't fail the job, plan-timefaileddoes.
This is the largest change in the plan. Today jobs and steps run level by level through mergeWithConcurrency, so lanes need a scheduler nested inside one job. Keep it beside the job loop in execution_service.ts, and keep the existing forEach and plain-job paths byte-for-byte.
4. Eligibility at dispatch
The plan excludes unverified and draining workers, but a worker can change after the plan is made.
- A lane whose worker turns unverified or starts draining before its first dispatch fails with
worker_ineligible, and is never sent. Today a targeted step queues behind such a worker untilqueueTimeout. offline: waitmembers queue until their worker reconnects. They fail atqueueTimeoutwith the existing "no worker matching target" error.- A worker lost mid-lane follows today's lost-worker rules (re-dispatch before a write, fail after one), scoped to its lane.
5. Remove the run guard
Remove refuseFanOutJobs from run, --last-evaluated, resume and recover (execution_service.ts). Replace it on resume and recover with a refusal that keeps resume deferred: "resuming a run with a fan-out job arrives in the next release". Resume and output then turns that back on with the real rules and fanOutPlanMissing. A fan-out job must never run as a plain job on any path.
6. Memory
#4477 (open) reports that a workflow run keeps every step's resource contents in memory until the run ends. Lanes are steps, so a fan-out across many hosts grows the same way. That fix stays with #4477. This item must not add another structure that holds every lane's outputs for the whole run: keep per-lane state to what the lane's own later steps and the plan need.
Blocked by
- #4379 (shipped)
Blocks
- Resume and output for fan-out jobs (Phase 2)
- The Phase 3 rollout (batches,
maxFailures), which plugsnextBatchinto this scheduler
Done when
Through integration/fleet_harness.ts, with dispatch counts:
- Every connected matching worker runs its lane, in parallel up to
concurrency, each on its own worker and its own instance (host-patch-<worker>). #4271's "one shared definition serializes" row does not happen for a fan-out job. - An offline member is in the plan as
waiting, runs when it reconnects mid-run, and fails atqueueTimeoutif it never does.offline: skipandfailare recorded in the plan and behave as specified. - Unverified and draining workers never receive a dispatch. One that turns unverified or draining after planning fails its lane with
worker_ineligible. - An empty selector match, or fewer live members than
minWorkers, fails the job with nothing dispatched. - A failing assert in one lane fails that lane and the job, and other lanes still run to completion.
self.workerresolves in step names, inputs, global arguments and guards.- A local run without serve is refused with the
--servermessage. forEachand plain-job behaviour is unchanged: every #4271 baseline row stays green, and the existing execution and resume suites pass unedited.- Resume and recover of a fan-out run are refused with the "next release" message, and nothing runs.
--last-evaluatedruns lanes like any other run.
Notes
- New domain files and a new setter in
src/domain/remote/: run all ofintegration/before verifying, sincearchitecture_boundary_test.tsis not a*_rules_test. - Before verifying, red-check the new integration tests against main. They must fail there on the run guard.
- Keep the scheduler's decisions pure where possible (which lane starts next, what a lane's result is) and unit test them apart from dispatch.
Shipped
Click a lifecycle step above to view its details.
Sign in to post a ripple.