Skip to main content
← Back to list
01Issue
FeatureShippedSwamp CLIPublic
Assigneesstack72

Relationships

#4506 feat(workflows): the fan-out lane engine: a fanOut job plans against serve's worker pool and runs one pinned lane per matching worker

Opened by stack72 · 11/7/2026· Shipped 11/8/2026

Context

This is the lane engine, the second item of Phase 2 of the fleet fan-out work. It is the change that makes fan-out real: a job with fanOut runs once per matching worker, as a lane pinned to that worker. Already shipped:

  • #4271: baseline tests and integration/fleet_harness.ts, the red checks this item flips
  • #4279: WorkerPoolReader, planFanOut, planBatches / nextBatch (src/domain/remote/)
  • #4375: the pool reader's mint-clock fix
  • #4379: the job-level fanOut field, its refusals, JobRun.recordFanOutPlan, fanOutPlanMissing, and the run guard refuseFanOutJobs that this item removes

Links:

Resume and output (the plan event, log and JSON rendering, the resume rules) are the next item. This one runs lanes for a fresh run. Rollout batches and the error budget come in Phase 3: here every lane goes in one batch, and rollout is stored but not acted on.

What changes

1. Pool at job start

Serve registers its WorkerPoolReader (createServeWorkerPoolReader, src/serve/worker_pool_reader.ts) next to setRemoteStepDispatcher in src/cli/commands/serve.ts, through a domain-level setter and getter in src/domain/remote/ that follow the dispatcher's pattern, and clears it on shutdown.

When a fan-out job starts and no reader is registered (a local swamp workflow run without --server), the run fails before any step with a UserError: "fanOut needs the worker pool of swamp serve; run it with --server".

2. Plan and record

At job start, before any dispatch:

  1. read the pool
  2. call planFanOut(selector, job.fanOutPolicy, pool), where the selector is the job's labels and platform after workflow inheritance
  3. call planBatches
  4. call JobRun.recordFanOutPlan, and save the run

A refused plan (fewer live members than minWorkers) fails the job with a message that names the selector and the live count, and nothing is dispatched.

3. Lanes

Each dispatch or waiting member gets a lane (job[worker]) that runs the job's step DAG on its own:

  • Pinned to its worker. An internal target override, with an affinity key per lane (runId:job:worker) bound at plan time. This is not today's "bind on the first completed step" rule: a lane must never move to another worker. Release the keys when the job ends, as job-level affinity does today.
  • self.worker (name, labels, platform, arch) is in scope for every step's inputs, global arguments, name and guard.
  • Per-worker instances. behaviorName gets a -<worker> suffix unless it already references self.worker. Per-model locks and data names are then per host, and lanes run in parallel.
  • Assert steps evaluate on the orchestrator, as today, against the data their lane's host steps wrote.
  • Concurrency. The job's concurrency caps lanes in flight. Steps inside a lane follow their own dependsOn.
  • Lane result. A lane fails when one of its steps fails, after allowFailure. The job succeeds only if every lane does. The excluded, skipped and plan-time failed members are reported in the plan; skipped and excluded don't fail the job, plan-time failed does.

This is the largest change in the plan. Today jobs and steps run level by level through mergeWithConcurrency, so lanes need a scheduler nested inside one job. Keep it beside the job loop in execution_service.ts, and keep the existing forEach and plain-job paths byte-for-byte.

4. Eligibility at dispatch

The plan excludes unverified and draining workers, but a worker can change after the plan is made.

  • A lane whose worker turns unverified or starts draining before its first dispatch fails with worker_ineligible, and is never sent. Today a targeted step queues behind such a worker until queueTimeout.
  • offline: wait members queue until their worker reconnects. They fail at queueTimeout with the existing "no worker matching target" error.
  • A worker lost mid-lane follows today's lost-worker rules (re-dispatch before a write, fail after one), scoped to its lane.

5. Remove the run guard

Remove refuseFanOutJobs from run, --last-evaluated, resume and recover (execution_service.ts). Replace it on resume and recover with a refusal that keeps resume deferred: "resuming a run with a fan-out job arrives in the next release". Resume and output then turns that back on with the real rules and fanOutPlanMissing. A fan-out job must never run as a plain job on any path.

6. Memory

#4477 (open) reports that a workflow run keeps every step's resource contents in memory until the run ends. Lanes are steps, so a fan-out across many hosts grows the same way. That fix stays with #4477. This item must not add another structure that holds every lane's outputs for the whole run: keep per-lane state to what the lane's own later steps and the plan need.

Blocked by

  • #4379 (shipped)

Blocks

  • Resume and output for fan-out jobs (Phase 2)
  • The Phase 3 rollout (batches, maxFailures), which plugs nextBatch into this scheduler

Done when

Through integration/fleet_harness.ts, with dispatch counts:

  • Every connected matching worker runs its lane, in parallel up to concurrency, each on its own worker and its own instance (host-patch-<worker>). #4271's "one shared definition serializes" row does not happen for a fan-out job.
  • An offline member is in the plan as waiting, runs when it reconnects mid-run, and fails at queueTimeout if it never does. offline: skip and fail are recorded in the plan and behave as specified.
  • Unverified and draining workers never receive a dispatch. One that turns unverified or draining after planning fails its lane with worker_ineligible.
  • An empty selector match, or fewer live members than minWorkers, fails the job with nothing dispatched.
  • A failing assert in one lane fails that lane and the job, and other lanes still run to completion.
  • self.worker resolves in step names, inputs, global arguments and guards.
  • A local run without serve is refused with the --server message.
  • forEach and plain-job behaviour is unchanged: every #4271 baseline row stays green, and the existing execution and resume suites pass unedited.
  • Resume and recover of a fan-out run are refused with the "next release" message, and nothing runs. --last-evaluated runs lanes like any other run.

Notes

  • New domain files and a new setter in src/domain/remote/: run all of integration/ before verifying, since architecture_boundary_test.ts is not a *_rules_test.
  • Before verifying, red-check the new integration tests against main. They must fail there on the run guard.
  • Keep the scheduler's decisions pure where possible (which lane starts next, what a lane's result is) and unit test them apart from dispatch.
02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 8 MOREREVIEW+ 22 MOREPR_LINKED+ 2 MORESESSION_SUMMARIZED

Shipped

11/8/2026, 12:58:22 AM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack7211/7/2026, 9:32:31 PM

Sign in to post a ripple.