#3293 A running dispatch counts twice against worker capacity, so a --concurrency 2 worker queues a step while it has a free slot

Severity: medium — a worker's usable concurrency is roughly halved for steps that arrive while another dispatch is running (a --concurrency 2 worker runs them one at a time), and a step queued this way fails with a queue timeout (exit 1) if the running dispatch outlasts --queue-timeout. No data is lost and the failed step can be resumed, but runs are slower than configured and automation sees spurious failures.

Affects: swamp 20261009.215038.0-sha.986f9022; the UAT test below still fails the same way on 20261012.141205.0-sha.58191ab3 (at the time of writing, source HEAD dbd7ba9a still has it: src/serve/dispatch_service.ts, src/serve/worker_gateway.ts and src/domain/remote/scheduler.ts are identical between 986f9022 and dbd7ba9a)

Summary

swamp serve schedules a placed step onto a worker only if the worker's active dispatches plus the orchestrator's own "reservations" are below its capacity. A reservation is taken when a worker is chosen and is not released until the dispatch has finished, while the same dispatch is also listed in the worker's activeDispatchIds for its whole duration. Every running dispatch is therefore counted twice.

On a worker connected with --concurrency 2, one running dispatch makes the scheduler see 2 of 2 slots used. A second step with matching labels is queued ("queued, waiting for worker matching labels ...") even though swamp worker list shows 1/2 slots in use. It is dispatched only after the first dispatch ends, or fails with "Timed out waiting for a worker ..." if the queue timeout expires first.

Two steps scheduled at the same instant do run concurrently (neither is in activeDispatchIds yet when the second is scheduled, so the count is 0 + 1 reservation), which is why this is only visible for staggered arrivals.

Steps to reproduce

Needs swamp, curl and python3. Runs everything in a scratch directory.

W=$(mktemp -d); mkdir -p "$W/repo" "$W/cache"; cd "$W/repo"
swamp init >/dev/null
swamp vault create local_encryption main >/dev/null
mkdir -p models/command/shell workflows

mk() {  # mk <suffix> <model id> <workflow id>
cat > models/command/shell/$2.yaml <<Y
type: command/shell
typeVersion: 2026.02.09.1
id: $2
name: long-step-$1
version: 1
tags: {}
globalArguments:
  run: "sleep 6"
methods:
  execute:
    arguments:
      run: "sleep 6"
Y
cat > workflows/workflow-$3.yaml <<Y
id: $3
name: long-$1
tags: {}
jobs:
  - name: long-job
    steps:
      - name: long-step
        task:
          type: model_method
          modelIdOrName: long-step-$1
          methodName: execute
        labels:
          pool: worker
        dependsOn: []
        weight: 0
    dependsOn: []
    weight: 0
version: 1
Y
}
# Two different models, so the model lock does not serialise the two runs.
mk a bd83a14c-75bd-43a0-bb96-5186061dd5d5 fc786c30-fb49-425e-9803-525237cd9c47
mk b 1d83a14c-75bd-43a0-bb96-5186061dd5d6 2c786c30-fb49-425e-9803-525237cd9c48

PORT=$(python3 -c 'import socket;s=socket.socket();s.bind(("127.0.0.1",0));print(s.getsockname()[1])')
swamp serve --no-schedule --port $PORT > "$W/serve.log" 2>&1 &
SERVE=$!
until curl -s -o /dev/null http://127.0.0.1:$PORT/; do sleep 0.2; done

TOKEN=$(swamp worker token create t1 --duration 1h --json \
  | python3 -c 'import sys,json;print(json.load(sys.stdin)["token"])')
swamp worker connect ws://127.0.0.1:$PORT --token "$TOKEN" --json \
  --label pool=worker --concurrency 2 --cache-dir "$W/cache" > "$W/worker.log" 2>&1 &
WORKER=$!
until grep -q enrolled "$W/worker.log"; do sleep 0.2; done

# Run 1, and wait until the worker has actually started it.
swamp workflow run long-a --server http://127.0.0.1:$PORT > "$W/run1.out" 2>&1 &
until grep -q dispatch_started "$W/worker.log"; do sleep 0.1; done
swamp worker list            # 1/2 slots in use

# Run 2 arrives while run 1 is still running.
swamp workflow run long-b --server http://127.0.0.1:$PORT; echo "run2 exit=$?"
wait %3 2>/dev/null
grep -o '"kind":"dispatch_[a-z]*"' "$W/worker.log"

kill $WORKER $SERVE

To see the failure rather than the delay, start the server with swamp serve --no-schedule --queue-timeout 5s --port $PORT.

Expected behaviour

With one of two slots in use, the second step is dispatched immediately: it is never queued, both dispatches overlap (worker events dispatch_started, dispatch_started, dispatch_finished, dispatch_finished), and run 2 finishes about 6s after it started.

What establishes it:

• design/enablers/remote-execution.md:909-911: "A worker accepts up to capacity concurrent dispatches (--concurrency N on worker connect, default 1). When all slots are full, a further dispatch is rejected with worker_busy and the orchestrator re-queues it." Queueing is specified only for the all-slots-full case.
• design/enablers/remote-execution.md:981-985 (phase 4b): "The scheduler picks the worker with the most free slots".
• src/domain/remote/scheduler.ts:148-152: "dispatch names an eligible worker with available capacity; queue means every eligible worker is at capacity, or no connected worker matches the placement at all."
• swamp worker connect --help: "--concurrency <n> - Number of concurrent dispatch slots".
• The reservation field's own comment, src/serve/dispatch_service.ts:152: "Per-worker reservation count for queued steps not yet dispatched" — a reservation is meant to cover the gap before the dispatch is registered, not the dispatch's lifetime.
• swamp worker list itself reports 1/2 slots while the second step sits in the queue.

Actual behaviour

Default queue timeout (10m): run 2 is queued behind run 1 and the two run back to back.

--- worker list while run 1 is dispatched
NAME  STATUS  SLOTS  LABELS       PLATFORM/ARCH  LAST SEEN
t1    busy    1/2    pool=worker  linux/x86_64   2026-10-10T14:29:05.251Z
--- worker list 2s after run 2 was started
NAME  STATUS  SLOTS  LABELS       PLATFORM/ARCH  LAST SEEN
t1    busy    1/2    pool=worker  linux/x86_64   2026-10-10T14:29:05.251Z
run1 exit=0 at +8s
run2 exit=0 at +14s
--- run2 output
   system │ Starting workflow long-b · 14:29:06 UTC

 long-job │ start 14:29:06 UTC
 long-job │ step long-step · long-step-b · execute · start 14:29:06 UTC
 long-job │ queued, waiting for worker matching labels pool=worker
 long-job │ Still running... [10s elapsed]
 long-job │ done long-step in 11.9s · 14:29:18 UTC
 long-job │ completed in 11.9s · 14:29:18 UTC

   system │ Completed workflow long-b succeeded in 11.9s · 14:29:18 UTC
--- worker dispatch events
dispatch_started long-a
dispatch_finished
dispatch_started long-b
dispatch_finished

With swamp serve --queue-timeout 5s: run 2 fails with exit code 1 and the worker never receives it.

run2 exit=1
 long-job │ step long-step · long-step-b · execute · start 14:29:28 UTC
 long-job │ queued, waiting for worker matching labels pool=worker
 long-job │ failed long-step in 5.0s · 14:29:33 UTC
 long-job │ failed · 14:29:33 UTC

   system │ Failed workflow long-b in 5.0s · 14:29:33 UTC

   system │ Timed out waiting for a worker matching labels pool=worker to become available
--- worker dispatch events
dispatch_started long-a
dispatch_finished

Root cause

Paths and lines at 986f9022 (unchanged at dbd7ba9a).

• src/serve/dispatch_service.ts:740-743 (#acquireWorker): when scheduleStep returns dispatch, #reserved[worker] is incremented.
• src/serve/dispatch_service.ts:380-449: the reservation is decremented only in the finally of the try that awaits this.#dispatchOnce(workerName, request) (line 381), i.e. after the dispatch has completed, not once it has been handed to the gateway.
• src/serve/worker_gateway.ts:466: WorkerGateway.dispatch pushes the same dispatch into entry.activeDispatchIds for its duration.
• src/serve/dispatch_service.ts:763-779 (#poolSnapshot): returns activeDispatchIds plus one __reserved_<i> entry per reservation, so a running dispatch contributes two entries.
• src/domain/remote/scheduler.ts:162-176 (scheduleStep): filters on activeDispatchIds.length < capacity; with 2 entries against capacity 2 it returns queue.

In general a worker of capacity N accepts a newly arriving step only while fewer than N/2 dispatches are running.

The counted reservation and the overlay in #poolSnapshot were introduced by 700203dc "feat(remote): capacity slots and concurrent dispatch (phase 4b) (#1769)". Before that (f625a295) #reserved was a set that only affected workers whose status was still idle, so it stopped counting once the gateway marked the worker busy.

A fix was not attempted. One direction consistent with the field's comment is to release the reservation as soon as the gateway has recorded the dispatch in activeDispatchIds, rather than when #dispatchOnce returns.

How the UAT found it

Test file tests/cli/worker/dispatch/concurrency_test.ts, test "swamp worker dispatch gives a free slot to a step arriving mid-dispatch". It fails against 986f9022 at line 165 (Run 2 failed: ... Timed out waiting for a worker matching labels pool=worker to become available, exit code 1 instead of 0).

Before the audit this test did not exist. The file's two --concurrency 2 tests both started the two workflow runs in the same instant, which is the one case that works, and asserted only weakly: the first checked that both runs exited 0 and that two dispatch_finished events were seen (true whether the dispatches overlapped or ran one after the other); the second slept 500ms and asserted only that the worker's status was not idle. Both also ran the same workflow on the same model twice, so the model lock serialised the runs regardless of capacity. Nothing exercised a step arriving while a slot was already occupied.

The audit (commits 549e26c and bd29a8b on the UAT branch) switched the fixtures to two workflows on two different models, replaced the count checks with an assertion on the order of dispatch_started / dispatch_finished events, and added the new test. It starts run 1, waits for the worker's dispatch_started event, checks swamp worker list --json reports capacity 2 with exactly one active dispatch, then starts run 2 and asserts that both runs exit 0, that run 2's output never contains "queued, waiting for worker", and that the worker's events are started, started, finished, finished. The harness starts swamp serve with --queue-timeout 5s, which is why the test sees a failure rather than a delay; the not-queued and event-order assertions would fail with the default timeout as well.

The sibling tests "runs two dispatches concurrently with --concurrency 2" (simultaneous start) and "queues the N+1th dispatch at capacity" pass against the same binary.

Run it from the swamp-uat repo:

deno test --allow-read --allow-write --allow-env --allow-run --allow-net \
  --filter "mid-dispatch" tests/cli/worker/dispatch/concurrency_test.ts