Skip to main content
← Back to list
01Issue
BugShippedSwamp CLIPublic
Assigneesskunk-ape

Relationships

#4504 A worker enrolling into a waiting queue can get its step before it can accept it, failing the step

Opened by skunk-ape · 11/7/2026· Shipped 11/8/2026

When a step is queued waiting for a worker, a worker that enrolls can receive the step before it is ready for it. The dispatch fails with No handler registered for method 'worker.dispatch', and the step fails.

Why it happens

The gateway's #handleEnroll (src/serve/worker_gateway.ts) calls onWorkerEnrolled and onWorkerIdle, which wake queued dispatches, before it returns the enroll reply. The worker (src/worker/connect.ts) registers its worker.dispatch handler only after that reply arrives. A dispatch that wins that race reaches a channel with no handler, and RpcChannel answers unknown_method.

Evidence

integration/remote_execution_test.ts, "elastic queueing — empty pool queues, worker enrolls, step dispatches", failed with this error within about 400 ms:

  • Sharded test runs (4 processes × 8 workers, 8 × 4): 3 of 4 runs failed.
  • Today's layout (1 process × 32 workers): 0 of 3 runs failed.

The log from a failing run shows the gateway dispatching at .759 and the worker logging that it enrolled at .764. Heavy contention in the current layout seems to hide the ordering, so less contention brings it out. This has only been observed in that test, not in a real deployment, but the code path is the production one.

Expected

A step queued for a worker dispatches once the worker can accept it, and a worker enrolling into a waiting queue runs the step.

02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 5 MOREREVIEW+ 9 MOREPR_LINKED+ 2 MORESESSION_SUMMARIZED

Shipped

11/8/2026, 1:03:51 AM

Click a lifecycle step above to view its details.

03Sludge Pulse
skunk-ape assigned skunk-ape11/7/2026, 8:20:01 PM

Sign in to post a ripple.