Relationships
#4504 A worker enrolling into a waiting queue can get its step before it can accept it, failing the step
Opened by skunk-ape · 11/7/2026· Shipped 11/8/2026
When a step is queued waiting for a worker, a worker that enrolls can receive the step before it is ready for it. The dispatch fails with No handler registered for method 'worker.dispatch', and the step fails.
Why it happens
The gateway's #handleEnroll (src/serve/worker_gateway.ts) calls onWorkerEnrolled and onWorkerIdle, which wake queued dispatches, before it returns the enroll reply. The worker (src/worker/connect.ts) registers its worker.dispatch handler only after that reply arrives. A dispatch that wins that race reaches a channel with no handler, and RpcChannel answers unknown_method.
Evidence
integration/remote_execution_test.ts, "elastic queueing — empty pool queues, worker enrolls, step dispatches", failed with this error within about 400 ms:
- Sharded test runs (4 processes × 8 workers, 8 × 4): 3 of 4 runs failed.
- Today's layout (1 process × 32 workers): 0 of 3 runs failed.
The log from a failing run shows the gateway dispatching at .759 and the worker logging that it enrolled at .764. Heavy contention in the current layout seems to hide the ordering, so less contention brings it out. This has only been observed in that test, not in a real deployment, but the code path is the production one.
Expected
A step queued for a worker dispatches once the worker can accept it, and a worker enrolling into a waiting queue runs the step.
Shipped
Click a lifecycle step above to view its details.
Sign in to post a ripple.