Relationships
#3388 A worker that fails its enrollment probe becomes idle and schedulable in memory
Opened by zack · 10/13/2026· Shipped 10/13/2026
Found by the adversarial review while verifying the fix for #3294, then reproduced with a unit-level harness on 97ca429b (not reproduced against a running serve).
Summary
With verifyOnEnroll, a worker enrolls as unverified and the gateway sends it a probe dispatch (probeMarker set). The probe goes through WorkerGateway.dispatch in src/serve/worker_gateway.ts, which sets entry.status to busy and, in its finally, sets it to idle whenever the worker is not draining and has no active dispatches. It does not restore the status the worker had before the dispatch.
When the probe fails, the failure branches of #runEnrollmentVerify record verifyFailureReason and persist status unverified to the worker model, but never set entry.status back to unverified. The in-memory entry therefore stays idle, onWorkerIdle has already fired, and the scheduler (which excludes only unverified and draining workers) will place real steps on a worker that failed verification. The persisted record says unverified while the live pool says idle.
Reproduction
In the worker_gateway_test.ts harness: verifyOnEnroll true, verifyWorker dispatches a probe through gateway.dispatch, and the simulated worker answers the probe with status error. After the probe finishes the snapshot reads status idle with verifyFailureReason set, the idle callback has fired once, and a following dispatch without a probeMarker is accepted and sent to the worker.
The existing verifyOnEnroll failure tests stub verifyWorker without dispatching, so they never run the dispatch finally and do not catch this.
Expected
A worker whose probe fails stays unverified in memory as well as in its record, and is not offered to the scheduler.
Suggested direction
Capture the status before the slot is claimed in dispatch and restore it in the finally instead of assuming idle, or have the failure branches of #runEnrollmentVerify set entry.status back to unverified. Add a test whose verifyWorker really dispatches.
Relation to #3294
Separate, older defect. Before the #3294 fix a probe whose busy-status write failed left the worker stuck busy; after it, that path also ends idle, the same as a probe that reaches the worker and fails.
Shipped
Click a lifecycle step above to view its details.
Sign in to post a ripple.