Skip to main content
← Back to list
01Issue
BugShippedSwamp CLIPublic
Assigneeszack

Relationships

#3388 A worker that fails its enrollment probe becomes idle and schedulable in memory

Opened by zack · 10/13/2026· Shipped 10/13/2026

Found by the adversarial review while verifying the fix for #3294, then reproduced with a unit-level harness on 97ca429b (not reproduced against a running serve).

Summary

With verifyOnEnroll, a worker enrolls as unverified and the gateway sends it a probe dispatch (probeMarker set). The probe goes through WorkerGateway.dispatch in src/serve/worker_gateway.ts, which sets entry.status to busy and, in its finally, sets it to idle whenever the worker is not draining and has no active dispatches. It does not restore the status the worker had before the dispatch.

When the probe fails, the failure branches of #runEnrollmentVerify record verifyFailureReason and persist status unverified to the worker model, but never set entry.status back to unverified. The in-memory entry therefore stays idle, onWorkerIdle has already fired, and the scheduler (which excludes only unverified and draining workers) will place real steps on a worker that failed verification. The persisted record says unverified while the live pool says idle.

Reproduction

In the worker_gateway_test.ts harness: verifyOnEnroll true, verifyWorker dispatches a probe through gateway.dispatch, and the simulated worker answers the probe with status error. After the probe finishes the snapshot reads status idle with verifyFailureReason set, the idle callback has fired once, and a following dispatch without a probeMarker is accepted and sent to the worker.

The existing verifyOnEnroll failure tests stub verifyWorker without dispatching, so they never run the dispatch finally and do not catch this.

Expected

A worker whose probe fails stays unverified in memory as well as in its record, and is not offered to the scheduler.

Suggested direction

Capture the status before the slot is claimed in dispatch and restore it in the finally instead of assuming idle, or have the failure branches of #runEnrollmentVerify set entry.status back to unverified. Add a test whose verifyWorker really dispatches.

Relation to #3294

Separate, older defect. Before the #3294 fix a probe whose busy-status write failed left the worker stuck busy; after it, that path also ends idle, the same as a probe that reaches the worker and fails.

02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 5 MOREREVIEW+ 16 MOREPR_MERGED+ 2 MORESESSION_SUMMARIZED

Shipped

10/13/2026, 9:32:54 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
zack assigned zack10/13/2026, 8:25:35 PM

Sign in to post a ripple.