Skip to main content
← Back to list
01Issue
BugOpenSwamp CLIPublic
AssigneesNone

Relationships

#3404 A manual worker verify racing an enrollment probe can leave a healthy worker unverified until it reconnects

Opened by hammz · 10/13/2026

Found by the adversarial review while verifying the fix for #3388 (commit ccfb960a). Read from the code, not reproduced.

Summary

Affects swamp serve with verify-on-enroll, on a worker with capacity 1.

  1. A worker enrolls. Its enrollment verification starts, and the probe does asynchronous setup in DispatchService.executeRemote before it reaches WorkerGateway.dispatch and takes a slot.
  2. In that window an operator runs swamp worker verify. The manual probe reaches gateway.dispatch first and takes the only slot (a probe is accepted on an unverified worker).
  3. The enrollment probe is refused as at capacity. Because it runs with skipScheduler it is not retried, and verification records unverified with an at-capacity reason.
  4. The manual probe ends. Nothing re-probes the worker, because the re-probe added in #3388 only fires when the dispatch that freed the slot was sent on a socket the worker has since replaced.
  5. The worker stays unverified until its socket drops and it reconnects.

Before #3388 the manual probe ending flipped the worker to idle by accident, which was the defect that issue fixed.

Expected

A worker whose enrollment probe was refused only for lack of a slot is probed again when the slot frees.

Suggested direction

Decide the re-probe on why verification failed rather than on the socket changing: remember on the pool entry that a probe was refused for capacity, and in the dispatch finally start enrollment verification when the worker is unverified, that flag is set and no slots are held, whichever socket the ending dispatch used. This replaces the channelAtStart check. Add a test where a manual probe holds the slot while the enrollment probe is refused.

  • A probe answered on a half-open old socket after a reconnect can promote the worker without the new socket being probed.
  • The deferred re-run in startEnrollmentVerify checks that the channel is present but not that it is open, unlike the trigger in dispatch.
  • With capacity 2 or more, a reconnect now waits for a probe hung on the old socket instead of probing the new socket alongside it.
  • swamp worker list hints to run swamp worker verify to diagnose an unverified worker, but neither output says a reconnect is needed to restore it.
02Bog Flow
◉OPEN○TRIAGED○IN PROGRESS○SHIPPED

Open

10/13/2026, 9:26:30 PM

No activity in this phase yet.

03Sludge Pulse

Sign in to post a ripple.