#3984 serve: enrolling a new worker name scans every stored worker definition twice inside the serialized gateway tail, so bursts of new workers time out and block existing workers' status, re-enrollment and dispatch

Summary

When a worker enrolls under a new name, serve scans every stored swamp/worker auto-definition twice, inside the serialized gateway transition tail. Cost is linear in the number of stored worker definitions. A burst of new workers (ephemeral workers without a stable --cache-dir, so every process is a new fleet member name) queues serially behind these scans. That queueing causes:

• 30 s client RPC timeouts;
• control socket closed before enrollment began and control socket closed during enrollment;
• retries, and then the worker giving up after three consecutive failures.

Workers that are already enrolled are delayed too: their disconnect records, re-enrollments and dispatches wait behind the same tail.

Production telemetry (one day, version b4dd513d):

• 6,031 enrollment attempts: 5,680 closed-before, 212 closed-during, 139 OK.
• Successful attempts waited 25 s (median) in the gateway queue.
• The elapsed gap between swamp.model.method.run and swamp.model.method.setup was 99% of worker enroll method time and 95% of gateway transition execution.
• First-observed names had an 18.8 s median; repeat names had a 97 ms median.
• One correlated request: 27.8 s in the queue; the client timed out 1.8 s after the method began; the server kept working 240 s more; the method body took 132 ms.

Root cause (reproduced locally on main 4fe6f225d and on b4dd513d)

The worker run deps use type-scoped lookup (definitionType: swamp/worker). For a name with no definition yet, YamlDefinitionRepository.findByName misses the primary name path and runs findAll(type) over models/swamp/worker. It then misses the secondary name path and runs findAllInDir over every file in .swamp/auto-definitions/swamp/worker, reading and YAML-parsing each one. resolveOrCreateDefinition then repeats the whole lookup under the auto-definition lock before saving. Total: about 2·N reads per new name. An existing name takes one direct read.

An in-process profile, built exactly as WorkerGateway#defaultRunModelMethod builds the run (medians of 5):

The scan is CPU-bound: process CPU ≈ elapsed. Lock wait and save are negligible. Inside real serve the cost was ~25 µs per definition read on a fast desktop with a warm page cache.

End-to-end reproduction (real swamp serve and real swamp worker connect processes)

Setup: filesystem datastore (v2); a fleet token (--max-enrollments) minted locally; N synthetic auto-definitions written to .swamp/auto-definitions/swamp/worker/ in the same format save() writes; 3 long-lived workers with a stable --cache-dir; then K new workers launched at once, each with a fresh cache dir. Traces were collected through OTEL_EXPORTER_OTLP_ENDPOINT into a local receiver. No delays were injected.

Observed in these runs:

• The queue is strictly serial. Waits step by one transition's duration (0 → 2.6 → 5.0 → … → 27 s), with queue_depth up to 26.
• Closed during enrollment. A worker whose turn came just before its 30 s timeout submitted rpc.cancel and closed its socket. The server finished the scan anyway, up to 8.8 s past the timeout, and the transition ended closed during enrollment. Its definition was created, so its retry took 13 ms.
• Closed before enrollment began. Requests queued behind it were skipped (0.2 ms each), but their names were never created, so every retry pays the full double scan again.
• Retries and give-up. Retry delays were 1 s, then 2 s, then: connection failed 3 consecutive times before enrollment — giving up.
• Cancellation is not observed. The enroll RPC handler never passes ctx.signal to #handleEnroll. It checks state.channel.closed only before redeem and after the durable enroll, so the scan runs to completion after the client has gone.
• Existing worker, durable status. After a SIGKILL, its worker_disconnected waited ~30 s at depth 12–27. swamp worker list kept reporting it idle for ~31 s.
• Existing worker, re-enrollment. Its re-enrollment, about 13 ms of real work, took 59–61 s: one timeout plus queueing behind the new workers' retries.
• Dispatch to an idle existing worker. A one-step workflow placed on it went from 0.74 s to 30.4 s during the burst. dispatch() awaits worker_busy on the gateway tail (it waited 29.7 s at depth 26) before sending the dispatch. Step-lease operations on the separate dispatch tail waited 0 ms.
• Control. Skipping only the secondary full scan (not a valid fix, since it would miss legacy UUID-named auto-definitions) made the same 50k / 24 burst finish in 1.65 s with no timeouts. That makes the scan causal locally.

What is not yet proven for production

At the local per-file cost, production's 18.8 s median would need ~190k stored worker definitions, and its 242 s case ~4.8M. So production also needs a slower per-file cost (VM CPU or disk) or concurrent load on the same event loop. The new-name-only signature matches production: set_status, redeem and repeat enroll stay at single-digit ms all day on the same tail, while first-observed enrolls grow from ~3–4 s in the early morning to 15–30 s medians later. Not accounted for: the wide variance within an hour (15 s median vs 136 s max). Needed from production:

• the counts of .swamp/auto-definitions/swamp/worker/*.yaml and models/swamp/worker/*.yaml over the day;
• host CPU, and whether workers share the serve host;
• spans for lookup, scan, lock and save inside the pre-setup gap.

Fix direction (high level)

1. Remove the O(N) miss for type-scoped lookups. For example, a per-type name→path index built once and maintained on save and delete, or a single scan reused for the re-check under the lock. Keep correctness for UUID-named and mismatched files.
2. Keep definition resolution/creation off the serialized gateway tail. Do it after redeem, so no definition is created for an unauthenticated request, and outside the tail, so the tail holds only the cheap durable worker, token and lease writes. The auto-definition lock already serializes creation per name.
3. Honor cancellation. Pass the handler's abort signal into enrollment; check it, or the socket state, after redeem and before the worker method run; make long lookups abortable. Verify that redeem's binding/allowance accounting stays correct when an attempt is abandoned.
4. Dispatch head-of-line blocking. dispatch() awaiting a durable worker_busy write on the shared gateway tail means any slow gateway transition delays work for idle workers.
5. Instrument the pre-setup gap: lookup (files read, scan or direct), lock wait, save.

A longer client timeout alone leaves the work on the tail; abandoned attempts still run to completion.

Related: #3405 (type-scoped lookup), #3755 (transition queue spans), #3786 (OTel attribution gaps).

Environment

• Production: swamp serve 20261025.114930.0-sha.b4dd513d, filesystem datastore, engine v2.
• Local reproduction: main 4fe6f225d and b4dd513d, Deno 2.9.8, Linux, loopback only, synthetic tokens and definitions.