#4037 Serve globally scans definitions twice per new direct-type step, delaying worker dispatch by minutes

Summary

Serve's direct-type workflow-step resolver performs a global definition-name lookup even though the requested model type is already known. When the name is new, that lookup recursively reads and parses the primary and auto-definition trees. Auto-creation repeats the lookup inside its creation lock.

Concurrent steps with distinct new names therefore perform independent full-repository searches. In the investigated run, 36 steps started together but their definitions were not published until approximately 204–205 seconds later. Worker acquisition after that preparation was effectively immediate.

The same global lookup path remains on main at 0f682899. This is a Swamp core runtime issue, not a request to change UAT workflows or fixtures.

Environment and evidence identifiers

Mechanism

serve directTypeResolver
  → resolveOrCreateDefinition
    → lookupDefinition(newName)
      → findDefinitionByIdOrName
        → findByNameGlobal
          → recursively search primary definitions
          → recursively search auto-definitions
    → acquire name-specific creation lock
    → lookupDefinition(newName) again
    → save new definition
  → subsequent method preparation and worker dispatch

Code on inspected main:

• src/serve/deps.ts:204–211: supplies the global lookup.
• src/libswamp/models/direct_execution.ts: lookup calls at lines 567 and 722.
• src/infrastructure/persistence/yaml_definition_repository.ts:639–689: findByNameGlobal().
• The same repository's searchDefinitionByName(), starting at line 752, recursively reads YAML and constructs definitions until it finds a match.

Existing path hints help previously resolved names. They do not help a genuinely new name: proving absence requires walking both trees. For 36 distinct names that remain absent until creation, this requests 72 full searches. This is a code-path count, not a count inferred from Bitbison records.

Axiom evidence: delay precedes worker acquisition

For the 36 shard steps:

A separate readiness example shows the same shape: ready-cli-1 starts at 21:27:52.038; model-lock acquisition starts at 21:30:17.095, 145.06 seconds later, and takes only 31.7 ms.

These timings localize the large delay to preparation before those locks and worker dispatch, rather than waiting for an available worker.

The bounded shard-launch trace export returned 397 of 397 matching spans. Reproduce in Axiom with the trace above, service swamp-serve-vm, time range 21:32:22–21:36:30 UTC, selecting swamp.workflow.step, swamp.lock.acquire, swamp.dispatch.*, swamp.step.*, and swamp.remote.dispatch.

Bitbison evidence: serve reads definitions during the gap

Restricting reads to the confirmed serve process and auto-definition paths:

SELECT toStartOfMinute(ts_ms) AS minute,
       count() AS read_records,
       uniqExact(node_path) AS distinct_paths
FROM edges
WHERE stream_id = 'b5257a30-7297-4632-9030-6292d077b26c'
  AND ts_ms >= '2026-10-26 21:32:22'
  AND ts_ms < '2026-10-26 21:36:30'
  AND op = 'read'
  AND node_path LIKE '/var/lib/swamp/repo/.swamp/auto-definitions/%'
  AND if(tgid_surrogate > 1, tgid_surrogate, thread_surrogate)
      = toUInt128('554601050635676792091667423413')
GROUP BY minute ORDER BY minute

Bitbison also records 36 distinct final definition paths published by rename, between 21:35:46.902 and 21:35:48.159. Paths match:

/var/lib/swamp/repo/.swamp/auto-definitions/@swamp/ci-uat/runner/
ut-5b5b8600-5e0d-4bb5-bb8b-1fc326eaf8ad*

The publication interval falls immediately before the model-lock and dispatch activity seen in Axiom.

Interpretation limits: Code establishes the repeated full-scan mechanism; host telemetry confirms substantial definition-reading activity by serve during the preparation gap. The evidence does not assign every millisecond of that gap exclusively to lookup, nor establish CPU or disk saturation. Bitbison read records are not exact syscall counts.

Impact

• Ready workers cannot receive work until serve finishes definition preparation.
• New-name resolution becomes more expensive as unrelated definitions accumulate.
• Concurrent new names multiply repository work rather than sharing discovery.
• A nominally lightweight readiness operation can spend minutes before reaching a worker.
• Adding workers does not address this preparation cost.

Latest-main status and related fix

PR #3394 / 470a198a introduces an auto-definition name index for type-specific findByName() used during worker enrollment (Lab #3984). This does not replace this global lookup: serve/deps.ts still calls findDefinitionByIdOrName(), and findByNameGlobal() still recursively searches on a miss.

Requested change

Make repeated global name resolution, including missing names, reuse repository discovery instead of reparsing every definition per caller.

Preserve global-name semantics, primary-over-auto precedence, external-edit visibility, and the locked recheck that prevents duplicate creation. A type-specific shortcut must not silently change those semantics.

Acceptance criteria

• Benchmark 36 concurrent distinct new names against approximately 15,000 definitions.
• Separate cold index construction from subsequent lookup costs.
• Demonstrate that warm misses and locked rechecks do not each reread the entire repository.
• Cover concurrent creation of the same name, additions/deletions/renames, primary shadowing, and unreadable definitions.
• Record lookup duration, files read, and step-start-to-dispatch latency.
• Re-run the production-shaped workload to quantify improvement; do not treat the historical 204-second gap as guaranteed recoverable time.