#4280 Lookup by id of an auto-definition parses every file in its type directory; serve CPU grows with run count

Summary

YamlDefinitionRepository.locateByIdInDir falls back to a slow path that reads and YAML-parses every file in a type directory until one declares the requested id. Auto-definitions are saved as <name>.yaml, not <id>.yaml, so every lookup by id of an auto-definition takes the slow path. The type directories grow without bound when a workflow gives each run its own behaviorName (recommended so forEach items and concurrent runs do not serialize on one model lock). Nothing removes old auto-definitions, so the per-lookup cost grows with the total number of runs ever made.

Version

swamp 20261031.063736.0-sha.e459d219, swamp serve --remote-only --hot-reload --auth-mode token, filesystem datastore, Linux x86_64.

Measurements (one serve, 2026-11-02)

• serve CPU 120–124% steady (about 65% user, 55% system), 81% on the main JS thread, while the serve log shows almost nothing (about 50 lines per 10 minutes).
• Auto-definition type directories: @swamp/ssh 13,595 files (oldest from 2026-10-31, growing every run), command/shell 1,940, @nckrtl/certificate-script-assets 1,536.
• A 4-second strace of serve: 10,798 statx calls on .swamp/auto-definitions/@swamp/ssh, 25 getdents64 listings of that directory, and repeated full reads of route-cert-assets-<uuid>.yaml and observation-*.yaml auto-definition files.
• A 3-second strace: 833 realpath-style walks (each about 12 readlink calls, about 3,000 failing readlink/s); 48% resolve paths under .swamp/outputs/swamp/worker/set_status (24,215 files at the time).
• Retention is configured and works (swamp doctor datastores --retention: no stored item older than its retention). Running swamp run gc by hand removed 123 runs, 36,623 outputs and 62 MB and cut statx calls about 10×, but serve CPU stayed at 120–124%. So the cost is not old runs; it is lookups that scale with directory size.

Code path

locateByIdInDir (definition repository): the fast path tries ${id}.yaml; on a miss it does for await (const entry of Deno.readDir(typeDir)), then Deno.readTextFile + parseYaml + Definition.fromData for each entry until the id matches. For auto-definitions the fast path always misses.

Expected

A lookup by id of an auto-definition costs O(1), for example through the id index serve already keeps for repository definitions, or by also storing auto-definitions under their id. A retention setting (or GC phase) for auto-definitions whose owning runs and data are gone would bound the directories as well.

Reproduce

1. Create a workflow whose step uses behaviorType with behaviorName: '${{ "x-" + run.id }}' (one auto-definition per run).
2. Run it a few thousand times under swamp serve.
3. Watch serve CPU and strace -f -e trace=statx,openat -p <serve pid> while any later run of that behavior type executes: each lookup by id lists and parses the whole .swamp/auto-definitions/<type>/ directory.