Relationships
#4310 serve: give the definition index a change signal (file watcher with a sync barrier) so a write to a definitions directory costs O(1), not a stat of every file in it
Opened by zack · 11/2/2026
Follow-up to #4307.
serve's DefinitionIdIndex (src/infrastructure/persistence/definition_id_index.ts) validates by stat. When a definitions directory's stat moves, it re-lists the directory and stats every file in it. Deno.readDir returns no inode, and a save that renames a temp file over an existing name shows up only in that file's own stat. So each write to a type directory costs O(N) stats, where N is the number of definitions in it. For 2s after a write (RUN_INDEX_RACY_WINDOW_MS), every index check re-lists again, because sharing a listing would mean trusting a directory stat younger than the filesystem's timestamp resolution.
#4307 makes each re-list cheaper (only changed entries are re-indexed, and stats run sync in time slices), but the cost per write is still proportional to the directory size. A workflow that gives each run its own behaviorName keeps growing the auto-definitions directory, so serve CPU per run still grows with the number of runs ever made, just with a smaller slope.
Direction
Give the index a change signal so one write costs O(1):
- Watch the definition roots (Deno.watchFs) and mark only the paths in events as dirty.
- Add a sync barrier before trusting the watcher: write a cookie file and wait for its event, as watchman does. Otherwise a check that starts after a write completes but before the write's event arrives would answer stale, and a lookup by id that misses in the index is final since #4280.
- Fall back to the current stat-based re-list on watcher overflow, error, or an unsupported filesystem (network filesystems, inotify watch limits).
Constraints
- A miss must stay correct for writes from any process: the CLI, workers, and v3 datastore pulls, not only serve's own saves.
- Must hold on Linux, macOS and Windows.
Open
No activity in this phase yet.
Sign in to post a ripple.