Relationships
#4056 Bounded behavior history search still sequentially scans filesystem output records before limiting results
Opened by zack · 10/27/2026· Shipped 10/28/2026
Problem
Swamp core's filesystem behavior-output search performs work proportional to stored history even when a caller requests a bounded recent result set. Investigate and optimize this path for the behaviors dashboard and other output-history callers.
Source evidence
In src/infrastructure/persistence/yaml_output_repository.ts, YamlOutputRepository.findAllGlobalSince iterates registered types and method directories, sequentially stats output YAML files, then reads/parses files passing its mtime prefilter before checking startedAt. Old files still incur directory enumeration and stat work.
src/libswamp/models/output_search.ts materializes candidates before enrichment and sorting. In src/serve/handlers/model_handlers.ts, handleModelOutputSearch passes limit: undefined so authorization can precede the final result limit. Therefore a result cap does not bound the scan or enrichment. Preserve the authorization guarantee when optimizing.
This is distinct from shipped Lab #3959 (repeated missing-definition scans) and #3960 (refresh starvation). Their fixes already batch definition resolution and preserve pending refresh results; the sequential history scan remains.
Investigation and reproduction
Use a v2 filesystem repository containing many method-output records across types/methods, with both old and recent records. Request history through serve with since=7d and limit=5000. Grow the inventory while holding requested result count fixed, and measure directory enumeration, stat, read and parse counts and elapsed time. Compare isolated requests with concurrent workflow activity.
Instrument candidate scanning, definition/workflow-name resolution, authorization and response construction separately. The source establishes the unbounded sequential work, but this report does not establish the fraction of end-to-end latency attributable to that stage. A representative benchmark is needed before claiming a particular speedup.
Expected behavior / acceptance
Use an indexed or summary query, or another measured approach, to avoid a full sequential history walk for bounded recent reads where possible. Demonstrate reduced file operations and latency as inventory grows. Preserve newest-first ordering, since/until semantics, authorization before final limiting, and concurrent-write/delete behavior. Add phase timings and counts to identify remaining costs. Do not simply slice before authorization or parallelize file reads without a concurrency bound.
Ownership: Swamp core/serve, not an extension.
Shipped
Click a lifecycle step above to view its details.
Sign in to post a ripple.