Relationships
#4147 model.output.search response phase takes 3.9s for 5,000 items with no attribution (redaction vs serialization vs send)
Opened by zack · 10/29/2026
Problem
The response phase of model.output.search took 3,901ms to return 5,000 items. It runs after search and authorization and covers the limit slice, redactWorkflowLinks(...) with nestedRunReadDecider, and serializing and sending the payload (src/serve/handlers/model_handlers.ts:~2045). The span reports only owner_lookups: 14 and results: 5000, so we can't tell how much is redaction versus JSON encoding versus socket send.
Evidence: ops.swamp-club.com Behaviors page (/dashboard/behaviors), trace 5fe27ec0852f537cc6fac88c0166ec4b (2026-10-29T22🔞07Z), build 20261029.200213.0-sha.c2ed52ab, Axiom dataset swamp-serve-traces. The model.output.search request (since: "7d", limit: 5000) took 80,380ms: output search 53,743ms (candidate_scan 33,088 + definition_names 19,297 + workflow_names 224) + authorization 22,736 + response 3,901.
Proposal
- Add counters or child timings to the
responsephase (redaction decisions, runs looked up, payload bytes, serialize ms, send ms) to attribute the 3.9s. - Then address the dominant cost. Likely candidates: per-item awaits in
redactWorkflowLinksthat could be batched or memoized perworkflowRunId, and the size of the 5,000-row payload (for the Behaviors page, the server-side usage aggregate filed alongside this one removes most of it).
Expected impact
Low priority on its own. Mostly superseded for the Behaviors page by the usage aggregate. Worth having the attribution so other large output searches don't hide the same cost.
Acceptance
The response span attributes its time to redaction, serialization and send, and a 5,000-item response completes in well under 1s.
Open
No activity in this phase yet.
zack commented 10/29/2026, 10:26:47 PM
Sibling issues from the same ops trace (5fe27ec0852f537cc6fac88c0166ec4b, Behaviors page model.output.search, 80.4s):
- #4141 definitions read twice (names + authorization)
- #4142 authorize from path/definition before reading outputs
- #4143 newest-first scan, stop at limit
- #4144 Behaviors page: server-side usage aggregate instead of 5,000 rows
- #4145 output summary index
- #4146 long scans starve the event loop
- #4147 response phase attribution (3.9s)
Suggested order: #4141 (small, ~19s) → #4144 (fixes wrong counts) → #4142 + #4143 → #4145 → #4146 → #4147.
Sign in to post a ripple.