Skip to main content
← Back to list
01Issue
BugOpenSwamp CLIPublic
AssigneesNone

Relationships

#4147 model.output.search response phase takes 3.9s for 5,000 items with no attribution (redaction vs serialization vs send)

Opened by zack · 10/29/2026

Problem

The response phase of model.output.search took 3,901ms to return 5,000 items. It runs after search and authorization and covers the limit slice, redactWorkflowLinks(...) with nestedRunReadDecider, and serializing and sending the payload (src/serve/handlers/model_handlers.ts:~2045). The span reports only owner_lookups: 14 and results: 5000, so we can't tell how much is redaction versus JSON encoding versus socket send.

Evidence: ops.swamp-club.com Behaviors page (/dashboard/behaviors), trace 5fe27ec0852f537cc6fac88c0166ec4b (2026-10-29T22🔞07Z), build 20261029.200213.0-sha.c2ed52ab, Axiom dataset swamp-serve-traces. The model.output.search request (since: "7d", limit: 5000) took 80,380ms: output search 53,743ms (candidate_scan 33,088 + definition_names 19,297 + workflow_names 224) + authorization 22,736 + response 3,901.

Proposal

  1. Add counters or child timings to the response phase (redaction decisions, runs looked up, payload bytes, serialize ms, send ms) to attribute the 3.9s.
  2. Then address the dominant cost. Likely candidates: per-item awaits in redactWorkflowLinks that could be batched or memoized per workflowRunId, and the size of the 5,000-row payload (for the Behaviors page, the server-side usage aggregate filed alongside this one removes most of it).

Expected impact

Low priority on its own. Mostly superseded for the Behaviors page by the usage aggregate. Worth having the attribution so other large output searches don't hide the same cost.

Acceptance

The response span attributes its time to redaction, serialization and send, and a 5,000-item response completes in well under 1s.

02Bog Flow
◉OPEN○TRIAGED○IN PROGRESS○SHIPPED

Open

10/29/2026, 10:26:21 PM

No activity in this phase yet.

03Sludge Pulse
Editable. Press Enter to edit.

zack commented 10/29/2026, 10:26:47 PM

Sibling issues from the same ops trace (5fe27ec0852f537cc6fac88c0166ec4b, Behaviors page model.output.search, 80.4s):

  • #4141 definitions read twice (names + authorization)
  • #4142 authorize from path/definition before reading outputs
  • #4143 newest-first scan, stop at limit
  • #4144 Behaviors page: server-side usage aggregate instead of 5,000 rows
  • #4145 output summary index
  • #4146 long scans starve the event loop
  • #4147 response phase attribution (3.9s)

Suggested order: #4141 (small, ~19s) → #4144 (fixes wrong counts) → #4142 + #4143 → #4145 → #4146 → #4147.

Sign in to post a ripple.