Relationships
#1310 serve: workflow.approvals and workflow.run.search still full-scan the workflow-run store after the #1284 data.query pushdown
Opened by ehazlett · 7/21/2026· Shipped 7/21/2026
Summary
The read-scan fix from Lab #1284 landed for data.query — its zero-row floor is gone and predicate matches are now fast. But the sibling workflow.* read ops, workflow.approvals and workflow.run.search, were not covered: they still linearly scan every workflow run and CEL-filter after, so they pay a ~500ms floor regardless of how few rows match. Any client that polls workflow.approvals eats a full scan on every cycle, and the cost grows with the total number of runs rather than with the number of pending approvals.
Relationship to #1284
#1284 reported data.query, workflow.approvals, and workflow.run.search all scanning. The shipped fix resolved data.query (confirmed below) but left the two workflow.* ops unchanged — this issue tracks that remaining half.
Environment
- swamp
20260720.234931.0-sha.4770b0fd servewith token auth, single repo, ~600 total data records, 38 workflow runs (1 suspended at a gate).
Measurements
Server-side cost, isolated by subtracting a constant client/transport baseline; each op run several times, median reported.
| op | rows returned | before (#1284, swamp 20260718) |
after (swamp 20260720) |
|---|---|---|---|
data.query, zero-row predicate |
0 | ~346ms | ~0ms (fixed) |
data.query, match-all |
~600 | ~490ms | ~130ms (fixed) |
workflow.run.search, all runs |
38 | ~486ms (37 runs) | ~545ms (unchanged) |
workflow.approvals |
1 | ~500ms | ~528ms (unchanged) |
Why this reads as an unfixed scan
workflow.approvals returns 1 row but costs the same (~528ms) as workflow.run.search returning all 38 (~545ms). The cost tracks rows scanned, not rows returned — the same "load and evaluate every record, then filter" signature #1284 described for data.query, on a store the pushdown did not reach. As the run store grows, both ops get slower even when the pending-approval count stays at one.
Impact
A client that lists pending approvals must either poll workflow.approvals and pay a full scan per cycle, or front it with a client-side cache. The floor scales with total run history, so it degrades over time.
Suggested direction
Extend the #1284 predicate pushdown / index to the workflow-run store: an index for the common shapes (run status such as suspended, workflow name), and short-circuit cheap top-level equality before materializing full run bodies — so workflow.approvals costs proportional to pending gates, not to total runs.
Repro
- Populate a repo with a few dozen workflow runs, one suspended at a manual-approval gate.
- Over a
serveconnection, timeworkflow.approvals(returns the 1 suspended run) andworkflow.run.search(returns all runs). - Observe both cost ~500ms and are roughly equal despite very different result sizes; compare against a fast
data.queryon the same connection to confirm the floor is specific to theworkflow.*read path.
Related
- Lab #1284 (the
data.queryhalf — shipped) - Lab #1285 (idle serve event-loop CPU — aggravates the tail latencies here, separate root cause)
Shipped
Click a lifecycle step above to view its details.