#1284 serve: read ops (data.query, workflow.approvals, workflow run.search) linearly scan the whole datastore — ~350–600ms floor regardless of match count

Summary

Over a serve connection, every scan-style read op costs ~350–600ms even when it matches zero or a handful of rows, because it appears to deserialize and CEL-evaluate every record in the datastore with no predicate pushdown or index. An indexed lookup (model.get) on the same connection is ~3ms, so the transport and dispatch are not the cost — the per-record scan is. A client that polls these reads therefore pays a full scan on every cycle.

Environment

• swamp 20260718.081020.0-sha.fc14abbd
• serve with token auth, single repo, ~581 total data records.

Measurements

A single persistent WS connection issues each op five times; medians below. The harness is a small polling probe (open one connection, time repeated reads).

Two things stand out:

1. A predicate matching zero rows still costs ~346ms. That is a floor paid on every call — consistent with "load + evaluate all N records, then filter," not an indexed/pushed-down query.
2. Adding limit: 5 to a predicate that matches early drops 570ms to 90ms, i.e. the scan short-circuits when the limit is satisfied. So the cost scales with records scanned, and there is no index to jump to matches.

Impact

A client that polls any of these reads pays the full scan on every cycle. One data.query per poll at a few-second interval turns a 6-row result into a steady stream of ~500ms full scans, and the cost grows with the datastore rather than with the result.

Suggested direction

• Predicate pushdown / an index for the common shapes: modelType, modelName, and tags.<key> equality.
• Short-circuit evaluation for cheap top-level equality before materializing full record bodies.

Repro

1. Populate a repo with a few hundred data records.
2. Over a serve WS connection, time data.query with a predicate that matches nothing vs. model.get for a single model.
3. Observe the ~350ms+ floor on data.query vs. ~3ms for model.get; add limit to a matching predicate and observe the scan short-circuit.

Related

Tail latencies here (the 3.4s workflow.approvals outlier) are aggravated by a separate idle-CPU issue in the serve event loop — filed separately.