Relationships
#2985 data query --limit returns fewer live records than exist and reports limited: false when stale catalog rows fall inside the limit
Opened by hammz · 10/2/2026· Shipped 10/5/2026
Problem
swamp data query --limit N can return fewer than N records while more live matches exist, and still report the result as complete: limited is false in --json output and log mode prints the count without limit reached. A script that pages or checks limited is told it has everything while live matches are missing.
Reproduction
Released binary 20261002.172406.0-sha.0e793350, filesystem datastore:
- Create three command/shell models and run execute on each, so three resource artifacts named result exist.
- Delete one artifact's backing file: .swamp/data/command/shell//result/1/raw (leaves a stale catalog row).
- swamp data query 'name == "result" && dataType == "resource"' --json returns two records (the two live ones).
- The same query with --limit 2 --json returns one record, total 1, limited false. Log mode prints 1 result.
Cause
DataQueryService.executeMatch (src/domain/data/data_query_service.ts) stops matching catalog rows when it reaches the limit, then drops stale rows (backing file absent, filterStaleRows, swamp-club#1737) during hydration. Each stale row inside the first N matches costs one live result that was never scanned. limited is derived from the returned count in the libswamp dataQuery generator, so a shortened page reads as not limited. If every scanned row is stale the query returns nothing.
Only the unfiltered path is affected, which is every local CLI query. The include branch of DataQueryService.query (used by swamp serve for read-access filtering since 567309cb, swamp-club#2675) already re-matches in growing batches until enough rows survive the stale check. Projections (--select) skip hydration, so stale rows are not dropped there.
Expected
--limit N returns N live records when at least N exist, and limited reflects whether more matched.
Suggested fix
Apply the limit after the stale-row filter on the unfiltered path, the way the include branch already does (match in growing batches until the limit is met or matches run out). This would also let data query --single (swamp-club#2961), which currently runs locally without a limit to avoid this, stop after two surviving rows.
Shipped
Click a lifecycle step above to view its details.
Sign in to post a ripple.