Relationships
#4963 data query --select reads only the content paths the expression needs
Opened by zack · 11/20/2026
Problem
swamp data query ... --select '{phase: content.phase}' returns a tiny projection, but each matching JSON record is still read whole and parsed with JSON.parse (fromRow / parseContent in src/domain/data/data_record_mapper.ts, and DataQueryService.projectedContents). Query memory scales with the full size of every matched record, not with what the select reads. The same applies to queryData from behaviors, which goes through the same service.
Proposal
- Work out from the select's CEL AST which content paths it reads: member chains rooted at
contentand itsattributesalias, static field and index access, and the comprehension macros narrowed toreceiver[*]plus the iteration variable's chains. Anything not recognised falls back to the whole document. - When the select reads content and the filter does not, parse each record with the streaming JSON projector from swamp-club#4942, keeping only those paths.
- A pruned record must never reach an output that shows the full record.
Expected outcome
Select results are identical to today. Memory is proportional to what the select reads. A property test shows that a select over the pruned document equals the select over the full document.
Context
Split out of swamp-club#4942. Depends on the streaming JSON parser and the projection types delivered there.
Open
No activity in this phase yet.
Sign in to post a ripple.