Relationships
#4942 Bounded-memory access to large resources from behaviors: partial reads and streaming writes
Opened by zack · 11/20/2026
Problem
context.readResource returns the whole resource as one parsed object, and
context.writeResource takes one object and serializes it whole. Memory for a method therefore
scales with the full size of every resource it touches, not with what it uses. A 390 MB JSON
resource is roughly 1–2 GB once parsed into JavaScript objects, so a method that needs four
fields from each of four large resources exceeds the heap even though it reads a few kilobytes.
A single resource larger than V8's maximum string length cannot be written at all.
The CLI shows the same shape: swamp data query ... --select '{phase: content.phase}' returns a
tiny projection but still loads and parses each full record first.
Proposal
- Partial reads:
context.readResource(name, { select: "<CEL or path>" })(and the same projection pushed down forswamp data query --select) that parses only what the selection needs, or at least discards everything else while parsing. - Lazy iteration: a way to iterate a large array field (for example a history list) item by item without materializing the whole array.
- Streaming writes: a writer that serializes incrementally, so no single string holds the whole resource and the write path has no ~512 MiB ceiling.
Expected outcome
A behavior that reads only small parts of large resources runs in memory proportional to what it reads. Authors who need the whole value still can, and they accept the memory cost knowingly.
In Progress
Click a lifecycle step above to view its details.
Sign in to post a ripple.