Skip to main content
← Back to list
01Issue
FeatureIn ProgressSwamp CLIPublic
Assigneeszack

Relationships

#4942 Bounded-memory access to large resources from behaviors: partial reads and streaming writes

Opened by zack · 11/20/2026

Problem

context.readResource returns the whole resource as one parsed object, and context.writeResource takes one object and serializes it whole. Memory for a method therefore scales with the full size of every resource it touches, not with what it uses. A 390 MB JSON resource is roughly 1–2 GB once parsed into JavaScript objects, so a method that needs four fields from each of four large resources exceeds the heap even though it reads a few kilobytes. A single resource larger than V8's maximum string length cannot be written at all.

The CLI shows the same shape: swamp data query ... --select '{phase: content.phase}' returns a tiny projection but still loads and parses each full record first.

Proposal

  • Partial reads: context.readResource(name, { select: "<CEL or path>" }) (and the same projection pushed down for swamp data query --select) that parses only what the selection needs, or at least discards everything else while parsing.
  • Lazy iteration: a way to iterate a large array field (for example a history list) item by item without materializing the whole array.
  • Streaming writes: a writer that serializes incrementally, so no single string holds the whole resource and the write path has no ~512 MiB ceiling.

Expected outcome

A behavior that reads only small parts of large resources runs in memory proportional to what it reads. Authors who need the whole value still can, and they accept the memory cost knowingly.

02Bog Flow
✓OPEN✓TRIAGED◉IN PROGRESS○SHIPPED+ 1 MOREASSIGNED+ 5 MOREREVIEW+ 14 MOREVERIFICATION_STARTED

In Progress

11/20/2026, 7:38:13 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
zack assigned zack11/20/2026, 7:26:10 PM

Sign in to post a ripple.