← Back to list
01Issue
FeatureOpenSwamp CLIPublic
AssigneesNone
Relationships
#4964 Streaming resource writes from behaviors with no single-string size ceiling
Opened by zack · 11/20/2026
Problem
context.writeResource serializes the whole object with JSON.stringify (or the redaction path) into one string, sometimes re-parses it, and writes it with writeText. The handle also keeps a copy of the data as attributes. Memory scales with the full resource, and a resource larger than the V8 maximum string length (about 512 MiB) cannot be written at all.
Proposal
- An incremental JSON serializer that writes plain values exactly as
JSON.stringifydoes, writes anAsyncIterableor a non-arrayIterableas an array one element at a time, flushes buffered chunks, and respects backpressure. context.writeResourceStream(specName, name, value, overrides?). The tag, lifetime, vary-suffix and name-claim resolution moves out ofwriteClaimedResourceinto shared code, so both write paths stamp identical records. Bytes go throughDefaultDataWriter.writeStream. The handle carries size but no attributes.- Schema validation stays warn-only: the skeleton is checked with streamed arrays empty, and each element against the element schema.
- The run's redactor is applied per element, with a defined maximum size for one string or windowed redaction.
- Specs with sensitive fields or
sensitiveOutputare refused for now; per-element vaulting is a follow-up. - The test context in
packages/testinggetswriteResourceStream, materialized into the captured resources.
Expected outcome
A behavior can write a resource of any size in bounded memory. Existing writeResource behavior is unchanged.
Context
Split out of swamp-club#4942. Needs swamp-club#4962, so the end of a workflow step does not load the streamed resource whole. Worker support is a separate issue.
02Bog Flow
Open
No activity in this phase yet.
03Sludge Pulse
Sign in to post a ripple.