Relationships
#4962 Workflow steps parse every written resource in full, and past the V8 string limit their outputs are silently empty
Opened by zack · 11/20/2026
Problem
After every workflow step, the execution service builds the step's resource records by calling fromResourceHandle (src/domain/data/data_record_mapper.ts, called from src/domain/workflows/execution_service.ts) for each resource the step wrote. It reads the whole content with getContent, decodes it to one string and runs JSON.parse to fill attributes, which liveStepOutputs uses to build the step outputs.
Two consequences:
- Memory: the orchestrator holds the full parsed size of every resource a step wrote when the step ends, even when the method itself used bounded memory. A 390 MB resource is roughly 1 to 2 GB once parsed.
- Silent data loss: for content past the V8 maximum string length, the decode throws into a bare
catch("Not valid JSON, skip attributes"). The record gets empty attributes, so the step outputs and the expressions that read them are missing with no error or warning.
Expected
The step-end path has a bounded memory cost. For a resource too large to load as attributes, the run records that clearly (a warning, and a defined shape for the step outputs) instead of silently producing empty outputs. Expressions that need such a resource read it from storage.
Context
Split out of swamp-club#4942 (bounded-memory access to large resources). This blocks streaming resource writes from being useful inside workflows.
Open
No activity in this phase yet.
Sign in to post a ripple.