Relationships
#4477 A workflow run keeps every step's resource contents in memory, so a large forEach fan-out can exhaust the heap
Opened by zack · 11/6/2026
Description
A workflow whose steps write many or large resources keeps every resource's contents in memory for the whole workflow run, so memory grows with the total data written rather than with what later steps actually read. A large forEach fan-out (thousands of iterations, each writing a resource of a few hundred KB) can exhaust the heap. This is the workflow counterpart of swamp-club#4469, which fixes the same problem for a single swamp behavior run.
Where the contents are kept
All in src/domain/workflows/execution_service.ts unless noted:
- The deprecated expression context. When a model-method step completes, each resource is re-read and parsed (
fromResourceHandle, around line 2818) and merged intostepExprContext.model[...].resource(around lines 6495-6540), where it stays until the run ends. steps.NAME.outputsand its sources.liveStepOutputs(src/domain/workflows/step_output_resolver.ts) merges every resource's attributes and keeps each record inStepOutputSources(viaoutputRecordOf). The result goes onstepExprContext.steps[NAME]for every step and everyforEachiteration, whether or not a later expression reads it.- Workflow report handles.
dataHandlesByStep(around lines 3869 and 4792, run and resume) keeps every step's handles, each carryinghandle.attributes, until the workflow report runs. - Resume.
resume()(around lines 4630-4645) and the already-completed step path (around line 5628) rebuildsteps.NAMEfor every completed step throughStepOutputResolver.resolve, which reads and keeps every resource again.
A per-method or per-step cap does not bound any of these: each forEach iteration is its own step and its own method run.
Expected
Workflow memory stays bounded by what the workflow's expressions can read, plus a fixed allowance. steps.NAME.outputs and the results of expressions are unchanged for every step a later expression references.
Suggested approach (from the #4469 planning review)
- One resource-attribute budget per workflow run, shared across steps and iterations, bounding
dataHandlesByStepand themodel.NAME.resourcemerge. Reuse theResourceAttributeBudgetvalue object introduced by #4469. - Merge
steps.NAME.outputsincrementally as each resource is read, and pruneStepOutputSourcesto records that still contribute a key. Keep any superseded record whose validation insensitive_record_adapter.tscould refuse a merged read. Use one helper for the live path andStepOutputResolver. - Keep
steps.NAME.outputsonly for steps a later expression references. Build that set with a conservative extractor overcollectWorkflowAuthoredExpressionsand the definitions of the models the steps run. Treat any dynamic use of thestepsnamespace, or a model named at run time, as keeping everything. Apply the set on resume too. - Avoid lazy getters on
DataRecord: the sensitive record adapter depends on content and attributes being the same object.
Steps to reproduce
- An extension model method that writes one resource of about 200 KB per call.
- A workflow with a
forEachstep over about 4,000 items calling that method. swamp workflow run <workflow>: process memory grows with every iteration and the run can end with a V8 heap out-of-memory error.
Open
No activity in this phase yet.
Sign in to post a ripple.