Skip to main content
← Back to list
01Issue
BugOpenSwamp CLIPublic
AssigneesNone

Relationships

#4477 A workflow run keeps every step's resource contents in memory, so a large forEach fan-out can exhaust the heap

Opened by zack · 11/6/2026

Description

A workflow whose steps write many or large resources keeps every resource's contents in memory for the whole workflow run, so memory grows with the total data written rather than with what later steps actually read. A large forEach fan-out (thousands of iterations, each writing a resource of a few hundred KB) can exhaust the heap. This is the workflow counterpart of swamp-club#4469, which fixes the same problem for a single swamp behavior run.

Where the contents are kept

All in src/domain/workflows/execution_service.ts unless noted:

  1. The deprecated expression context. When a model-method step completes, each resource is re-read and parsed (fromResourceHandle, around line 2818) and merged into stepExprContext.model[...].resource (around lines 6495-6540), where it stays until the run ends.
  2. steps.NAME.outputs and its sources. liveStepOutputs (src/domain/workflows/step_output_resolver.ts) merges every resource's attributes and keeps each record in StepOutputSources (via outputRecordOf). The result goes on stepExprContext.steps[NAME] for every step and every forEach iteration, whether or not a later expression reads it.
  3. Workflow report handles. dataHandlesByStep (around lines 3869 and 4792, run and resume) keeps every step's handles, each carrying handle.attributes, until the workflow report runs.
  4. Resume. resume() (around lines 4630-4645) and the already-completed step path (around line 5628) rebuild steps.NAME for every completed step through StepOutputResolver.resolve, which reads and keeps every resource again.

A per-method or per-step cap does not bound any of these: each forEach iteration is its own step and its own method run.

Expected

Workflow memory stays bounded by what the workflow's expressions can read, plus a fixed allowance. steps.NAME.outputs and the results of expressions are unchanged for every step a later expression references.

Suggested approach (from the #4469 planning review)

  • One resource-attribute budget per workflow run, shared across steps and iterations, bounding dataHandlesByStep and the model.NAME.resource merge. Reuse the ResourceAttributeBudget value object introduced by #4469.
  • Merge steps.NAME.outputs incrementally as each resource is read, and prune StepOutputSources to records that still contribute a key. Keep any superseded record whose validation in sensitive_record_adapter.ts could refuse a merged read. Use one helper for the live path and StepOutputResolver.
  • Keep steps.NAME.outputs only for steps a later expression references. Build that set with a conservative extractor over collectWorkflowAuthoredExpressions and the definitions of the models the steps run. Treat any dynamic use of the steps namespace, or a model named at run time, as keeping everything. Apply the set on resume too.
  • Avoid lazy getters on DataRecord: the sensitive record adapter depends on content and attributes being the same object.

Steps to reproduce

  1. An extension model method that writes one resource of about 200 KB per call.
  2. A workflow with a forEach step over about 4,000 items calling that method.
  3. swamp workflow run <workflow>: process memory grows with every iteration and the run can end with a V8 heap out-of-memory error.
02Bog Flow
◉OPEN○TRIAGED○IN PROGRESS○SHIPPED

Open

11/6/2026, 3:32:40 PM

No activity in this phase yet.

03Sludge Pulse

Sign in to post a ripple.