Relationships
#4638 workflow run: every run-record save re-reads and parses the whole record just to find the previous status, so cost grows with step count
Opened by zack · 11/11/2026· Shipped 11/11/2026
Description
YamlWorkflowRunRepository.save() reads and parses the run's whole record from disk before every write (src/infrastructure/persistence/yaml_workflow_run_repository.ts, the findById call under if (this.eventBus)). It does this only to learn the previous status, which decides whether to publish WorkflowRunStarted, WorkflowRunCompleted or WorkflowRunFailed.
A workflow run saves its record once per step, and the record grows with each step. Every step's output embeds the full metadata of the data it wrote, about 9 KB per step for a command/shell step. So every save parses a larger file than the last, and a run's total parsing cost grows with the square of its step count. A 50-step run ends with a record of about 450 KB that it has read back 54 times.
Nothing receives those events in production. The bus is passed in by default (enableIndexing), but the index service is NoopRepoIndexService, nothing subscribes to the three run events, and nothing calls subscribeAll. The only subscriber is the test "save: repairs an unreadable record, sends no second start event, and still sends the terminal event".
Impact
The bundled build was measured with 50 serial steps that run echo on one command/shell behavior (fresh repo, quiet machine):
- In a V8 tick profile of the run, the pre-save
readById(read, YAML parse, decode) was about 16.6% of samples, roughly 0.3 s of the run. - Each extra step costs more as the run gets longer: 23 ms per step at 10–25 steps, 36 ms at 25–50 and 53 ms at 50–100. A 100-step run takes 4.5 s. This read is one of two costs that grow with the record; the other is re-stringifying and rewriting the whole record on every save.
Steps to reproduce
swamp repo init --tool none --no-vault, thenswamp behavior create command/shell sh1and give it anexecutemethod that runsecho bench.- Create a workflow with 50
model_methodsteps callingsh1/execute, withconcurrency: 1so lock backoff stays out of the measurement. - Run
swamp workflow run <name>and time it. Repeat with 10, 25 and 100 steps: the cost per step rises with the step count. - Under
strace, the run record is renamed into place once per save, about 54 times for 50 steps. Each save is preceded by a full read of the existing file.
Proposed fix
Keep the last persisted status on the WorkflowRun itself instead of reading it back from disk:
runFromRecordsets it when a run is loaded, andsave()updates it after each successful write.save()compares against that value to decide which event to publish.- A process that loads a run it didn't create (resume, cancel, approve/reject, recover, the reaper, serve boot reconciliation) starts from the loaded status, so it never sends a second "started" event.
This doesn't change any concurrency guarantee. The read takes no lock and compares nothing before writing, and the record is replaced whole whatever was read. Writes from different processes are kept apart by run claims (runClaims.withClaim), the fence check inside save() and atomic writes.
Two behaviour changes need deciding:
- The repair warning. Today a damaged record is noticed during this read and logged as "Replacing the unreadable record of workflow run …" (from #2517). The save would still repair the record but say nothing.
swamp run doctorstill finds damaged records. Either keep a cheap check (for example, only on a run's first save in a process) or accept losing the warning. - The existing test should still pass: after the record is truncated, the in-memory status is still
running, so no second start event is sent and the terminal event still is.
A smaller alternative is to skip the read when the bus has no subscribers for these event types. That changes nothing for any future subscriber, but the cost comes back as soon as one subscribes.
Environment
Current main, Linux x86_64, both the default and the --bundle compiled builds.
Shipped
Click a lifecycle step above to view its details.
Sign in to post a ripple.