Skip to main content
← Back to list
01Issue
BugShippedSwamp CLIPublic
Assigneeszack

Relationships

#4638 workflow run: every run-record save re-reads and parses the whole record just to find the previous status, so cost grows with step count

Opened by zack · 11/11/2026· Shipped 11/11/2026

Description

YamlWorkflowRunRepository.save() reads and parses the run's whole record from disk before every write (src/infrastructure/persistence/yaml_workflow_run_repository.ts, the findById call under if (this.eventBus)). It does this only to learn the previous status, which decides whether to publish WorkflowRunStarted, WorkflowRunCompleted or WorkflowRunFailed.

A workflow run saves its record once per step, and the record grows with each step. Every step's output embeds the full metadata of the data it wrote, about 9 KB per step for a command/shell step. So every save parses a larger file than the last, and a run's total parsing cost grows with the square of its step count. A 50-step run ends with a record of about 450 KB that it has read back 54 times.

Nothing receives those events in production. The bus is passed in by default (enableIndexing), but the index service is NoopRepoIndexService, nothing subscribes to the three run events, and nothing calls subscribeAll. The only subscriber is the test "save: repairs an unreadable record, sends no second start event, and still sends the terminal event".

Impact

The bundled build was measured with 50 serial steps that run echo on one command/shell behavior (fresh repo, quiet machine):

  • In a V8 tick profile of the run, the pre-save readById (read, YAML parse, decode) was about 16.6% of samples, roughly 0.3 s of the run.
  • Each extra step costs more as the run gets longer: 23 ms per step at 10–25 steps, 36 ms at 25–50 and 53 ms at 50–100. A 100-step run takes 4.5 s. This read is one of two costs that grow with the record; the other is re-stringifying and rewriting the whole record on every save.

Steps to reproduce

  1. swamp repo init --tool none --no-vault, then swamp behavior create command/shell sh1 and give it an execute method that runs echo bench.
  2. Create a workflow with 50 model_method steps calling sh1 / execute, with concurrency: 1 so lock backoff stays out of the measurement.
  3. Run swamp workflow run <name> and time it. Repeat with 10, 25 and 100 steps: the cost per step rises with the step count.
  4. Under strace, the run record is renamed into place once per save, about 54 times for 50 steps. Each save is preceded by a full read of the existing file.

Proposed fix

Keep the last persisted status on the WorkflowRun itself instead of reading it back from disk:

  • runFromRecord sets it when a run is loaded, and save() updates it after each successful write.
  • save() compares against that value to decide which event to publish.
  • A process that loads a run it didn't create (resume, cancel, approve/reject, recover, the reaper, serve boot reconciliation) starts from the loaded status, so it never sends a second "started" event.

This doesn't change any concurrency guarantee. The read takes no lock and compares nothing before writing, and the record is replaced whole whatever was read. Writes from different processes are kept apart by run claims (runClaims.withClaim), the fence check inside save() and atomic writes.

Two behaviour changes need deciding:

  1. The repair warning. Today a damaged record is noticed during this read and logged as "Replacing the unreadable record of workflow run …" (from #2517). The save would still repair the record but say nothing. swamp run doctor still finds damaged records. Either keep a cheap check (for example, only on a run's first save in a process) or accept losing the warning.
  2. The existing test should still pass: after the record is truncated, the in-memory status is still running, so no second start event is sent and the terminal event still is.

A smaller alternative is to skip the read when the bus has no subscribers for these event types. That changes nothing for any future subscriber, but the cost comes back as soon as one subscribes.

Environment

Current main, Linux x86_64, both the default and the --bundle compiled builds.

02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 5 MOREREVIEW+ 8 MOREPR_LINKED+ 2 MORESESSION_SUMMARIZED

Shipped

11/11/2026, 2:37:02 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
zack assigned zack11/11/2026, 2:06:44 PM

Sign in to post a ripple.