Relationships
#4470 A workflow with a 500-step forEach takes 7m22s to complete after its last step has finished
Opened by zts · 11/6/2026· Shipped 11/6/2026
Description
When a job runs a large forEach, the run stays open long after every step has finished. The time between the last step finishing and the job finishing grows much faster than the number of steps.
For example, in one run of 500 steps:
- all the steps finished between 09🔞44 and 09:19:10 (28s after the run started)
- the job finished at 09:26:31.8, and the run at 09:26:32
- nothing was written under
.swampduring that 7m22s
All 500 steps and the run reported succeeded.
| forEach steps | Run time | Time after the last step finished |
|---|---|---|
| 20 | 2.5s | 0.3s |
| 100 | 14.6s | ~9.5s |
| 200 | 48.4s | ~38s |
| 250 | ~120s | ~67–91s |
| 500 | 472.5s | ~442s |
The four 250-step runs took about the same time (119–132s) whether 500 or 1,250 instances already existed, so the delay depends on how many steps the run has, not on how much the repo holds.
Steps to reproduce
Create a trivial behavior type,
@bench/letter, whoseingestmethod writes 4 tiny JSON resources (extracted,proposal,status,filed).Add this workflow:
name: bench-ingest inputs: type: object properties: ids: { type: array, items: { type: string } } jobs: - name: ingest steps: - name: ingest-${{ self._index }} forEach: { item: lid, in: "${{ inputs.ids }}" } task: type: behavior behaviorType: '@bench/letter' behaviorName: letter-${{ self.lid }} methodName: ingest inputs: { id: "${{ self.lid }}" }
Run
swamp workflow run bench-ingest --json, withidsset to an array of 500 strings ("0000"to"0499").Compare the
completedAtof the last step underjobs[0].stepswith the job's and the run'scompletedAt.
Each step auto-creates its own instance. I haven't checked whether a forEach over a single existing instance shows the same delay.
Expected
The job finishes within a second or two of its last step.
Environment
- swamp 20261105.055054.0-sha.30608949 (Deno 2.9.8)
- macOS (Darwin 24.6.0), local filesystem datastore, not run through serve
SWAMP_NO_INBOX_NOTICE=1
This may be related to the planned job-level fanOut lane engine (#4379), but
that isn't available yet.
Shipped
Click a lifecycle step above to view its details.
zack commented 11/6/2026, 3:38:41 PM
Thanks @zts for reporting this! We shipped: Stop the run loop from re-saving the whole run once per queued step event. A parallel forEach finishes all iterations in about a second, but the run loop then awaits a full run-record save for each of the 2N queued step events, each O(N), so the job ends O(N^2) later. Stamp each step event with a sequence number when runStep produces it; record in saveRun the sequence reached when a save begins and raise the run's horizon when it lands; skip the per-event save for any event a landed save already covers. Every event still reaches downstream only after a save that includes its change has landed, and a sequential step's start is still saved before the step runs.. The fix has been merged and a release is on its way. We appreciate your contribution to swamp.
Sign in to post a ripple.