Skip to main content
← Back to list
01Issue
BugShippedSwamp CLIPublic
Assigneeszack

Relationships

#4470 A workflow with a 500-step forEach takes 7m22s to complete after its last step has finished

Opened by zts · 11/6/2026· Shipped 11/6/2026

Description

When a job runs a large forEach, the run stays open long after every step has finished. The time between the last step finishing and the job finishing grows much faster than the number of steps.

For example, in one run of 500 steps:

  • all the steps finished between 09🔞44 and 09:19:10 (28s after the run started)
  • the job finished at 09:26:31.8, and the run at 09:26:32
  • nothing was written under .swamp during that 7m22s

All 500 steps and the run reported succeeded.

forEach steps Run time Time after the last step finished
20 2.5s 0.3s
100 14.6s ~9.5s
200 48.4s ~38s
250 ~120s ~67–91s
500 472.5s ~442s

The four 250-step runs took about the same time (119–132s) whether 500 or 1,250 instances already existed, so the delay depends on how many steps the run has, not on how much the repo holds.

Steps to reproduce

  1. Create a trivial behavior type, @bench/letter, whose ingest method writes 4 tiny JSON resources (extracted, proposal, status, filed).

  2. Add this workflow:

    name: bench-ingest
    inputs:
      type: object
      properties:
        ids: { type: array, items: { type: string } }
    jobs:
      - name: ingest
        steps:
          - name: ingest-${{ self._index }}
            forEach: { item: lid, in: "${{ inputs.ids }}" }
            task:
              type: behavior
              behaviorType: '@bench/letter'
              behaviorName: letter-${{ self.lid }}
              methodName: ingest
              inputs: { id: "${{ self.lid }}" }
  3. Run swamp workflow run bench-ingest --json, with ids set to an array of 500 strings ("0000" to "0499").

  4. Compare the completedAt of the last step under jobs[0].steps with the job's and the run's completedAt.

Each step auto-creates its own instance. I haven't checked whether a forEach over a single existing instance shows the same delay.

Expected

The job finishes within a second or two of its last step.

Environment

  • swamp 20261105.055054.0-sha.30608949 (Deno 2.9.8)
  • macOS (Darwin 24.6.0), local filesystem datastore, not run through serve
  • SWAMP_NO_INBOX_NOTICE=1

This may be related to the planned job-level fanOut lane engine (#4379), but that isn't available yet.

02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 2 MOREREVIEW+ 8 MOREPR_LINKED+ 2 MORESESSION_SUMMARIZED

Shipped

11/6/2026, 3:38:40 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
zack assigned zack11/6/2026, 2:12:44 PM
Editable. Press Enter to edit.

zack commented 11/6/2026, 3:38:41 PM

Thanks @zts for reporting this! We shipped: Stop the run loop from re-saving the whole run once per queued step event. A parallel forEach finishes all iterations in about a second, but the run loop then awaits a full run-record save for each of the 2N queued step events, each O(N), so the job ends O(N^2) later. Stamp each step event with a sequence number when runStep produces it; record in saveRun the sequence reached when a save begins and raise the run's horizon when it lands; skip the per-event save for any event a landed save already covers. Every event still reaches downstream only after a save that includes its change has landed, and a sequential step's start is still saved before the step runs.. The fix has been merged and a release is on its way. We appreciate your contribution to swamp.

Sign in to post a ripple.