Skip to main content
← Back to list
01Issue
BugShippedSwamp CLIPublic
Assigneesstack72

Relationships

#1552 Workflow runs with wide dataArtifact fan-out (~400/run) OOM both workflow history get and swamp serve boot, crash-looping the scheduler; rollback doesn't help (state-triggered, not release-triggered)

Opened by magistr · 8/7/2026· Shipped 8/7/2026

Description

Two related memory-exhaustion failures on a repo whose workflow runs carry wide dataArtifact fan-out (~400 artifacts per run), both V8 "Fatal JavaScript out of memory: Ineffective mark-compacts near heap limit" at a ~4 GB heap:

  1. swamp workflow history get <workflow> --json OOMs. 415 accumulated runs whose on-disk records total only 22 MB blow a 4 GB heap (~185x amplification), so the assembly path is materializing far more than the input. Sibling of #1173 (history search, shipped) — reproduced on 20260807.031228.0, i.e. after that fix, so history get was not covered.

  2. swamp serve OOM-crash-loops at BOOT on the same repo — dying immediately after Policy snapshot loaded (the next, unlogged boot phase), restart-looping forever (6+ consecutive identical crashes). Critically:

    • reproduces on 20260807.031228.0 and on 20260728.192016.0 — the exact binary that had served this repo continuously for the previous 2.5 weeks (the first failing boot was simply the first restart in weeks);
    • reproduces with the workflow-runs directory trimmed from 416 run files to 20 — so it is NOT the .swamp/workflow-runs scan;
    • a plain CLI in the same container (swamp model method run, swamp workflow run, full repo init included) works throughout — the failure is specific to a serve boot phase.

    Net effect: a scheduled workflow with wide per-run artifact fan-out eventually bricks its own scheduler — one day serve simply cannot boot its repo any more, and rolling back the binary does not help because the trigger is accumulated state, not the release.

Environment

  • swamp 20260807.031228.0-sha.9a36314e and 20260728.192016.0-sha.f2a1f6b9 (Linux container, scheduler-only serve: --host 127.0.0.1 --auth-mode none, init: true)
  • Datastore @keeb/mongodb-datastore (namespaced); vault @webframp/hashicorp-vault
  • Workflow: 4 model-method steps per run, hourly schedule since 2026-07-20; one step fans out over ~400 targets, so each run records ~400+ dataArtifacts
  • .swamp/workflow-runs/<workflow-id>/: 415 files, 22 MB; repo data tree ~302k files / ~4 GB

Steps to reproduce

  1. Schedule a workflow whose steps produce hundreds of dataArtifacts per run; let it run hourly for a few weeks.
  2. swamp workflow history get <name> --json → OOM.
  3. Restart the serve container → boot OOM crash-loop.

Observed

history get on 20260728 (~93 s in):

[4682:0x555ae8c9a000]    91933 ms: Scavenge (during sweeping) 4083.0 (4086.1) -> 4082.2 (4086.4) MB ... allocation failure;
# Fatal JavaScript out of memory: Ineffective mark-compacts near heap limit

(on 20260807 the same command dies as a bare exit-137 kill with zero output.)

serve boot (both versions, every restart, last logged line before death):

[INF] serve: HA: detached runs, pending-run durability (no remote control-plane — heartbeat and reconciliation disabled)
[INF] serve: Control-plane store: local filesystem fallback
[INF] swamp·domain·access·policy-snapshot: Loaded policy snapshot: 0 active grant(s), 0 group(s)
[INF] serve: Policy snapshot loaded (reload mode: "manual")
# Fatal JavaScript out of memory: Ineffective mark-compacts near heap limit

Expected / suggestions

  1. history get should load only the most recent run by default and paginate beyond that (mirror of whatever #1173 did for search); a --last N bound for the whole history family.
  2. Whichever serve boot phase follows the policy snapshot needs the same bounding — it currently scales with accumulated run/artifact state until boot is impossible. A log line naming that phase would also make this diagnosable in the field (the crash is currently silent about what it was doing).
  3. The ~185x blow-up (22 MB of run records -> 4 GB heap) suggests per-run dataArtifact expansion is held in memory all at once; worth profiling independently.

Workaround in production

Serve disabled (command: sleep infinity keeps the container as an exec host) + OS cron running docker exec <container> swamp workflow run <name> hourly — the CLI path handles the same repo fine (same pattern as the #914 workaround).

02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 7 MOREREVIEW+ 4 MOREPR_MERGED+ 2 MORESESSION_SUMMARIZED

Shipped

8/7/2026, 10:07:07 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack728/7/2026, 6:14:18 PM
Editable. Press Enter to edit.

magistr commented 8/7/2026, 8:28:08 PM

Extra environment detail from the affected host, in case it helps narrow the OOM analysis: neither serve container has a memory limit, so the ~4 GB ceiling in the traces is not a cgroup cap.

Measured on the Unraid host (Docker, cgroup v2), 2026-08-07:

container /sys/fs/cgroup/memory.max
swamp-serve max
swamp-serve-move (the repo in this report) max

Host: MemTotal: 65726192 kB (~64 GB), 32 cores.

So the process reached 4083.0 (4086.1) MB and died with the container cgroup unlimited on a 64 GB box — the heap ceiling comes from inside the binary rather than from the container.

Also probed whether the compiled binary takes a runtime V8 flag:

$ DENO_V8_FLAGS=--bogus-flag-xyz swamp --version
swamp 20260728.192016.0-sha.f2a1f6b9
rc=0

An unrecognized V8 flag produced no complaint and no failure, i.e. DENO_V8_FLAGS does not appear to reach V8 in the standalone build.

stack72 commented 8/7/2026, 8:56:06 PM

@magistr — PR #2097 fixes the workflow history get OOM. The root cause was findLatestByWorkflowId loading all 415 run files as full aggregates just to return the most recent one — it now uses the runs index to load a single file.

For the swamp serve boot OOM, we haven't been able to identify the root cause from the code. The PR adds diagnostic log lines to every boot phase after "Policy snapshot loaded", so the next crash will show which phase was executing when it died.

Could you redeploy with the next release and share the serve boot logs from the crash directly with @stack72? We'll get this taken care of.

stack72 commented 8/7/2026, 10:07:33 PM

Thanks @magistr for reporting this! The fix has been merged and a release is on its way. We appreciate your contribution to swamp.

magistr commented 8/7/2026, 10:35:11 PM

@stack72 — redeployed on 20260807.203020.0-sha.02bf006e. Splitting the two symptoms:

1. workflow history get — FIXED. Same repo, same 415-run history that OOMed before:

$ swamp workflow history get @magistr/move-dossier-full --repo-dir /workspace --json
rc=0 — 199909 bytes of JSON, empty stderr, 216 s wall

216 s is slow for what should be a single-run lookup, but it completes and the heap stays flat. Thanks for the quick turnaround on #2097.

2. serve boot OOM — still reproduces, and your new phase logging pins it down. Serve enabled on the same repo, 2 crash-restarts in 4 minutes, every boot identical:

2026-08-07T22:30:12.962Z [INF] serve: Policy snapshot loaded (reload mode: "manual")
2026-08-07T22:30:12.963Z [INF] serve: Boot: reaping stale runs via tracker
2026-08-07T22:30:12.976Z [INF] serve: Boot: reconciling workflow run state
[7:0x56546e704000]    86125 ms: Incremental Mark-Compact (reduce) 4088.1 (4093.9) -> 4087.8 (4091.2) MB, pooled: 0.0 MB, 1560.03 / 0.00 ms (+ 18.4 ms in 4 steps since start of marking, biggest step 5.0 ms, walltime since start of marking 1587 ms) (average
#
# Fatal JavaScript out of memory: Ineffective mark-compacts near heap limit
#
==== C stack trace ===============================
    swamp(+0x23efc5e) [0x56546ac90c5e]
    swamp(+0x23eec7b) [0x56546ac8fc7b]
    swamp(+0x23f0ec8) [0x56546ac91ec8]
    swamp(+0x249fb75) [0x56546ad40b75]
    swamp(+0x2656487) [0x56546aef7487]
    swamp(+0x2655d1d) [0x56546aef6d1d]
    swamp(+0x2654904) [0x56546aef5904]
    swamp(+0x25f362b) [0x56546ae9462b]
    swamp(+0x2a3f34d) [0x56546b2e034d]
    [0x14811be81fb6]

So the dying phase is Boot: reconciling workflow run state. Both new lines print, then it spends ~86 s allocating and dies at ~4088 MB. Nothing after those two lines ever appears.

Watched live during the climb: RSS 4.34 GiB at ~75 s in, CPU ~103% (one core saturated), container memory limit still unlimited.

Repo shape unchanged from the report: 415 run files / 22 MB in .swamp/workflow-runs/<id>/, ~400 dataArtifacts per run.

Back on the sleep infinity + cron bridge meanwhile, now running the new binary. Happy to run any instrumented build or dump whatever state would help.

stack72 commented 8/7/2026, 10:54:04 PM

@magistr — thanks for the detailed logs and the redeployment turnaround, that pinpointed everything we needed.

1. workflow history get (216s): That was a one-time index build — the fix in #2097 introduced a per-workflow .runs-index.json that makes subsequent lookups near-instant, but your 415 runs predated the fix so the first access had to parse all of them to build the index. Now that it's built, future workflow history get calls on that workflow will be fast. If you have other workflows with large histories that haven't been accessed yet, those will also see a one-time slow first call, then fast after that. No action needed on your end.

2. Serve boot OOM: Filed as #1558. The root cause is the same load-all-to-find-few pattern: the boot reconciliation phase loads every workflow run from the last 7 days as a full aggregate (jobs, steps, outputs, 400 data artifacts each) just to find the handful with status: running. We're going to use the runs index to identify running runs first and only load those. Fix incoming.

Sign in to post a ripple.