I am experiencing an out-of-memory bug that results in a crash loop. Running a large data processing job hits a Fatal OOM. It is a workflow that parses 17 WhatsApp groups, runs them through a multi-stage parsing and extraction flow, organizes them, and reports back. The workflow runs a set of parallel parse jobs, then a sequential chain of extraction jobs, all in one swamp workflow run process.

<--- Last few GCs --->

[53699:0x7a13694000] 15302 ms: Scavenge (during sweeping) 4085.1 (4092.9) -> 4082.5 (4093.4) MB, pooled: 0.0 MB, 2.48 / 0.00 ms (average mu = 0.222, current mu = 0.168) allocation failure; [53699:0x7a13694000] 15696 ms: Incremental Mark-Compact (reduce) 4088.2 (4096.2) -> 4083.0 (4088.4) MB, pooled: 0.0 MB, 371.37 / 0.00 ms (+ 0.1 ms in 90 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 381 ms) (average

Fatal JavaScript out of memory: Ineffective mark-compacts near heap limit

==== C stack trace ===============================

0   swamp                               0x0000000103347e14 uv_unref + 13258976
1   swamp                               0x000000010334b7d8 uv_unref + 13273764
2   swamp                               0x0000000103343f28 uv_unref + 13242868
3   swamp                               0x00000001033adce4 uv_unref + 13676464
4   swamp                               0x000000010358fa78 uv_unref + 15650116
5   swamp                               0x000000010358f254 uv_unref + 15648032
6   swamp                               0x000000010358de48 uv_unref + 15642900
7   swamp                               0x000000010350a7f8 uv_unref + 15104708
8   swamp                               0x00000001039767bc uv_unref + 19741320
9   swamp                               0x000000010481f26c uv_unref + 35112248
10  ???                                 0x000000011cb30840 0x0 + 4776462400
11  ???                                 0x000000011cb22d4c 0x0 + 4776406348
12  ???                                 0x000000011cb2fe18 0x0 + 4776459800
13  ???                                 0x000000011cb35490 0x0 + 4776481936
14  ???                                 0x000000011cb2ff40 0x0 + 4776460096
15  ???                                 0x000000011cb22d4c 0x0 + 4776406348
16  ???                                 0x000000011cb2fe18 0x0 + 4776459800
17  ???                                 0x000000011cb35490 0x0 + 4776481936
18  ???                                 0x000000011cb2ff40 0x0 + 4776460096
19  ???                                 0x000000011cb35490 0x0 + 4776481936
20  ???                                 0x000000011cb2ff40 0x0 + 4776460096
21  ???                                 0x000000011cb35490 0x0 + 4776481936
22  ???                                 0x000000011cb2ff40 0x0 + 4776460096
23  ???                                 0x000000011cb35490 0x0 + 4776481936
24  ???                                 0x000000011cb2ff40 0x0 + 4776460096
25  ???                                 0x000000011cb35490 0x0 + 4776481936
26  ???                                 0x000000011cb2ff40 0x0 + 4776460096
27  ???                                 0x000000011cb35490 0x0 + 4776481936
28  ???                                 0x000000011cb2ff40 0x0 + 4776460096
29  ???                                 0x000000011cb22d4c 0x0 + 4776406348
30  ???                                 0x000000011cb2fe18 0x0 + 4776459800
31  ???                                 0x000000011cb35490 0x0 + 4776481936
32  ???                                 0x000000011cb2ff40 0x0 + 4776460096
33  ???                                 0x000000011cb22d4c 0x0 + 4776406348
34  ???                                 0x000000011cb2fe18 0x0 + 4776459800
35  ???                                 0x000000011cb35490 0x0 + 4776481936
36  ???                                 0x000000011cb2ff40 0x0 + 4776460096
37  ???                                 0x000000011ca98748 0x0 + 4775839560
38  ???                                 0x000000011ca6e230 0x0 + 4775666224
39  swamp                               0x00000001047bb40c uv_unref + 34703064
40  ???                                 0x000000011ca975c0 0x0 + 4775835072
41  ???                                 0x000000011cae2828 0x0 + 4776142888
42  swamp                               0x00000001047b8884 uv_unref + 34691920
43  swamp                               0x00000001048a3218 uv_unref + 35652836
44  swamp                               0x00000001047a766c uv_unref + 34621752
45  swamp                               0x0000000104776778 uv_unref + 34421316
46  swamp                               0x00000001034bc340 uv_unref + 14784012
47  swamp                               0x00000001034bcc08 uv_unref + 14786260
48  swamp                               0x00000001034bcd40 uv_unref + 14786572
49  swamp                               0x0000000103508c90 uv_unref + 15097692
50  swamp                               0x0000000103508bc8 uv_unref + 15097492
51  swamp                               0x0000000102301ac0 uv_default_loop + 24428
52  swamp                               0x000000010230054c uv_default_loop + 18936
53  swamp                               0x000000010211b41c uv_free_cpu_info + 4624
54  swamp                               0x00000001021302ec uv_free_cpu_info + 90336
55  swamp                               0x00000001021182d0 swamp + 17104
56  swamp                               0x0000000102123058 uv_free_cpu_info + 36428
57  swamp                               0x0000000102131660 uv_free_cpu_info + 95316
58  swamp                               0x000000010211f138 uv_free_cpu_info + 20268
59  swamp                               0x0000000102133a8c uv_free_cpu_info + 104576
60  dyld                                0x0000000194c63e88 start + 6688

A second instance of the same crash, captured from swamp workflow run daily (process 70733, ended with trace trap):

<--- Last few GCs --->

[70733:0x7b49694000] 417760 ms: Incremental Mark-Compact (reduce) 4032.4 (4057.4) -> 3946.1 (3971.1) MB, pooled: 0.0 MB, 62.33 / 0.00 ms (+ 141.7 ms in 29 steps since start of marking, biggest step 5.0 ms, walltime since start of marking 210 ms) (averag

Fatal JavaScript out of memory: Ineffective mark-compacts near heap limit

==== C stack trace ===============================

0   swamp                               0x0000000101d4be14 uv_unref + 13258976
1   swamp                               0x0000000101d4f7d8 uv_unref + 13273764
2   swamp                               0x0000000101d47f28 uv_unref + 13242868
3   swamp                               0x0000000101db1ce4 uv_unref + 13676464
4   swamp                               0x0000000101f93a78 uv_unref + 15650116
5   swamp                               0x0000000101f93254 uv_unref + 15648032
6   swamp                               0x0000000101f91e48 uv_unref + 15642900
7   swamp                               0x0000000101f0e7f8 uv_unref + 15104708
8   swamp                               0x000000010237a7bc uv_unref + 19741320
9   swamp                               0x000000010322326c uv_unref + 35112248
10  ???                                 0x000000011b7e6a0c 0x0 + 4756236812
11  ???                                 0x000000011b940b08 0x0 + 4757654280
12  ???                                 0x000000011b94201c 0x0 + 4757659676
13  ???                                 0x000000011b940e20 0x0 + 4757655072
14  ???                                 0x000000011b932cd0 0x0 + 4757597392
15  ???                                 0x000000011b941064 0x0 + 4757655652
16  ???                                 0x000000011b94201c 0x0 + 4757659676
17  ???                                 0x000000011b940e20 0x0 + 4757655072
18  ???                                 0x000000011b94201c 0x0 + 4757659676
19  ???                                 0x000000011b940e20 0x0 + 4757655072
20  ???                                 0x000000011b94201c 0x0 + 4757659676
21  ???                                 0x000000011b940e20 0x0 + 4757655072
22  ???                                 0x000000011b94201c 0x0 + 4757659676
23  ???                                 0x000000011b940e20 0x0 + 4757655072
24  ???                                 0x000000011b94201c 0x0 + 4757659676
25  ???                                 0x000000011b940e20 0x0 + 4757655072
26  ???                                 0x000000011b94201c 0x0 + 4757659676
27  ???                                 0x000000011b940e20 0x0 + 4757655072
28  ???                                 0x000000011b932cd0 0x0 + 4757597392
29  ???                                 0x000000011b941064 0x0 + 4757655652
30  ???                                 0x000000011b94201c 0x0 + 4757659676
31  ???                                 0x000000011b940e20 0x0 + 4757655072
32  ???                                 0x000000011bb3f724 0x0 + 4759746340
33  ???                                 0x000000011bb3c020 0x0 + 4759732256
34  ???                                 0x000000011bb3e880 0x0 + 4759742592
35  ???                                 0x000000011bb3c33c 0x0 + 4759733052
36  ???                                 0x000000011b8d6fc8 0x0 + 4757221320
37  ???                                 0x000000011b8d654c 0x0 + 4757218636
38  ???                                 0x000000011bb0d748 0x0 + 4759541576
39  swamp                               0x00000001031bc884 uv_unref + 34691920
40  swamp                               0x00000001032a7218 uv_unref + 35652836
41  swamp                               0x00000001031ab66c uv_unref + 34621752
42  swamp                               0x000000010317a778 uv_unref + 34421316
43  swamp                               0x0000000101ec0340 uv_unref + 14784012
44  swamp                               0x0000000101ec0c08 uv_unref + 14786260
45  swamp                               0x0000000101ec0d40 uv_unref + 14786572
46  swamp                               0x0000000101f0cc90 uv_unref + 15097692
47  swamp                               0x0000000101f0cbc8 uv_unref + 15097492
48  swamp                               0x0000000100d05ac0 uv_default_loop + 24428
49  swamp                               0x0000000100d0454c uv_default_loop + 18936
50  swamp                               0x0000000100b1f41c uv_free_cpu_info + 4624
51  swamp                               0x000000010b342ec uv_free_cpu_info + 90336
52  swamp                               0x0000000100b1c2d0 swamp + 17104
53  swamp                               0x0000000100b27058 uv_free_cpu_info + 36428
54  swamp                               0x0000000100b35660 uv_free_cpu_info + 95316
55  swamp                               0x0000000100b23138 uv_free_cpu_info + 20268
56  swamp                               0x0000000100b37a8c uv_free_cpu_info + 104576
57  dyld                                0x000000018667be88 start + 6688

[1] 70733 trace trap swamp workflow run daily

Part 2: The crash is not shown as a failure

When the OOM crash kills the process, the run does not move to failed status. It stays in running status. This is what I see:

The last line of my run log is a normal INF line mid-workflow. No error, warning, or exception comes before it. Nothing comes after.
The OS process is absent. The PID no longer exists.
swamp run history --active shows the run as running [STALE]. The status uses heartbeat age only. swamp workflow run list still shows status: running.
swamp run doctor is the only command that shows me the problem. It shows "stale", not a crash. swamp run doctor --fix removes the stale run. swamp produces no failed-status record.

I recommend that, when the workflow process dies before it reaches a terminal state, the run move to failed or crashed status and record the last-known step with an "unexpected process exit" marker. For me, that would mean:

swamp workflow run list and history show the run as failed, not as running.
swamp tells the operator that the work did not complete, so I do not need to run swamp run doctor to find this.
Downstream automation (cron loops, CI) can detect the failure and retry, and does not assume that the run is still in progress.

The current behavior is the worst case for an unattended workflow, in my experience: the run dies in silence and stays in running status.

Part 3: No native per-process isolation and the shell workaround

It appears there is no native, daemon-free, on-demand way to put each step of a long workflow into its own process.

type: workflow (nested sub-workflows) and forEach both run in-process in the parent engine, from what I can tell. The execution-semantics doc states that there is "no async, detached, or fire-and-forget mode" for swamp workflow run. A "separate run" for a nested workflow is a separate run record, not a separate process. So the OOM still accumulates across all iterations in one process. I verified this empirically: a parent that invokes a child via type: workflow runs the child in the same OS process as the parent (I sampled one swamp process throughout the child run, even though the child is tracked as a separate run in history).
The only native primitive that makes separate OS processes, as far as I know, is worker placement. It needs the swamp serve orchestrator. The doc states: "Placement requires running under swamp serve; a placed step outside it fails." I also confirmed that the step-level driver field (an older per-step isolation mechanism) is removed in the current build — swamp workflow validate rejects it with: "The 'driver' field has been removed — execution drivers are replaced by remote execution (see design/remote-execution.md). Remove the field; for isolation, run a containerized worker and use step 'labels' placement." So worker placement via swamp serve is the only native isolation path I can find.

I hit the OOM and want per-process isolation. To get it without swamp serve, I have to run an external shell loop of swamp workflow run commands. I recognize this is non-native and conflicts with swamp guidance, which tells operators to use swamp fan-out methods and not shell loops. The OOM and the missing isolation primitive make me bypass swamp to work around a swamp crash.

I propose a native primitive that runs each iteration or sub-workflow in its own process, on demand, with no daemon. I suspect it can fix this and remove the need for the shell workaround.

Steps to Reproduce

I run swamp workflow run <workflow> --log. The workflow runs many model_method jobs in one process (parallel parse of 17 groups, then sequential extraction).
I wait several minutes. The run progresses normally through parse and into extract, and it writes data.
The process crashes mid-workflow with an OOM (Fatal JavaScript out of memory: Ineffective mark-compacts near heap limit). The run prints or logs no error. The process exits with trace trap.
I run swamp run history --active. It shows running [STALE].
I run swamp workflow run list. It shows status: running.
I run swamp run doctor --fix. It removes the stale run. swamp produces no failed-status record.

Environment

swamp version: swamp 20260815.023608.0-sha.a648c130
OS: macOS, Apple silicon (M-series), 32 GB RAM
Workflow shape: a set of parallel model_method parse jobs. A sequential chain of model_method extraction jobs follows. The jobs run in one DAG.

Workaround in place

My workaround splits the monolithic workflow into one workflow per group. An external shell loop runs each as its own swamp workflow run process. Each process handles one group and exits. State cannot accumulate across groups, so the OOM does not occur. All per-group runs succeed, where the monolithic run crashed mid-workflow with an OOM. The workaround is effective but non-native. I know it is the shell-bypass pattern that swamp guidance tells operators not to use. I suggest a native per-process-on-demand primitive can remove the need for it.

02Bog Flow

Triaged

8/16/2026, 8:53:16 PM

Click a lifecycle step above to view its details.

03Sludge Pulse

stack72 assigned stack728/16/2026, 8:50:45 PM

stack72 commented 8/17/2026, 8:15:52 PM

Hi @jesserobbins - was it a workflow or a model you were running here? I am trying to track down the critical path to help

Paul

jesserobbins commented 8/21/2026, 12:18:38 AM

it's a workflow that parses 17 WhatsApp groups, runs them through a multi stage parsing and extraction flow, organizes them and reports back.

jesserobbins commented 8/21/2026, 12:26:45 AM

<--- Last few GCs --->

Fatal JavaScript out of memory: Ineffective mark-compacts near heap limit

==== C stack trace ===============================

0   swamp                               0x0000000101d4be14 uv_unref + 13258976
1   swamp                               0x0000000101d4f7d8 uv_unref + 13273764
2   swamp                               0x0000000101d47f28 uv_unref + 13242868
3   swamp                               0x0000000101db1ce4 uv_unref + 13676464
4   swamp                               0x0000000101f93a78 uv_unref + 15650116
5   swamp                               0x0000000101f93254 uv_unref + 15648032
6   swamp                               0x0000000101f91e48 uv_unref + 15642900
7   swamp                               0x0000000101f0e7f8 uv_unref + 15104708
8   swamp                               0x000000010237a7bc uv_unref + 19741320
9   swamp                               0x000000010322326c uv_unref + 35112248
10  ???                                 0x000000011b7e6a0c 0x0 + 4756236812
11  ???                                 0x000000011b940b08 0x0 + 4757654280
12  ???                                 0x000000011b94201c 0x0 + 4757659676
13  ???                                 0x000000011b940e20 0x0 + 4757655072
14  ???                                 0x000000011b932cd0 0x0 + 4757597392
15  ???                                 0x000000011b941064 0x0 + 4757655652
16  ???                                 0x000000011b94201c 0x0 + 4757659676
17  ???                                 0x000000011b940e20 0x0 + 4757655072
18  ???                                 0x000000011b94201c 0x0 + 4757659676
19  ???                                 0x000000011b940e20 0x0 + 4757655072
20  ???                                 0x000000011b94201c 0x0 + 4757659676
21  ???                                 0x000000011b940e20 0x0 + 4757655072
22  ???                                 0x000000011b94201c 0x0 + 4757659676
23  ???                                 0x000000011b940e20 0x0 + 4757655072
24  ???                                 0x000000011b94201c 0x0 + 4757659676
25  ???                                 0x000000011b940e20 0x0 + 4757655072
26  ???                                 0x000000011b94201c 0x0 + 4757659676
27  ???                                 0x000000011b940e20 0x0 + 4757655072
28  ???                                 0x000000011b932cd0 0x0 + 4757597392
29  ???                                 0x000000011b941064 0x0 + 4757655652
30  ???                                 0x000000011b94201c 0x0 + 4757659676
31  ???                                 0x000000011b940e20 0x0 + 4757655072
32  ???                                 0x000000011bb3f724 0x0 + 4759746340
33  ???                                 0x000000011bb3c020 0x0 + 4759732256
34  ???                                 0x000000011bb3e880 0x0 + 4759742592
35  ???                                 0x000000011bb3c33c 0x0 + 4759733052
36  ???                                 0x000000011b8d6fc8 0x0 + 4757221320
37  ???                                 0x000000011b8d654c 0x0 + 4757218636
38  ???                                 0x000000011bb0d748 0x0 + 4759541576
39  swamp                               0x00000001031bc884 uv_unref + 34691920
40  swamp                               0x00000001032a7218 uv_unref + 35652836
41  swamp                               0x00000001031ab66c uv_unref + 34621752
42  swamp                               0x000000010317a778 uv_unref + 34421316
43  swamp                               0x0000000101ec0340 uv_unref + 14784012
44  swamp                               0x0000000101ec0c08 uv_unref + 14786260
45  swamp                               0x0000000101ec0d40 uv_unref + 14786572
46  swamp                               0x0000000101f0cc90 uv_unref + 15097692
47  swamp                               0x0000000101f0cbc8 uv_unref + 15097492
48  swamp                               0x0000000100d05ac0 uv_default_loop + 24428
49  swamp                               0x0000000100d0454c uv_default_loop + 18936
50  swamp                               0x0000000100b1f41c uv_free_cpu_info + 4624
51  swamp                               0x0000000100b342ec uv_free_cpu_info + 90336
52  swamp                               0x0000000100b1c2d0 swamp + 17104
53  swamp                               0x0000000100b27058 uv_free_cpu_info + 36428
54  swamp                               0x0000000100b35660 uv_free_cpu_info + 95316
55  swamp                               0x0000000100b23138 uv_free_cpu_info + 20268
56  swamp                               0x0000000100b37a8c uv_free_cpu_info + 104576
57  dyld                                0x000000018667be88 start + 6688

[1] 70733 trace trap swamp workflow run daily

Only documented non-interactive auth puts the password in argv; the safe SWAMP_API_KEY path is undocumented

Retyping a model instance strands catalog rows no command can reclaim — prune matches on model_id, delete resolves the current type (follow-up to #1009)

Docs: update cancel endpoint response statuses in REST API reference and workflows page

docs: update vault audit trail reference for write operation support

docs: document --dashboard flag for swamp serve

ExtensionWorkflowRepository scans manifest.yaml as a workflow, emitting 'Unknown key manifestVersion' warning

Relative path for custom agent skills setup doesn't provision skills

swamp serve daemon enable: add --dashboard flag

CLI writes runtime state (lastStalenessWarning) into tracked .swamp.yaml, dirtying a committed config file

extension install serializes files[] in readdir order, producing machine-dependent reorder diffs in upstream_extensions.json

Workflow YAML rewritten in place: block-scalar description truncated, file left unparseable, and validate still reports PASSED

workflow validate does not resolve modelType on factory-pattern steps — a non-existent extension validates clean

vault.get() arguments are verbatim, so a per-target secret cannot be expressed in workflow YAML

Cancellation/--timeout skips remaining steps including `completed`/`always` branches; --timeout waits for the in-flight subprocess

Remote "write-bearing" classification is inferred from data-plane writes, not declared; external side effects can be re-executed on worker disconnect (docs say the opposite)

Audit timeline is not populated by swamp CLI commands unless the AI-tool hook fires (manual says recording is automatic)

manual_approval suspends the run while parallel sibling jobs are mid-flight; in-flight steps are torn down and left `running`

`workflow resume --from` does nothing for forEach steps whose template name contains `${{ }}`

Step-level dependsOn on a forEach step never evaluates true (except `always`)

rum/site-info: lookup GETs a non-existent endpoint, always 405

vault read-secret appends a newline to piped output

wss:// direct to serve's own TLS listener still fails on 20260817 — client ALPN lands on h2, WS-over-h2 unimplemented (follow-up to #1363)

serve: vault audit repository not wired up — audit trail always empty

Support Jira webhooks with X-Hub-Signature HMAC-SHA256 verification

Add typed read-only remote ref lookup to @swamp/git

pullChanged skips same-size updated files — size-only change detection ignores the index sha256

A run's terminal status reaches the shared datastore only when a later run happens to push it

workflow cancel --all on the serve host shuts down the serve process

Readability of tables in documentation

repo init/upgrade --tool kiro references .kiro/skills/ but installs skills to ~/.kiro/skills/

vault: audit trail should record put, delete, and annotate operations

data.findBySpec() returns only the first in-run record, not all of them, so an .all() assert silently checks one element

repo upgrade --tool kiro reports wrong changedFiles paths

Suppress boot-time "pulled extension(s) have missing source files" WRN when invoked command is `swamp extension install`

s3-datastore: a stale machine's sync pushes its old 'latest' pointer over the newer remote one, and the rollback is permanent

`search` → `list` aliases are hand-rolled Command clones, not aliases: 10 of 11 have drifted from their parent's option set

Make official Docker image OCI-compliant

Method-name collision between a base model type and an extending extension silently drops the whole extension file — no error, doctor reports pass

data prune still misclassifies live records as orphans on 20260820 (after #1396) — serve-minted token/grant records, and report data of every live workflow

vault put prompts for the secret before validating the vault name exists

workflow validate: false 'Missing required inputs' for CEL-templated globalArgs (${{ inputs.* }})

Add SWAMP_CLI_OUTPUT env var to control default output mode (plaintext/tui/json)

@swamp/git fetch: support local path as remote

Dispatched steps over 15 minutes cannot write results: dispatch credentials expire and never refresh

Concurrent pushes to one namespace fail with HTTP 409 ConditionalRequestConflict — the conditional PUT is never retried

serve: triggerSource not persisted to workflow run summary

dataOutputOverrides not applied during remote execution — ephemeral resources GC'd before override takes effect

extension rm also deletes the lockfile entry, and install on an empty lockfile is a silent success — rm/install is not a round trip

--max-run-duration only applies to detached runs, not attached runs

Cancel endpoint reports status: cancelled for runs it did not actually cancel

@swamp/git: no way to amend a commit, but push can --force

Installation with new account creation times out

pre-flight checks run on orchestrator, not on worker — breaks remote execution with filesystem-dependent checks

docs: document --remote-only flag for swamp serve

Swamp serve daemon doesn't run with --hot-reload

serve: include workflowName in HealthSnapshotSchedule

knowledge-base retrieve persists query output into the state resource, with lifetime infinite and the query text in the instance name

Skip 50MB monolithic index rewrite on v2 shard-first path

Share single AwsS3Client instance across lock, sync, and verifier

Double shard assembly per command lifecycle (pull + push both read all shards)

pullChanged never records commitSeq for shard-first (v2) index

localHasAllRemoteEntries() permanently blocks sidecar update on partial caches

serve: workflow.history.get cannot find CLI-created runs or look up by run ID

"TRIAL ENDED" modal shown to team plan user when toggling extension visibility

Concurrent suspended workflow runs can't be disambiguated from 'approvals'; fresh runs should supersede matching-input suspended runs

I _think_ I might like a guard in the trigger yaml

swamp cmds don't realise the user is sshed into the system

serve dashboard: add cluster.instances and serve.config endpoints

CEL expressions have timestamp/duration but no current-time value, so no guard can express "older than N"

Support direct type execution (@type) with --server

access token mint writes to user vault, not serve's _token-secrets control plane in HA mode

Generic OTLP endpoint appends signal path after query or fragment

Issue lifecycle start drops top-level issue comments

serve dashboard: add instanceId, triggerSource, and step progress to WorkflowRunSummary

Document worker affinity (affinity: true) for workflows and jobs

Datastore MongoDB driver leaks connections — 5960 accumulated over 8 days

workflow approvals displays the latest run's prompt for every suspended run

@swamp/git: add a read-only working-tree diff for pre-commit review

docs: document --oauth-client-name flag for swamp serve

Docs: update workflow reference to include trigger in JSON output shapes