#4224 swamp serve exhausts its heap when a workflow's run history carries large inputs: every run save parses and rewrites the whole .runs-index.json

Summary. Serve's memory grows with the total size of a workflow's stored run inputs, not with current load. Each run save reads, parses and rewrites the whole per-workflow .runs-index.json, and that index holds every historical run's full inputs. In our deployment one workflow takes a ~0.6–0.8 MB base64 file as an input. After 709 runs its index is 433 MB, and serve hit the V8 heap limit and restarted 9 times across 4 sittings. We reproduced this off our deployment with a filesystem datastore. Only the stored input size changes between the two arms, and peak RSS goes from about 250 MB to 4.3–5.3 GB, with serve unresponsive.

Controlled comparison (swamp a2f815f3, filesystem datastore, no S3)

Setup:

• serve and one worker (capacity 1) in an 8 GB / 2 CPU container;
• a 9-step workflow of small steps;
• per arm-run: a fresh serve start, 1 min idle, then two concurrent runs, with a client polling workflow run search --since 6h --limit 20 every ~4 s and history get every 2–10 s.

Run histories were cloned from real records under new run ids, dated 1–30 days back. We then deleted the index and let swamp rebuild it. The only difference between arms is the stored input size per run.

Peak RSS is the largest 5-second peak (VmHWM reset after each sample).

• In arm A, serve stopped answering WebSocket pings (10 per run), so every client request failed.
• Most of the memory is taken before any run starts: 2.2 GB right after start and 2.6 GB idle, just from reading the index.
• Arm B has more runs than arm A (709 vs 515), so run count is not the driver.

A second, hard limit: Invalid string length

In a first attempt with 711 × 840 KB inputs, the index would have been ~600 MB:

• serve logged Failed to write repaired index, will retry next read: RangeError: Invalid string length 62 times. V8's maximum string length is ~2^29 characters, about 512 MB.
• every read then re-parsed all 655 MB of run records and failed again;
• searches took 10–20 s, and run discovery never succeeded.

Past that size the index cannot be written at all, whatever the heap limit. At our average entry size (611 KB) a workflow reaches it after about 880 runs.

Mechanism, from the source at a2f815f3

• Where inputs are stored. The run record keeps the full inputs (execution_service.ts:4093-4094, serialized at workflow_run.ts:2434-2435). The run summary keeps inputs (workflow_run_summary.ts:49,91), and .runs-index.json stores each summary, inputs included (workflow_run_index.ts:25,42).
• Every save rewrites everything. Each save(run), i.e. every step status change, calls updateIndexEntry (yaml_workflow_run_repository.ts:963-1004, 2155-2180). That reads the whole index, JSON.parses it, replaces one entry, then JSON.stringifys and writes all of it (workflow_run_index.ts:215-223).
• The next search rebuilds a snapshot. After any save, the next run search / history search rebuilds runIndexSnapshot (yaml_workflow_run_repository.ts:1603-1643). That is another whole-index parse, and the snapshot keeps every run's inputs resident.
• Search answers carry the inputs. run search copies each listed run's full inputs into its answer (run_search.ts:662-682), with no CLI option to leave them out.
• Inputs cannot be excluded. Workflow inputs have no sensitive flag or size bound (.claude/skills/swamp/references/workflow/reference.md:560). The per-run evaluated workflow (workflows-evaluated/runs/<runId>/, execution_service.ts:4144) holds them again.

A stand-alone micro-benchmark of just that JSON cycle (plain deno 2.9.7, not swamp's code; 3 repeats, identical heap figures):

That is about 2.5 MB of transient heap per indexed run, per save. Extrapolating linearly to 709 runs gives ~1.7 GB per save (an extrapolation, not a measured serve peak).

What is not shown yet

• Which functions allocate. We have not profiled the allocations, because the compiled binary takes no V8 flags. The comparison shows that the heap follows stored input size; it does not prove which functions hold the memory.
• The fatal crash. We did not reproduce the fatal heap OOM in the ~10-minute arms: they ended unresponsive at 4–5 GB, not crashed. Our deployment crashed after 6.5–8 min of a heavier load.

What would help, in priority order

1. Make one run's update independent of total history size. Updating one run should touch only that run's entry, not parse and rewrite every run (for example per-run index entries, a segmented index or database rows). This also removes the 512 MB wall. Splitting the file alone would fix the string limit but keep the repeated work.
2. Keep large inputs out of the index, snapshots and listings. Load full inputs only for replay and detailed inspection. Input filtering (run search --input) could use indexed fields or selective reads.
3. An opt-in heap diagnostic for the compiled binary, for example V8's sampling heap profiler dumped above a heapUsed threshold, or a heap snapshot on a signal. Then we could send you an allocation profile instead of RSS curves.
4. Regression coverage: large historical inputs, repeated saves and searches, responsiveness during them, and the string-length limit, while keeping filtering, permissions and replay working.

On our side we are moving the file out of the workflow inputs. That doesn't shrink existing history, and the missing upload route for remote members is filed separately: #4233. Any workflow that takes a webhook body, config document or manifest as an input accumulates the same way.

Reproduction. The micro-benchmark is one dependency-free script (run-index-bench.js) and can be attached. The serve comparison harness (record-set builder, driver, sampler) can be shared on request; it needs one real run record per arm as a template.

First observations (original report)

Crashes on f96841b5 and a2f815f3, under one serve on an AWS t4g.large (8 GiB) with @swamp/s3-datastore 2026.10.24.2 and one remote worker: 6 fatal Ineffective mark-compacts near heap limit / Reached heap limit exits across 3 sittings, 2.3–60 min after start, then 3 more in a fourth sitting (2026-10-31, after 8.0 and 6.5 min of uptime). The kernel OOM killer never fired. The S3 sync path and catalog are not needed to reproduce this: the comparison above uses a filesystem datastore. Our earlier S3-emulator attempt (~9.9k objects, small run history) stayed at a 290 MB heap peak, consistent with this.

Related: #4102 (worker query deadline during the same sittings), #3361 (catalog rebuild), #3407 (history search parsed every run record), #3468 (warm-up skips very large indexes), #636 (heap not configurable, by design: we are not asking for a bigger heap).