Lab #4224: swamp serve reaches the V8 heap limit within 2–60 min under a small remote-worker workload (~20 runs, ~3.5k record versions)

Setup.

swamp 20261030.160426.0-sha.a2f815f3, also seen on 20261027.021509.0-sha.f96841b5.
One swamp serve under systemd on an AWS t4g.large (2 vCPU arm64, 8 GiB), @swamp/s3-datastore 2026.10.24.2, one remote worker.
The workload: about 20 sequential analysis workflow runs, 7–11 model-method steps each, then two concurrent runs and a serve restart.

What happens. serve logs # Fatal JavaScript out of memory: Ineffective mark-compacts near heap limit (or Reached heap limit), exits code=dumped, status=5/TRAP and systemd restarts it. This happened 6 times across 3 sittings:

date (UTC)	build	uptime at crash
2026-10-30 14:15	f96841b5	~60 min
2026-10-30 21:25	a2f815f3	~55 min
2026-10-30 21:43	a2f815f3	18 min after restart
2026-10-31 11:22	a2f815f3	57 min
2026-10-31 11:44	a2f815f3	21 min after restart
2026-10-31 11:52	a2f815f3	2.3 min after a restart

The kernel OOM killer never fired.
The cgroup's MemoryCurrent was 3.1–3.8 GB afterwards, and that includes page cache.
The 2.3-minute crash suggests that either the start itself is heavy (catalog rebuild?) or the growth is fast under concurrent load.

Scale, for comparison with earlier serve OOMs.

About 3.5k record versions in the datastore (a retention plan listed 3,577), about 20 workflow runs per sitting, and one worker.
That's much smaller than #1684 (2,000+ model definitions), #1552 (~400 artifacts per run) or #1558 (all runs loaded at boot).

Not asking for a bigger heap. We read #636: a larger --max-old-space-size would only delay this, and the compiled binary's baked v8_flags contain none anyway. What we'd like to find is what grows.

Debug-level hints from the same window (SWAMP_LOG_LEVEL=debug, see #4102):

13,274 Query predicate skipped row "leases"/… and "pending"/… lines against 268 worker handler calls in 23 minutes.
Syncing behavior @… extension re-syncs: 22 in the same 23 minutes.

Asks.

Is this a known growth path in serve on a2f815f3? Possibly the boot full pull of #2435 (the 2.3-minute crash came right after a restart), the catalog rebuild, or per-connection state across ~200 authenticated WebSocket connections.
Is there a flag or debug mode for heap growth that works with the compiled binary? We can rerun a sitting with it and send the output.

Related: #4102 (trace of the same sittings), #636 (heap not configurable, by design), #2435 (serve boot hydration).

Worker queryData fails at the fixed 30 s RPC limit while it waits for a full catalog rebuild

cfgmgmt firewall: match rules by destination and comment

Align Caddy reverse-proxy TLS documentation with Kubernetes proxy guidance

Scheduled flaky-test detection is failing

vault put --server: accept the value on stdin (keep secrets out of argv)

swamp serve reaches the V8 heap limit within 2–60 min under a small remote-worker workload (~20 runs, ~3.5k record versions)

Hot reload can keep an old local behavior registry and memoized bundle after external catalog indexing (1602; e459 regression passes)

Add expiring service-account token requests and authorization checks to RBAC

Flaky: v3 replica check stall test reads restore.pid before the stub writes it

Reads other than findById still treat a lapsed temporary ban as active until the operative next signs in

verify-build upgrade gate silently skips models whose version is a constant

@swamp/agent-sessions: explore a decision model (TypeSafe Jev) for call labelling

agent-sessions: follow-ups from the 4211 reviews (repo resolution cache and edge cases)

agent-sessions: revisit the import's remaining review findings (redaction edge cases, import robustness)

Docs: document the datastore durable setting and its macOS ordering in the manual

Docs: working offline against swamp serve (leases, outbox, flush, quarantine) for swamp-club#4195

Swamp Club collective GUI: show the swamp serve instances currently connected to a collective

Docs: swamp data query --sort, --cursor and nextCursor in the data reference

A malformed --ca-cert file is reported as an unreadable OS certificate store

TLS hint blames a missing root CA and macOS for every invalid peer certificate failure

workflow cancel --server does not send SWAMP_SERVE_EXTRA_HEADERS

S3/GCS repairUnpushedFiles: the pull it asks for after a lost last-synced index does not stop it re-uploading copies a peer deleted

swamp-uat private-registry tests leave npm/<port> dirs in the shared Deno cache

Read-model catch-up of many data versions slows sharply under load

swamp serve --groups-field help says group/collective memberships; it names only the collectives field

A resume runs settled work after the teardown gated on it has already run, and nothing tears that work down

A step settled after the abort overwrites the expression-context entry of a same-named step that ran in another job

The extractContentMetadata property test generates invalid source and fails at random

A limited model.output.search copies and sorts the whole listing on the event loop for each request, so that step still scales with the window

Flaky test: access_poll_phase_telemetry_test sees grants_active 1 where the snapshot build read 2 records

A lone step in a job still starts under the aborted signal when another job has suspended the run

CLI command to change an existing extension's visibility (private → public)

Docs: team-mode model leases (swamp behavior lease, lease status in datastore lock status)

serve (token mode): --admins accepts idp-group entries that can never match

build-swamp-extension example: code-review isn't given the quality evidence, so the reviewer reports quality as missing

Filesystem control-plane put races empty-parent cleanup (cron pending-run ENOENT)

deployment: getLogs method (deployment-aware pod log tailing)

Agent sessions extension: import Codex sessions

model.output.search response phase takes 3.9s for 5,000 items with no attribution (redaction vs serialization vs send)

Docs: v3 serve team mode, followers forward writes and the advertise-address and ca-cert serve flags (swamp-club#4120)

Skill/docs: document registry listing limits, static-extraction rules, and splitting large APIs by domain

Check registry listing limits in extension quality/push and suggest splitting into per-domain extensions

Over-limit contentMetadata publishes a blank extension page; archive fallback extracts nothing

swamp inbox: with a filter active, sitting on the last row pages through the whole inbox in the background

@swamp/video/brand: follow-ups from the initial review (render hang, odd sizes, Chromium errors)

@swamp/video: follow-ups from the initial review (card length, input validation, OCR frames on disk)

@swamp/video/hyprland: recording hotkeys, retake marker and phone-mirror window rules

@swamp/video/obs: optional OBS recording with per-person hardware config for @swamp/video

serve daemon enable: add Windows support (service or startup task)

swamp skill docs: publishing.md contradicts itself on dynamic imports, overstates where git dependencies are listed, and never says consumers keep the publisher's bundle

Relationships

#4224 swamp serve reaches the V8 heap limit within 2–60 min under a small remote-worker workload (~20 runs, ~3.5k record versions)

Open