Lab #1030: swamp serve crashes with JavaScript heap OOM while running scheduled workflows

Summary

swamp serve -v crashed after ~3m17s with a fatal JavaScript heap out-of-memory while scheduled workflows were running. The process reached ~4 GB old-space and dumped core.

Environment

Swamp: 20260708.012813.0-sha.0f7166fe
OS: Linux laptop-tlaas-framework 7.0.11-200.fc44.x86_64 #1 SMP PREEMPT_DYNAMIC Mon Jun 1 22:50:37 UTC 2026 x86_64 GNU/Linux
Repository: local-infra
Command: time swamp serve -v
Auth mode: default/none (warning emitted)

What happened

The server initialized, loaded the policy snapshot, registered 7 scheduled workflows, began firing scheduled runs at minute boundaries, then crashed with V8 heap exhaustion.

Relevant timeline:

12:09:05.047 warning serve auth-mode is none — deprecated
12:09:05.048 info serve Initializing repository at /var/home/evrardjp/git/evrardjp/local-infra
12:09:05.093 info swamp·domain·access·policy-snapshot Loaded policy snapshot: 0 active grant(s), 0 group(s)
12:09:34.764 info scheduled-execution Scheduled execution service started with 7 schedules
12:09:34.765 info serve WebSocket API server listening on ws://127.0.0.1:9090
12:10:00.002 info serve Running scheduled workflow REDACTED2
12:10:00.004 info serve Running scheduled workflow cluster-pool-sync
12:10:00.004 info serve Running scheduled workflow github-project-stale-classification-daily
12:10:00.094 info datastore·lock Acquiring lock for data/@evrardjp/REDACTED/c0f6776b-8c31-44c4-92cd-967180a2c6e1/.lock#91ef498d
12:10:59.112 info serve Scheduled workflow REDACTED2 completed (run: 214dbc92-6fd3-41ae-9a84-333bd7872c68)
12:10:59.414 info serve Scheduled workflow cluster-pool-sync completed (run: 6a3695ed-3b63-4070-99c2-62c6d6c8f0a6)
12:11:00.002 info serve Running scheduled workflow REDACTED2
12:11:00.003 info scheduled-execution Firing scheduled run for workflow REDACTED2

Immediately after the 12:11 scheduled run fired:

<--- Last few GCs --->

[245115:0x55cdc08c8000]   189054 ms: Mark-Compact (reduce) 4082.5 (4099.6) -> 4082.5 (4099.6) MB, pooled: 0.0 MB, 451.57 / 0.00 ms (average mu = 0.003, current mu = 0.000) last resort; GC in old space requested
[245115:0x55cdc08c8000]   189498 ms: Mark-Compact (reduce) 4082.5 (4099.6) -> 4082.5 (4099.6) MB, pooled: 0.0 MB, 443.31 / 0.00 ms (average mu = 0.002, current mu = 0.000) last resort; GC in old space requested

#
# Fatal JavaScript out of memory: Reached heap limit
#
==== C stack trace ===============================

    swamp(+0x23efc5e) [0x55cda6065c5e]
    swamp(+0x23eec7b) [0x55cda6064c7b]
    swamp(+0x23f0ec8) [0x55cda6066ec8]
    swamp(+0x249fb75) [0x55cda6115b75]
    swamp(+0x2656487) [0x55cda62cc487]
    swamp(+0x2655bb4) [0x55cda62cbbb4]
    swamp(+0x265537d) [0x55cda62cb37d]
    swamp(+0x2648d9b) [0x55cda62bed9b]
    swamp(+0x264873c) [0x55cda62be73c]
    swamp(+0x2626f6c) [0x55cda629cf6c]
    swamp(+0x261d471) [0x55cda6293471]
    swamp(+0x262aa60) [0x55cda62a0a60]
    swamp(+0x24b1654) [0x55cda6127654]
    swamp(+0x4f35115) [0x55cda8bab115]
    [0x7fb787dcfdda]
Trace/breakpoint trap      (core dumped) swamp serve -v

real    3m17.500s
user    2m7.959s
sys     0m24.187s

Expected behavior

swamp serve should remain running, or fail a single scheduled workflow gracefully, rather than growing to the V8 heap limit and crashing the server process.

Notes

The crash happened while scheduled execution was enabled and multiple workflows fired at the same minute boundary.
One workflow (REDACTED2) runs every minute and appears to have overlapped with/started again shortly after the previous scheduled work completed.
Several lock acquisition lines targeted the same redacted model data directory before the crash.

Registry: bump @swamp-club/extension-scoring to the version carrying #3983 (and #3845) before the v5 rescore

Five Mongo queues can hand the same row to two replicas

Docs: run-workflows-on-a-schedule tells readers to curl /health for queue state, which no longer lists schedules

Collective invite shows "This invitation has expired or has already been used" after a successful join via Google sign-in

Move the Zod argument-field extractor into @swamp-club/extension-scoring so the CLI and registry read schemas alike

Docs: extension quality scores with the registry formula and reports grade (swamp-club#3845)

serve: dispatch to an idle worker waits for its worker_busy write on the shared gateway transition tail

serve: worker enrollment resolves and creates the worker definition inside the serialized gateway transition tail

Rescore every extension's current version under rubric v5

Scheduled flaky-test detection is failing

Manual: the supported extension dependency categories, the canonical form and the lock rule, rewritten once against the final design

Registry audits v2 extension archives as written, and a range costs the loose point only without a valid lock

Extension consumers keep the publisher's bundle: pull, update and auto-resolve stop rebuilding (direction A for #3634)

Extension push: inputs that open up (workspace root lock, ancestor config refused by name, JSR dist-tags) and the skill docs rewritten once

Extension push ships the canonical form: every npm:/jsr: import pinned exact, and a v2 dependency-audit.json

stagecraft studio ticket columns: round-5 review lows from #3828 (source watch cost, relationText lookup, cross-instance claim)

A long-lived serve client for extensions, and a stable rate_limited error code

swamp serve on a v3 datastore never settles a workflow run left running by a stopped instance: boot reaper skips it as belonging to a remote instance

swamp serve on a v3 datastore: a run interrupted at a clean shutdown is never committed when the bucket is slow, and stays running for every other checkout

Support creating 1Password vaults for new clients

Filesystem datastore: an unclean shutdown leaves zero-byte data versions as the latest version, and `doctor datastores --repair` does not detect them

datastore_v3_replica_verify_test stall case flakes under the full integration run: restore.pid is read before the stub writes it

Generated state schemas require every child of nested objects, so normal resources warn on create/get

Add LinkedIn as a login provider

Showcase reward backfill can double-pay 5,000,000 points when replicas start together

Showcase editor: caption counter is not announced to screen readers

Profile showcase read: approved list sorts on review.reviewedAt in memory

Showcase countByStatus: $match live statuses before the $group

Showcase admin queues: index-serve the updatedAt sort on approved/rejected

Annotate and retain past workflow runs and data after the fact

Docs: --trusted-hosts accepts host:port for public addresses on non-default ports

A workflow step's method output is invisible to `data query` and `data list`, and the method summary's retrieval command returns an older run's data

Skill propulsion rule is silent when no transition is satisfied because stage work is outstanding

package.json mode: no loose-specifier warning for a range push ships as the exact installed version

extension fmt (and any command) creates or rewrites deno.lock when the extension lives in the repo's loaded extensions/ dir

serve shutdown does not wait for worker status writes, and its listener accepts workers after the gateway is disposed

stagecraft: rename work mode `method` to `behavior`, matching swamp's rename of models to behaviors

auth token create --json fields key and signinToken are undocumented; no token-mode service setup page

trigger.schedule is evaluated in the serve host's local time zone, with no zone field and no doc saying so

workflow validate/run say "Workflow not found" for a file that doctor workflows explains is misnamed

telemetry stats says "No telemetry data found" right after successful sends; say the spool is empty because it flushed

Docs: publish typed-secrets Phase 4.6 expression and delivery semantics

Extension registry page: a supported way to include media (images) in the README

CEL: data.latest(...).version is a double, but data.version() only accepts int, so "version - 1" fails at run time

workflow resume --run refuses workflows with the same step name in two jobs, though validate and run accept them

Update public OpenTelemetry manual for remote waterfalls and execution metrics (#3786)

Server-driven warning and info messages to CLI users, targeted by swamp version (exploration)

Serve: delegate namespace-scoped extension pulls and reload without granting administrator access

workflow cancel --server ignores --ca-cert and SWAMP_CA_CERT

Preserve active worker execution across serve restarts and upgrades; reconnect and reattach dispatches

v3: chunk durability barrier costs a full drive-cache flush per command on macOS; use an ordering barrier

vary/forEach object values with no key/name/id produce data names Windows refuses

issue-lifecycle start drops issue comments: context.comments is always empty

swamp core: a non-string sensitive value reads back as a JSON string

Generated vault keys carry raw data-name and vary characters that strict vault providers reject

bundleExtension 'bundles on different caches do not wait for each other' test fails under CPU load

extension install and extension pull leave different bundles for the same extension version

Registry drops contentMetadata.extensions (method grafts) at confirm-push

Multi-model skill eval: gpt-5.4 and gemini jobs fail preflight every run (OpenAI credits exhausted, Google API key invalid)

local_source_failed drops the indexing error: a transient local extension failure leaves no cause (follow-up to 1672)

Finer-grained method locking: a long-running method on one definition blocks unrelated objects managed by the same definition

Surface (or warn about) values a model method returns besides dataHandles, with sensitive-field support

workflow placement: run a workflow transparently on a specified host, local or via swamp workers (incl. nested child workflows)

issue-lifecycle: pr_merged and ship report success and advance the phase when swamp-club is unreachable, writing nothing upstream

auth login leaves auth.json apiKey empty and stores the key only in ~/.config/swamp/api-key, which the auth gate never reads

swamp-extensions CI: Review Integrity explains a stalled or failed audit (streamed progress, timeout, did-not-finish comment)

Workflow run cancellation on concurrency-key conflict

Read non-string sensitive resource fields back with their type

Run index repair-by-stat misses an in-place, same-size record rewrite that restores mtime within the same second as its entry: ctime is reported in whole seconds

Configurable storage tiers: runs, outputs, data versions and audit move through an ordered list of user-defined tiers, and stay queryable across them

A global argument that evaluates to swamp-namespace expression text is silently dropped, and literal() cannot pass it through

serve: the per-IP WebSocket burst limit can't be configured, so workers behind one NAT IP throttle each other

S3-backed serve: a poll that pulls one file invalidates the whole data catalog, so the next query re-reads every version

extensions_search index is never created if the first attempt fails, until the process restarts

S3/GCS datastore sync: shared sync service lets one sync clear another's dirty marks and pull over unpushed files

swamp-uat: the SIGTERM drain test does not check that the run succeeded

stagecraft: a factory-wide read of work items waiting on a person (no studio, no per-item status loop)

stagecraft: rebind a work item to another tracker (mirrored built-in tickets to Jira)

s3-datastore: first push on a fresh cache prints 'Index merge failed' for a missing local .datastore-index.json

swamp-extensions: use explicit npm:/jsr: specifiers in root extensions/models