Skip to main content
← Back to list
01Issue
BugClosedSwamp CLI
Assigneesstack72

Relationships

#1094 swamp serve leaks esbuild bundler subprocesses as zombies (reparented to PID 1, never reaped)

Opened by magistr · 7/11/2026

Summary

A long-running swamp serve process accumulates zombie (defunct) esbuild processes that are never reaped. The bundler shells out to deno bundle (esbuild-powered); each bundle leaves esbuild's service subprocess to be orphaned and reparented to PID 1 — which, when PID 1 is swamp serve (the documented container deployment), is never wait()ed on. They pile up for the entire life of the process — roughly one per bundle — and only clear on restart.

Observed: 546 esbuild-linux-x <defunct> zombies on a swamp serve instance up ~3d21h (~140/day). The oldest is as old as the serve process itself.

Confirmed still present in the latest build 20260711.014414.0-sha.c3fd7b11 (source audit below).

Environment

  • swamp serve running 20260707.035329.0-sha.84dbdd93 (leak observed); root cause verified unchanged in 20260711.014414.0-sha.c3fd7b11 (latest).
  • Container debian:stable-slim; serve is PID 1 inside the container (/proc/<pid>/status → NSpid: <hostpid> 1); no init wrapper.
  • swamp serve --repo-dir /workspace --host [IP-2] --port 9090 --auth-mode token --admins user:magistr --ws-idle-timeout 5m --trust-proxy --log
  • Host: Unraid, kernel pid_max = 32768.

Evidence

Host process census, grouped by leaking parent → container:

totalZombies: 549
byParent:
  ppid <serve>  parentComm "swamp"     container swamp-serve  count 546  { "esbuild-linux-x <defunct>": 546 }
  ppid <traefik> parentComm "traefik"  container traefik      count 2    { "ssl_client <defunct>": 2 }   # unrelated
  ppid <atlas>  parentComm "beam.smp"  container atlas-app    count 1    { "epmd <defunct>": 1 }         # unrelated
  • All 546 esbuild zombies have PPID == the serve process (PID 1 in the container).
  • There are zero deno zombies — swamp correctly reaps the deno bundle children it spawns directly (await command.output()). Only the esbuild grandchild leaks.
  • ps -o etime on the oldest esbuild zombie ≈ 3-[IP-1] — defunct for the entire life of the serve process. A zombie persists only until its parent wait()s, so PID 1 (swamp serve) is provably never reaping these.

Mechanism

  1. bundleExtension() (src/domain/models/bundle.ts:506-515) runs new Deno.Command(denoPath, { args: ["bundle", ...] }) and await command.output(). It spawns and reaps the deno bundle child correctly.
  2. deno bundle is esbuild-powered and spawns an esbuild-linux-x64 service subprocess (grandchild of swamp).
  3. When deno bundle exits, its esbuild service is orphaned and reparented to PID 1. In the container PID 1 is swamp serve itself.
  4. swamp serve running as PID 1 installs no SIGCHLD reaper and is not a subreaper, so the reparented esbuild processes become permanent zombies.

Source audit of 20260711.014414.0-sha.c3fd7b11: no SIGCHLD / subreaper / waitpid / reaping logic exists in src/serve/**, src/domain/models/bundle.ts, or src/main.ts (the only "reap" references are run-tracker stale-run cleanup, unrelated to OS processes). So nothing collects reparented children.

Impact

  • Slow PID leak: ~140 zombies/day against a host-wide pid_max of 32768. Each holds a PID slot + task_struct until reaped. On a busy serve or a lower pid_max, PID exhaustion is reached correspondingly sooner.
  • Only remediation today is restarting the serve process/container (resets to zero, confirming the accumulation is per-process, not persistent state).

Reproduction

  1. Run swamp serve on a repo with user extension models that require bundling.
  2. Drive bundle operations over time (scheduled workflows / model-method runs that touch extensions — each triggers a deno bundle).
  3. On the host: ps -eo pid,ppid,stat,comm | grep esbuild → a monotonically growing set of Z (defunct) esbuild-linux-x whose PPID is the serve PID, none of which ever disappear.

Suggested fixes

  1. Serve-side reaper (preferred): when swamp serve is PID 1, install a SIGCHLD handler that wait()s reparented children (or run the Deno process as a subreaper), so orphaned esbuild services are collected.
  2. Bundler-side: ensure the esbuild service is torn down before/with the deno bundle child so it is never orphaned (may be an upstream deno bundle issue — it shouldn't leave its esbuild service running past exit).
  3. Deployment mitigation to document: run the serve container under an init (docker run --init, compose init: true, or tini). Because the zombies are reparented orphans (their real parent deno bundle has already exited), a PID-1 init reaper collects them — so this is a real workaround, and the docs/Dockerfile guidance for swamp serve should recommend it.

How this was diagnosed / how to verify a fix

Reproducible via a zombies method added to a @docker/host swamp model: it SSHes to the docker host, runs one ps census, groups defunct processes by leaking parent PID, and attributes each parent to its container via /proc/<ppid>/cgroup. After a fix (or with init: true), re-running it should report byComm["esbuild-linux-x <defunct>"] == 0 for container swamp-serve.

02Bog Flow
✓OPEN✓TRIAGED○IN PROGRESS◉CLOSED+ 1 MOREASSIGNEDCLASSIFICATION

Closed

7/11/2026, 10:15:24 PM

No activity in this phase yet.

03Sludge Pulse
stack72 assigned stack727/11/2026, 9:55:42 PM
Editable. Press Enter to edit.

stack72 commented 7/11/2026, 10:15:24 PM

@magistr Thanks for the incredibly detailed write-up and diagnosis — this is thorough work.

We've confirmed this is a container deployment configuration issue, not a swamp code bug. As you noted in your own analysis, swamp correctly reaps its direct children (deno bundle) — the zombie accumulation only happens because esbuild grandchildren get reparented to PID 1, and swamp isn't designed to be an init process.

We verified this by running swamp serve outside a container and triggering multiple bundle operations — zero esbuild zombies. The system init (launchd/systemd) reaps the orphans as expected.

The fix is on the deployment side, which you already identified in your suggested fix #3:

  • Docker Compose: add init: true to your service
  • Docker CLI: use docker run --init

Both inject tini as PID 1, which will reap the orphaned esbuild processes automatically. Since you're using your own base image (debian:stable-slim), not our Dockerfile, this is the right place to address it.

Sign in to post a ripple.