#1094 swamp serve leaks esbuild bundler subprocesses as zombies (reparented to PID 1, never reaped)

Summary

A long-running swamp serve process accumulates zombie (defunct) esbuild processes that are never reaped. The bundler shells out to deno bundle (esbuild-powered); each bundle leaves esbuild's service subprocess to be orphaned and reparented to PID 1 — which, when PID 1 is swamp serve (the documented container deployment), is never wait()ed on. They pile up for the entire life of the process — roughly one per bundle — and only clear on restart.

Observed: 546 esbuild-linux-x <defunct> zombies on a swamp serve instance up ~3d21h (~140/day). The oldest is as old as the serve process itself.

Confirmed still present in the latest build 20260711.014414.0-sha.c3fd7b11 (source audit below).

Environment

• swamp serve running 20260707.035329.0-sha.84dbdd93 (leak observed); root cause verified unchanged in 20260711.014414.0-sha.c3fd7b11 (latest).
• Container debian:stable-slim; serve is PID 1 inside the container (/proc/<pid>/status → NSpid: <hostpid> 1); no init wrapper.
• swamp serve --repo-dir /workspace --host [IP-2] --port 9090 --auth-mode token --admins user:magistr --ws-idle-timeout 5m --trust-proxy --log
• Host: Unraid, kernel pid_max = 32768.

Evidence

Host process census, grouped by leaking parent → container:

totalZombies: 549
byParent:
  ppid <serve>  parentComm "swamp"     container swamp-serve  count 546  { "esbuild-linux-x <defunct>": 546 }
  ppid <traefik> parentComm "traefik"  container traefik      count 2    { "ssl_client <defunct>": 2 }   # unrelated
  ppid <atlas>  parentComm "beam.smp"  container atlas-app    count 1    { "epmd <defunct>": 1 }         # unrelated

• All 546 esbuild zombies have PPID == the serve process (PID 1 in the container).
• There are zero deno zombies — swamp correctly reaps the deno bundle children it spawns directly (await command.output()). Only the esbuild grandchild leaks.
• ps -o etime on the oldest esbuild zombie ≈ 3-[IP-1] — defunct for the entire life of the serve process. A zombie persists only until its parent wait()s, so PID 1 (swamp serve) is provably never reaping these.

Mechanism

1. bundleExtension() (src/domain/models/bundle.ts:506-515) runs new Deno.Command(denoPath, { args: ["bundle", ...] }) and await command.output(). It spawns and reaps the deno bundle child correctly.
2. deno bundle is esbuild-powered and spawns an esbuild-linux-x64 service subprocess (grandchild of swamp).
3. When deno bundle exits, its esbuild service is orphaned and reparented to PID 1. In the container PID 1 is swamp serve itself.
4. swamp serve running as PID 1 installs no SIGCHLD reaper and is not a subreaper, so the reparented esbuild processes become permanent zombies.

Source audit of 20260711.014414.0-sha.c3fd7b11: no SIGCHLD / subreaper / waitpid / reaping logic exists in src/serve/**, src/domain/models/bundle.ts, or src/main.ts (the only "reap" references are run-tracker stale-run cleanup, unrelated to OS processes). So nothing collects reparented children.

Impact

• Slow PID leak: ~140 zombies/day against a host-wide pid_max of 32768. Each holds a PID slot + task_struct until reaped. On a busy serve or a lower pid_max, PID exhaustion is reached correspondingly sooner.
• Only remediation today is restarting the serve process/container (resets to zero, confirming the accumulation is per-process, not persistent state).

Reproduction

1. Run swamp serve on a repo with user extension models that require bundling.
2. Drive bundle operations over time (scheduled workflows / model-method runs that touch extensions — each triggers a deno bundle).
3. On the host: ps -eo pid,ppid,stat,comm | grep esbuild → a monotonically growing set of Z (defunct) esbuild-linux-x whose PPID is the serve PID, none of which ever disappear.

Suggested fixes

1. Serve-side reaper (preferred): when swamp serve is PID 1, install a SIGCHLD handler that wait()s reparented children (or run the Deno process as a subreaper), so orphaned esbuild services are collected.
2. Bundler-side: ensure the esbuild service is torn down before/with the deno bundle child so it is never orphaned (may be an upstream deno bundle issue — it shouldn't leave its esbuild service running past exit).
3. Deployment mitigation to document: run the serve container under an init (docker run --init, compose init: true, or tini). Because the zombies are reparented orphans (their real parent deno bundle has already exited), a PID-1 init reaper collects them — so this is a real workaround, and the docs/Dockerfile guidance for swamp serve should recommend it.

How this was diagnosed / how to verify a fix

Reproducible via a zombies method added to a @docker/host swamp model: it SSHes to the docker host, runs one ps census, groups defunct processes by leaking parent PID, and attributes each parent to its container via /proc/<ppid>/cgroup. After a fix (or with init: true), re-running it should report byComm["esbuild-linux-x <defunct>"] == 0 for container swamp-serve.