#1118 serve leaks file descriptors (EMFILE) — runFileSink.unregister runs after yield, not in finally

Summary

A long-running swamp serve process leaks file descriptors and eventually fails every operation with Too many open files (os error 24). The leaked resource is the per-run workflow log file opened by runFileSink.register(). The root cause is that runFileSink.unregister() in WorkflowExecutionService.run() and .resume() is called after yield statements instead of in a finally block, so when a consumer abandons the event generator early the descriptor is never closed.

Symptom

Scheduled workflows begin failing in bulk, all at the first filesystem access:

error  scheduled-execution  Scheduled run failed for workflow "organize-and-clean":
  "Too many open files (os error 24): readdir '/home/[REDACTED]/git/swamp-media/workflows'"

The readdir is a red herring — it is simply the first syscall in the fire path that needs a new FD. Once the process is at its RLIMIT_NOFILE soft limit, every Deno.readDir/Deno.open fails, so all scheduled workflows fail at every tick and never recover until restart (leaked FDs are held for the life of the process). Observed on a serve instance whose soft limit is the default 1024 (hard limit 524288).

Root cause

src/domain/workflows/execution_service.ts — run() registers the workflow log sink and then closes it at scattered points that follow yields rather than in a finally:

workflowLogHandle = await runFileSink.register(...);   // ~1634, opens a Deno.open FD
...
yield { kind: "completed", run };                      // 1860
await this.saveRun(workflow.id, run);
runFileSink.unregister(workflowLogHandle);             // 1869 — NOT in a finally

run() (and resume()) are async generators. When a consumer stops iterating after the terminal event, Deno invokes generator.return(), which unwinds only finally blocks. Every runFileSink.unregister() sits after a yield (lines ~1829, 1869, 1880, 1906, 1921 in run(); ~2184, 2206, 2213, 2233, 2242 in resume()), so they are skipped and the log-file descriptor leaks.

Trigger

Streaming serve handlers break out of the event loop when the client socket closes, abandoning the generator:

• src/serve/handlers/workflow_handlers.ts:859 (workflow resume) — if (socket.readyState !== WebSocket.OPEN) break;
• src/serve/handlers/model_handlers.ts:206

Any client (TUI/dashboard/monitoring UI) that connects to watch a run and then disconnects mid-run leaks one FD per occurrence. An exception thrown while a consumer processes a yielded event has the same effect.

Note: the purely-scheduled path (executeWorkflowWithLocks) and the primary workflow.run handler drain the generator fully, so they do not trigger the leak on their own. The model-method run path is already safe because its runLog.cleanup() is in a finally (src/libswamp/models/run.ts:1139) — the workflow-level sink is the one that isn't.

What was ruled out (to save the next investigator time)

• Deno.readDir early-return (e.g. unified_data_repository.ts:389,482) does not leak — Deno calls the iterator's .return() on break/return. Verified empirically: 500 early-return loops → 0 FD growth.
• The ephemeral :memory: store holds 0 OS file descriptors even when leaked. (A file-based sqlite with WAL costs 3 FDs: .db + -wal + -shm.)

Secondary findings (separate, lower priority)

• Several create*Deps() factories open a file-based catalog store (3 FDs) with no close() — e.g. createModelEvaluateDeps (src/libswamp/models/evaluate.ts:99). Serve handlers that call these with only ctx.repoDir (instead of injecting the shared ctx.repoContext repo, the way createDataListDeps does) leak 3 FDs per API request. model.evaluate (model_handlers.ts:1078) is one such endpoint. Harmless in one-shot CLI (process exits), a leak under serve.
• src/libswamp/workflows/run.ts:491 creates the ephemeral store before its try (which starts at 569), so the early returns at 506/529 skip dispose(). :memory: = 0 FDs, so this is a minor memory/handle leak, not the EMFILE cause.

Steps to reproduce

1. Start swamp serve (default soft RLIMIT_NOFILE of 1024).
2. Open a WebSocket to stream a workflow resume (or model run) and disconnect before the run completes. Repeat.
3. Watch open FDs climb: ls -l /proc/<pid>/fd shows accumulating workflow-run-*.log handles that are never released.
4. After enough leaks the process crosses 1024 open FDs and every subsequent operation fails with Too many open files (os error 24).

Suggested fix

• Move every runFileSink.unregister(workflowLogHandle) in execution_service.ts run() and resume() into the outer finally (alongside runSpan.end()), so it always runs when the generator is disposed — mirroring how the model-method path already uses finally.
• Separately, give the leaking create*Deps factories a shared-repo escape hatch (or close() the store in the handler), and hoist the ephemeral-store creation in run.ts inside its try.

Mitigation

Restart serve, and raise the soft FD limit above the 1024 default (hard limit is already 524288, so no privilege change needed): systemd LimitNOFILE=1048576, or ulimit -n 1048576 before launch.

Environment

• swamp serve (scheduled workflows enabled), Linux, Deno runtime.
• RLIMIT_NOFILE soft=1024, hard=524288 (default).