Skip to main content
← Back to list
01Issue
BugShippedSwamp CLI
Assigneeskeeb

Relationships

#1120 serve: create*Deps() factories leak file-based catalog stores (3 FDs) per API request

Opened by keeb · 7/13/2026· Shipped 7/13/2026

Summary

Several create*Deps() factory functions open a file-based SQLite CatalogStore (via createCatalogStore) with no way to close it. In one-shot CLI commands this is harmless (the process exits and the OS reclaims the FDs), but under swamp serve these factories are invoked per API request, so every request against an affected endpoint permanently leaks file descriptors. A file-based SQLite store in WAL mode holds 3 FDs (.db + -wal + -shm), so each leak burns 3 descriptors.

This was discovered while investigating #1118 (the workflow log-sink FD leak) and is filed separately per the "keep the PR focused" scope decision. It is an independent leak with a different mechanism.

Mechanism

createCatalogStore(repoDir, datastoreResolver) returns a new CatalogStore(...) that opens a SQLite connection. Some create*Deps factories only call it when no shared repo is injected — e.g. createDataListDeps skips it when the serve handler passes [HOST-1]. But others call it unconditionally and never expose a close(), e.g.:

  • createModelEvaluateDeps — src/libswamp/models/evaluate.ts:99 opens createCatalogStore(...) unconditionally; the serve handler model_handlers.ts:1078 calls it per model.evaluate request and never closes the store.

The general anti-pattern: any serve handler that calls a create*Deps(ctx.repoDir, ...) factory without injecting the shared ctx.repoContext repo, where that factory internally opens createCatalogStore, leaks 3 FDs per request. The cli/commands/datastore_* call sites correctly pair createCatalogStore with .close(); the libswamp create*Deps factories used by serve handlers do not.

Impact

A serve instance that is polled on an affected endpoint (dashboard, monitoring UI, scripted clients) climbs toward its RLIMIT_NOFILE soft limit (default 1024) and eventually fails every Deno.open/Deno.readDir with Too many open files (os error 24) — the same end state as #1118, via a different leak.

Empirical note (from the #1118 investigation)

  • A file-based SQLite DB in WAL mode costs 3 OS FDs each (verified: 100 leaked file DBs → +300 FDs).
  • An in-memory (:memory:) SQLite DB costs 0 OS FDs — so ephemeral stores are not implicated here; only the file-based createCatalogStore path is.

Steps to reproduce

  1. Start swamp serve.
  2. Repeatedly call an endpoint whose handler builds deps via a create*Deps factory that opens createCatalogStore without an injected shared repo (e.g. model.evaluate).
  3. Watch ls -l /proc/<pid>/fd — 3 new _catalog.db / -wal / -shm handles accumulate per request and are never released.

Suggested fix

Audit every create*Deps factory that calls createCatalogStore, and for each serve handler that invokes one:

  • Give the factory a shared-repo escape hatch (accept an injected UnifiedDataRepository / CatalogStore and skip createCatalogStore when provided) — mirroring how createDataListDeps already does it — and have serve handlers pass ctx.repoContext, or
  • Return a cleanup()/close() from the factory (or the deps object) and have the handler call it in a finally.

Prefer the shared-repo injection so serve reuses the one long-lived catalog connection rather than opening and closing one per request.

Environment

  • swamp serve, Linux, Deno runtime. RLIMIT_NOFILE soft=1024 by default.
  • Related: #1118 (workflow log-sink FD leak — separate mechanism, same end state).
02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 5 MOREREVIEW+ 4 MOREPR_MERGED+ 1 MORENOTIFICATION_SKIPPED

Shipped

7/13/2026, 10:21:56 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
keeb assigned keeb7/13/2026, 9:17:10 PM

Sign in to post a ripple.