Skip to main content
← Back to list
01Issue
BugShippedSwamp CLI
Assigneesstack72

Relationships

#884 serve: catalog backfill is unbounded and unsafe on large repos

Opened by stack72 · 6/29/2026· Shipped 6/29/2026

Summary

The DataQueryService.backfillAsync() method loads all data across all model types into memory via findAllGlobal(), then iterates every version of every data item to build the SQLite catalog. On large repos (~2300 models, 1.9GB data directory), this can exhaust V8's heap.

#882 fixed the immediate OOM by removing the startup-path trigger (admin materializer no longer calls DataQueryService.query()). But the backfill itself is still unbounded — it will trigger on the first runtime DataQueryService.query() call when the catalog isn't populated, e.g.:

  • Generic data.query WebSocket endpoint (connection.ts:1503)
  • Capability service worker data query (capability_service.ts:225)
  • Grant/group list endpoints (connection.ts:1103, connection.ts:1166)
  • Worker token expiry lookup (worker_gateway.ts:618)

Root Cause

backfillAsync() (src/domain/data/data_query_service.ts:310) calls findAllGlobal() which returns all latest Data objects at once, then for each item calls listVersions() and findByName() for every historical version. The total allocation volume (YAML parse + Zod validation intermediate objects across potentially hundreds of thousands of versions) can exceed V8's GC throughput, leading to OOM.

What Needs Investigation

The exact memory profile of the backfill on a large repo needs profiling before designing a fix. The 4GB heap usage may come from:

  • Cumulative allocation churn from parsing ~100K+ metadata.yaml files
  • The CatalogRow[] array at scale (each row is ~20 scalar fields)
  • V8 GC not keeping up with the allocation rate in a tight async loop

Potential Approaches

  • Process in explicit batches with GC breathing room (yield to event loop between batches)
  • Lazy per-type cataloging (only populate model types a query actually needs)
  • Reduce work: only catalog latest versions, skip historical unless queried with is_latest == false
  • Bypass catalog for remaining type-scoped serve queries (grants, groups, tokens) using findAllForType() — same pattern as #841 and #882
  • #841 — original slow startup from PolicySnapshotLoader triggering backfill (fixed)
  • #882 — OOM from admin materializer triggering backfill (fixed)
02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 6 MOREREVIEW+ 4 MOREPR_MERGED+ 1 MORENOTIFICATION_SKIPPED

Shipped

6/29/2026, 1:10:38 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack726/29/2026, 11:03:16 AM

Sign in to post a ripple.