Skip to main content
← Back to list
01Issue
BugShippedSwamp CLIPublicTeam
Assigneesstack72

Relationships

#1828 serve: catalog backfill indexes only session-local data, data query returns wrong results for pre-existing entries

Opened by webframp · 8/25/2026· Shipped 8/26/2026

Summary

When swamp serve boots on a filesystem-datastore repo with pre-existing data (produced by prior CLI sessions), the catalog backfill (backfillAsync) marks the catalog as populated=true but indexes only data written during the current serve session. Pre-existing data entries are invisible to data query through serve, producing incorrect results (not slow — wrong).

How We Hit This

Testing serve at scale (2335 models, 62 extension types, 234,492 data entries) to verify the OOM fix from #1684. Serve boots and handles requests fine, but data query 'true' returns only 9 results through serve despite 1,051,337 catalog rows visible on disk from prior CLI usage.

Steps to Reproduce

  1. Build up data history via CLI: run model methods over time so .swamp/data/ accumulates entries (our repo has 234,492 metadata.yaml files across 255,481 directories)
  2. Confirm the catalog is populated locally: sqlite3 .swamp/data/_catalog.db "SELECT COUNT(*) FROM catalog;" → 1,051,337 rows, populated=true
  3. Start serve: swamp serve --auth-mode none --no-schedule
  4. Query through serve: swamp data query 'true' --server ws://127.0.0.1:9090 --json
  5. Result: {"total": 9, "limited": false} — only entries written during THIS serve session

For comparison, scoped queries that bypass the catalog work correctly:

  • swamp data versions loadgen-02-loop-worker-td loadgen-02-loop-worker-td --server ws://... --json → 1815 versions (correct)
  • swamp data list loadgen-02-loop-worker-td --server ws://... --json → correct groups

The data repo itself sees the full filesystem. The catalog does not.

Root Cause Analysis

Boot sequence invalidates the catalog

src/cli/commands/serve.ts:1695-1699:

await hydrateLocalCache({
  syncService,
  catalogInvalidate: () => repoContext.catalogStore.invalidate(),
  signal: AbortSignal.timeout(hydrationTimeoutMs),
  namespace: serveNamespace,
});

src/serve/boot_reconciliation.ts:99 — hydrateLocalCache always calls deps.catalogInvalidate() after pullChanged, clearing the populated flag in SQLite regardless of whether any files were pulled. On a filesystem-only datastore (no remote), pullChanged returns 0 files immediately, then the catalog is invalidated anyway.

Backfill produces near-empty results

After invalidation, the first data query triggers backfillAsync() in src/domain/data/data_query_service.ts:318-327:

if (!this.catalogStore.isPopulated()) {
  const promise = this.backfillAsync();
  this.backfillPromise = promise;
  await promise;
}

backfillAsync() (line 592) calls this.dataRepo.findAllGlobal() which walks the filesystem. In our testing, this backfill completed and called markPopulated(), but the catalog ended up with only 9 rows (the entries produced by model method runs we executed during this serve session — burnin-sys-07 get_os_info, etc.).

The discrepancy

  • data versions for a specific model returns 1815 versions through serve (correct) — this uses dataRepo.listVersions() which is a direct filesystem read
  • data query 'true' returns 9 results (wrong) — this uses the catalog which was backfilled from findAllGlobal() within the serve context
  • The on-disk _catalog.db shows 1,051,337 rows and populated=true — but this was written by the CLI, and the serve process may be operating on a different in-memory state or the WAL read snapshot

Possible explanations we could not fully confirm

  1. WAL snapshot isolation: CatalogStore uses WAL mode. The serve process opens the database at boot, gets a read snapshot, then the CLI (or serve's own backfill) writes 1M rows — but the serve process's read connection never sees them because it's pinned to the old snapshot. The 9 rows it does see are from its own writes (same connection).

  2. Namespace filtering: backfillAsync calls findAllGlobal() which walks all data dirs — but the serve process's UnifiedDataRepository may be constructed with a namespace that causes it to skip entries lacking a namespace tag. The 9 entries it finds are the ones it wrote with its own namespace stamp.

  3. Concurrent write interference: The bulkUpsert of 9 rows + markPopulated() wins a race against a still-running findAllGlobal() walk from a different code path (e.g., the data search calls we ran concurrently).

We could not determine which explanation is correct from the code alone — it would require adding logging to backfillAsync within a running serve process to see exactly how many items findAllGlobal() returns and what gets passed to bulkUpsert.

Verified behavior

Operation Through serve Through local CLI
data query 'true' 9 results (would return 1M+ if run without limit)
data query 'modelName == "X" && isLatest == true' 1 result (correct, uses scoped backfill) 1 result
data versions model dataName 1815 (correct, direct filesystem) 1815
data list model correct groups correct
data query 'modelType == "@webframp/aws/networking" && isLatest == true' timeout (30s) works

The type-scoped query times out because with only 9 catalog rows, the predicate matches nothing from the catalog — but the DataQueryService may attempt another backfill or full scan to satisfy it.

Impact

data query through serve returns incorrect results for any predicate that doesn't trigger the scoped-backfill fast path. The scoped path (triggered by modelName == "X" predicates) works because getLatestRecord in DataQueryService (line 178-221) does a targeted single-model backfill that bypasses the broken full backfill. But broad predicates (true, type-scoped, tag-scoped) all return wrong results or time out.

This affects any serve deployment on a filesystem datastore with pre-existing data history.

Environment

  • swamp 20260825.200043.0-sha.3ba23b09
  • Linux (NixOS 26.11)
  • Repo: 2335 models, 62 extension types, filesystem datastore
  • Data volume: 234,492 data entries, 1,051,337 catalog rows (from CLI history)
  • Serve config: --auth-mode none --no-schedule
02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 4 MOREREVIEW+ 7 MOREPR_MERGED+ 2 MORESESSION_SUMMARIZED

Shipped

8/26/2026, 4:58:05 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack728/26/2026, 12:05:11 AM
Editable. Press Enter to edit.

stack72 commented 8/26/2026, 4:58:16 PM

Thanks @webframp for reporting this! The fix has been merged and a release is on its way. We appreciate your contribution to swamp.

Sign in to post a ripple.