#1828 serve: catalog backfill indexes only session-local data, data query returns wrong results for pre-existing entries

Summary

When swamp serve boots on a filesystem-datastore repo with pre-existing data (produced by prior CLI sessions), the catalog backfill (backfillAsync) marks the catalog as populated=true but indexes only data written during the current serve session. Pre-existing data entries are invisible to data query through serve, producing incorrect results (not slow — wrong).

How We Hit This

Testing serve at scale (2335 models, 62 extension types, 234,492 data entries) to verify the OOM fix from #1684. Serve boots and handles requests fine, but data query 'true' returns only 9 results through serve despite 1,051,337 catalog rows visible on disk from prior CLI usage.

Steps to Reproduce

1. Build up data history via CLI: run model methods over time so .swamp/data/ accumulates entries (our repo has 234,492 metadata.yaml files across 255,481 directories)
2. Confirm the catalog is populated locally: sqlite3 .swamp/data/_catalog.db "SELECT COUNT(*) FROM catalog;" → 1,051,337 rows, populated=true
3. Start serve: swamp serve --auth-mode none --no-schedule
4. Query through serve: swamp data query 'true' --server ws://127.0.0.1:9090 --json
5. Result: {"total": 9, "limited": false} — only entries written during THIS serve session

For comparison, scoped queries that bypass the catalog work correctly:

• swamp data versions loadgen-02-loop-worker-td loadgen-02-loop-worker-td --server ws://... --json → 1815 versions (correct)
• swamp data list loadgen-02-loop-worker-td --server ws://... --json → correct groups

The data repo itself sees the full filesystem. The catalog does not.

Root Cause Analysis

Boot sequence invalidates the catalog

src/cli/commands/serve.ts:1695-1699:

await hydrateLocalCache({
  syncService,
  catalogInvalidate: () => repoContext.catalogStore.invalidate(),
  signal: AbortSignal.timeout(hydrationTimeoutMs),
  namespace: serveNamespace,
});

src/serve/boot_reconciliation.ts:99 — hydrateLocalCache always calls deps.catalogInvalidate() after pullChanged, clearing the populated flag in SQLite regardless of whether any files were pulled. On a filesystem-only datastore (no remote), pullChanged returns 0 files immediately, then the catalog is invalidated anyway.

Backfill produces near-empty results

After invalidation, the first data query triggers backfillAsync() in src/domain/data/data_query_service.ts:318-327:

if (!this.catalogStore.isPopulated()) {
  const promise = this.backfillAsync();
  this.backfillPromise = promise;
  await promise;
}

backfillAsync() (line 592) calls this.dataRepo.findAllGlobal() which walks the filesystem. In our testing, this backfill completed and called markPopulated(), but the catalog ended up with only 9 rows (the entries produced by model method runs we executed during this serve session — burnin-sys-07 get_os_info, etc.).

The discrepancy

• data versions for a specific model returns 1815 versions through serve (correct) — this uses dataRepo.listVersions() which is a direct filesystem read
• data query 'true' returns 9 results (wrong) — this uses the catalog which was backfilled from findAllGlobal() within the serve context
• The on-disk _catalog.db shows 1,051,337 rows and populated=true — but this was written by the CLI, and the serve process may be operating on a different in-memory state or the WAL read snapshot

Possible explanations we could not fully confirm

1. WAL snapshot isolation: CatalogStore uses WAL mode. The serve process opens the database at boot, gets a read snapshot, then the CLI (or serve's own backfill) writes 1M rows — but the serve process's read connection never sees them because it's pinned to the old snapshot. The 9 rows it does see are from its own writes (same connection).

2. Namespace filtering: backfillAsync calls findAllGlobal() which walks all data dirs — but the serve process's UnifiedDataRepository may be constructed with a namespace that causes it to skip entries lacking a namespace tag. The 9 entries it finds are the ones it wrote with its own namespace stamp.

3. Concurrent write interference: The bulkUpsert of 9 rows + markPopulated() wins a race against a still-running findAllGlobal() walk from a different code path (e.g., the data search calls we ran concurrently).

We could not determine which explanation is correct from the code alone — it would require adding logging to backfillAsync within a running serve process to see exactly how many items findAllGlobal() returns and what gets passed to bulkUpsert.

Verified behavior

The type-scoped query times out because with only 9 catalog rows, the predicate matches nothing from the catalog — but the DataQueryService may attempt another backfill or full scan to satisfy it.

Impact

data query through serve returns incorrect results for any predicate that doesn't trigger the scoped-backfill fast path. The scoped path (triggered by modelName == "X" predicates) works because getLatestRecord in DataQueryService (line 178-221) does a targeted single-model backfill that bypasses the broken full backfill. But broad predicates (true, type-scoped, tag-scoped) all return wrong results or time out.

This affects any serve deployment on a filesystem datastore with pre-existing data history.

Environment

• swamp 20260825.200043.0-sha.3ba23b09
• Linux (NixOS 26.11)
• Repo: 2335 models, 62 extension types, filesystem datastore
• Data volume: 234,492 data entries, 1,051,337 catalog rows (from CLI history)
• Serve config: --auth-mode none --no-schedule