Skip to main content
← Back to list
01Issue
BugShippedSwamp CLIPublicTeam
Assigneesstack72

Relationships

#1827 data search: full filesystem walk on every call, bypasses catalog index

Opened by webframp · 8/25/2026· Shipped 8/26/2026

Summary

data search calls findAllGlobal() on every invocation, triggering a full recursive filesystem walk of .swamp/data/ regardless of filters or --limit. On repos with large data histories this takes 20+ seconds per call. The data query command avoids this by using the SQLite catalog (_catalog.db) for indexed lookups — data search should do the same.

How We Hit This

Testing swamp serve with a 2335-model repo (confirming the OOM fix from #1684 works — it does, great work). Through serve, data search --limit 5 consistently takes ~21 seconds while data query with a scoped predicate returns in 1.5 seconds. The limit value has no effect on timing.

Reproduction

$ time swamp data search --limit 5 --server ws://127.0.0.1:9090 --json | jq '.results | length'
5
real    0m21.5s

$ time swamp data search --limit 1 --server ws://127.0.0.1:9090 --json | jq '.results | length'
1
real    0m21.1s

$ time swamp data query 'modelName == "burnin-sys-07" && isLatest == true' --server ws://127.0.0.1:9090 --json | jq '.total'
1
real    0m1.5s

The repo has 234,492 metadata.yaml files across 255,481 directories in .swamp/data/. This is admittedly extreme (a loadgen fleet producing continuous output over several weeks), but the architectural gap exists at any scale — it's just invisible on small repos.

Root Cause

src/libswamp/data/search.ts line 280:

const allResults = await deps.findAllGlobal();

This calls UnifiedDataRepository.findAllGlobal() (src/infrastructure/persistence/unified_data_repository.ts:209) which runs collectAllData — a recursive Deno.readDir walk that descends through every type directory, every model-id directory, and calls findAllForModel on each one (which reads metadata.yaml for the latest version of each data item).

The full sequence:

  1. findAllGlobal() walks the entire filesystem tree (21 seconds on this repo)
  2. Each item gets enriched with its model name via findDefinitionById (line 283-288 of search.ts)
  3. filterData() applies filters in memory (line 310)
  4. Results are sorted by createdAt descending (line 313)
  5. .slice(0, limit) applies the limit (line 316)

The limit never short-circuits the scan because it's applied post-collection.

Contrast with data query

DataQueryService.query() (src/domain/data/data_query_service.ts:314) takes a different path:

  1. Checks catalogStore.isPopulated() — if not, runs a one-time backfillAsync() that populates the SQLite catalog from the filesystem
  2. After backfill, all subsequent queries execute against the indexed catalog via executeQuery(predicate, options) — sub-second regardless of repo size
  3. The backfill cost is amortized: paid once per serve boot (or per CLI process), then all queries are fast

data search never touches the catalog. It re-walks the filesystem on every call.

Suggested Fix

Route data search through DataQueryService (or the catalog directly) rather than findAllGlobal(). The search filters (type, lifetime, model, since, query string, etc.) can be expressed as catalog predicates or post-filter on catalog rows. This gives:

  • Indexed access instead of filesystem walk
  • Free pagination (limit applied before collection, not after)
  • Consistent performance regardless of repo size
  • One-time backfill cost shared with data query

The findAllGlobal path would remain as a fallback only when the catalog isn't populated and can't be (e.g., the initial backfill itself uses it).

Impact

  • Through serve: 21-second requests tie up I/O and inflate response times. The 30-second client-side WebSocket timeout makes unscoped data search fail entirely on large repos.
  • Local CLI: Same 21-second cost but more tolerable since there's no WebSocket serialization overhead or timeout. Still unnecessarily slow.
  • Concurrent requests through serve are NOT blocked — other WebSocket connections can interleave during the async walk. The issue is per-request latency, not server-wide blocking.

Environment

  • swamp 20260825.200043.0-sha.3ba23b09
  • Linux (NixOS 26.11)
  • Repo: 2335 models, 62 extension types, filesystem datastore
  • Data volume: 234,492 versioned data entries across 255,481 directories
  • Serve config: --auth-mode none --no-schedule
02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 2 MOREREVIEW+ 9 MOREPR_MERGED+ 2 MORESESSION_SUMMARIZED

Shipped

8/26/2026, 12:44:27 AM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack728/25/2026, 11:44:33 PM
Editable. Press Enter to edit.

stack72 commented 8/26/2026, 12:44:36 AM

Thanks @webframp for reporting this! The fix has been merged and a release is on its way. We appreciate your contribution to swamp.

Sign in to post a ripple.