Skip to main content
← Back to list
01Issue
BugShippedSwamp CLIPublicTeam
Assigneesstack72

Relationships

#2026 writeCatalogExport takes 55-150s per workflow step — serializes entire catalog as pretty-printed JSON

Opened by magistr · 9/4/2026· Shipped 9/7/2026

Summary

writeCatalogExport in repo_context.ts runs in the per-step flush (via stepLockHook → acquireModelLocks → flush). It iterates every row in the SQLite catalog, serializes each with JSON.stringify(row, null, 2) (pretty-printed), and writes them one-by-one to .catalog-export.json.

At 102k rows (post-GC; was 255k before swamp run gc), this takes 55s. It runs after EVERY workflow step, making a 0.5s method take 55s+ wall clock.

Measured breakdown (fleet-health, single-step workflow)

Phase Duration
pullChanged (with per-prefix watermark fix) 0.5s
Method execution <1s
writeCatalogExport (102k rows) 55s
pushChanged to MongoDB 5-10s

Root cause

  1. Full snapshot every step — the export writes ALL catalog rows, not just the ones that changed. For a step that writes 3 data artifacts, 102k unchanged rows are re-serialized.
  2. Pretty-printed JSON — JSON.stringify(row, null, 2) produces ~3x more bytes than compact JSON. The file is consumed by swamp datastore catalog pull, which doesn't need indentation.
  3. Per-row file.write — each row is a separate Deno.writeFile call instead of buffered/batched writes.

Suggested fixes (any one would help, all three together would bring it to <1s)

  1. Incremental export — only re-export rows that changed since the last export (track a catalog version/sequence number)
  2. Compact JSON — JSON.stringify(row) instead of JSON.stringify(row, null, 2)
  3. Buffer writes — accumulate rows in memory and write once, or use a streaming JSON writer with a write buffer

Context

  • Serve: v20260904.044433.0
  • Catalog: 102k rows (after swamp run gc --older-than 3d; was 255k before)
  • The pull-side bottleneck (per-prefix watermarks) was fixed in keeb/swamp-mongodb-datastore@9fde8a9 — this is the remaining overhead
02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 5 MOREREVIEW+ 7 MOREPR_MERGED+ 2 MORESESSION_SUMMARIZED

Shipped

9/7/2026, 9:01:30 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack729/7/2026, 6:41:27 PM
Editable. Press Enter to edit.

stack72 commented 9/7/2026, 9:01:47 PM

Thanks @magistr for reporting this! The fix has been merged and a release is on its way. We appreciate your contribution to swamp.

Sign in to post a ripple.