Skip to main content
← Back to list
01Issue
BugShippedSwamp Club
Assigneesstack72

Relationships

#1049 Catalog export re-uploads the full monolithic object every sync, even with zero changes (sibling to #1034)

Opened by mgreten · 7/9/2026· Shipped 7/9/2026

First — thank you for shard-first v2 and the #1034 fixes. We migrated our whole fleet to v2 today and the index side is exactly what we hoped: pullChanged/pushChanged fast-path in ~100-240ms on a namespace that used to take minutes. This report is a sibling to #1034: with the index now sharded and skipped-when-clean, the catalog export is the last monolithic object that gets rewritten on every sync regardless of whether anything changed.

What we observe

On a completely no-op sync — Sync complete: 0 pulled, 0 pushed, all three index phases fast-path hit — the catalog is still fully re-exported:

[s3-sync] pullChanged.fastpath 240ms hit
[s3-sync] pushChanged.fastpath 106ms hit
Sync complete: 0 pulled, 0 pushed
[s3-sync] pushChanged.fastpath 184ms hit
INF datastore·sync Exported catalog (12420 row(s)) for namespace "agentic-tooling"

We confirmed it's a real rewrite, not just a log line: the local {namespace}/.catalog-export.json mtime advances on every zero-change sync, and exportCatalog in s3_cache_sync.ts unconditionally serializes all rows and putObjects them:

async exportCatalog(namespace, rows, signal) {
  const key = `${namespace}/.catalog-export.json`;
  const data = new TextEncoder().encode(JSON.stringify(rows));
  await retryWithBackoff(() => this.s3.putObject(key, data, signal), { signal });
}

There's no dirty check, no commitSeq, no sharding — so unlike the index, the full catalog is re-uploaded every sync. For us that's a 9.8MB object (12,420 rows) PUT on every no-op sync. Over our ~1.1MiB/s uplink that's ~9s of transfer per no-op sync (the export itself is ~2-3s local), and it scales with catalog size — in #1034 we'd seen this same object reach ~135MB before we vacuumed. The unconditional re-PUT happens on every sync regardless of change.

Asks (all gently — entirely your call on priority/approach)

  • Could the catalog export be skipped when nothing changed, the way the shard-first index now skips on a clean commitSeq? A per-namespace catalog fingerprint/seq (write only when it differs from the last exported one) would make a no-op sync genuinely no-op.
  • Failing that, would incremental/sharded catalog export (mirroring the index partitioning) be feasible, so only changed rows move?
  • Smaller: JSON.stringify(rows) here is compact already (good), but the object is unconditional — the win is in skipping it, not shrinking it.

This is low-severity for us now that hot-path writers are on a local FS datastore and the raised SWAMP_S3_REQUEST_TIMEOUT_MS keeps the PUT from timing out — so no urgency at all. We just wanted to surface it as the natural next increment after the index work, since it's the same "monolithic object rewritten every push" shape #1034 addressed for the index. Happy to share full SWAMP_S3_SYNC_TRACE=1 captures, sizes, and before/after timings, and to test any candidate build against our two-machine MinIO-over-Tailscale setup.

Environment

  • Extension: @swamp/s3-datastore@2026.07.08.3 (shard-first v2, migrated 2026-07-08)
  • swamp: 20260708.012813.0-sha.0f7166fe
  • OS: darwin (aarch64)
  • Setup: MinIO over Tailscale (~95ms RTT, ~1.1MiB/s uplink from the Mac), two writer machines
  • Related: sibling to #1034 (which fixed the index side)

Upstream repository: https://github.com/systeminit/swamp-extensions

Environment

  • Extension: @swamp/s3-datastore@2026.07.08.3
  • swamp: 20260708.012813.0-sha.0f7166fe
  • OS: darwin (aarch64)
  • Deno: 2.8.3
  • Shell: /bin/zsh
02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 2 MOREREVIEW+ 4 MOREPR_MERGED+ 1 MORECONTRIBUTOR_NOTIFIED

Shipped

7/9/2026, 4:38:33 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack727/9/2026, 2:28:37 PM
Editable. Press Enter to edit.

stack72 commented 7/9/2026, 4:38:41 PM

Thanks @mgreten for reporting this! The fix has been merged and a release is on its way. We appreciate your contribution to swamp.

mgreten commented 7/9/2026, 4:53:46 PM

Confirmed working — thank you! We updated the fleet to 2026.07.09.1 and verified the fix against S3 ground truth (not just the local trace, which turned out to be a red herring worth flagging for the next person).

What we verified: exportCatalog now SHA-256s the serialized rows and skips the putObject when it matches lastCatalogHash — exactly the fingerprint approach. We confirmed the S3 object is genuinely no longer re-uploaded: its LastModified stays pinned across repeated no-op syncs:

S3 LastModified BEFORE:            2026-07-09T15:51:27+00:00
S3 LastModified after 2 no-op syncs: 2026-07-09T15:51:27+00:00   # unchanged — PUT skipped

So the ~9.8MB upload is gone from no-op syncs. On our slow uplink that's the win we were after.

One heads-up for future diagnosis (in case it saves someone the confusion it gave us): the CLI still writes the local {namespace}/.catalog-export.json and still logs Exported catalog (N rows) on every sync — so the local file's mtime advances and the log line appears even when the S3 PUT is correctly skipped. We briefly thought the fix hadn't taken because of those local-layer signals; checking the S3 object's LastModified is what confirmed the gate is working. If it's ever an easy win, suppressing the local rewrite/log on a skip would make the no-op path read as a true no-op — but that's cosmetic, and the actual transfer cost is solved. Thanks again for the quick turnaround on this.

Sign in to post a ripple.