Skip to main content
← Back to list
01Issue
BugShippedExtensions
Assigneesstack72

Relationships

#1034 Shard-first v2 still re-uploads the full monolithic index + catalog export every commit; silent writeback failure above the 30s request timeout

Opened by mgreten · 7/8/2026· Shipped 7/8/2026

Thanks again for shard-first v2 — after migrating today, scoped pulls are wonderfully fast (pullIndex ~423ms on a namespace that used to take minutes). This report is about the push side, which we traced today while chasing an 11-minute wall time on a trivial write.

What we observed

Post-migration (shard-first v2, _meta.json v2, 288 shards), a model method run whose method completed instantly still spent >11 minutes in the push phase, with no _index/ shard writes happening during the stall.

What we traced (s3.js bundle, v2026.07.02.1)

  1. Even in v2 mode, every commit still re-uploads the full monolithic .datastore-index.json (in our case 49.9MB) — the shard-first path doesn't skip the back-compat write.
  2. The catalog export was the second large per-push object: our local _catalog.db had grown to ~256k rows, producing a ~135MB export per push.
  3. With DEFAULT_REQUEST_TIMEOUT_MS = 30000 hardcoded (not in the config schema, no env var) and retryWithBackoff treating TimeoutError as retryable (3 attempts), any object over ~33MB on our ~1.1MiB/s uplink can never succeed — it deterministically times out three times, and the failure is swallowed. Net effect on our two-machine setup: the Mac's monolith writebacks silently failed for weeks (only the Linux box, local to MinIO, was actually updating it), and every Mac push burned minutes in the retry loop.
  4. preparePush/commitPush have no tracePhase instrumentation, so SWAMP_S3_SYNC_TRACE=1 (which was very helpful elsewhere — thank you for it) goes silent exactly where the time is spent.

After pruning our index (monolith 49.9MB → 10.7MB) and vacuuming the catalog (export 135MB → 6.5MB), the same push dropped to 1:43 — so the residual push cost scales with the monolith + catalog export sizes rather than with the change being pushed.

Asks (all gently — your call on priority/approach)

  1. Could the v2 path skip (or make optional / write incrementally) the monolithic back-compat index? For repos fully on v2 it appears to be pure overhead, and it's the object most likely to cross the timeout cliff.
  2. Could requestTimeoutMs be exposed via config or env? (Closely related to #1008 — same hardcoded constant.)
  3. Would trace points in preparePush/commitPush be reasonable? The silent phases made this diagnosis take much longer than it needed to.
  4. Smaller thought: shard/monolith JSON is pretty-printed (JSON.stringify(x, null, 2)), which we measured inflating objects ~30-40% — compact output would buy real headroom below the timeout ceiling.

We have full SWAMP_S3_SYNC_TRACE captures, before/after timings, and size breakdowns, and we're glad to test any candidate build against this setup.

Environment

CLI 20260708.012813 / @swamp/s3-datastore 2026.07.02.1 / MinIO over Tailscale (95ms RTT, ~1.1MiB/s uplink from the Mac), two writer machines, shard-first v2 migrated 2026-07-08.

Upstream repository: https://github.com/systeminit/swamp-extensions

Environment

  • Extension: @swamp/s3-datastore@2026.07.02.1
  • swamp: 20260708.012813.0-sha.0f7166fe
  • OS: darwin (aarch64)
  • Deno: 2.8.3
  • Shell: /bin/zsh
02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 2 MOREREVIEW+ 4 MOREPR_MERGED+ 1 MORECONTRIBUTOR_NOTIFIED

Shipped

7/8/2026, 10:38:40 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack727/8/2026, 8:51:37 PM
Editable. Press Enter to edit.

stack72 commented 7/8/2026, 10:38:51 PM

Thanks @mgreten for reporting this! The fix has been merged and a release is on its way. We appreciate your contribution to swamp.

mgreten commented 7/8/2026, 10:52:05 PM

Thank you — the fast turnaround on this is genuinely appreciated. We migrated all four fleet slots to 2026.07.08.3 and re-measured against the same MinIO-over-Tailscale setup (~95ms RTT, ~1.1MiB/s uplink from the Mac). Two of the asks are already paying off:

  • Configurable request timeout — requestTimeoutMs config + SWAMP_S3_REQUEST_TIMEOUT_MS env both work. We've set 180s on our cold-warehouse writers and the deterministic time-out-then-swallow failure is gone. This alone fixes the silent-writeback-for-weeks symptom, so thank you.
  • preparePush / commitPush trace points — these fire now, and they made the residual below diagnosable in one run instead of a fishing expedition. Really glad to have them.

One observation we wanted to share as data (no expectation — entirely your call whether it's worth a follow-up): the full monolithic-index re-upload on every changed push still appears to happen on our v2-migrated repo. The registry description mentions "v2 repos skip the monolithic index upload entirely," but a swamp model method run that only re-versions one artifact traces as:

[s3-sync] preparePush.fastpath 181ms hit
[s3-sync] commitPush 30201ms noop      # at the 30s default — timed out + swallowed

and with the timeout raised to 180s the same push completes rather than timing out:

[s3-sync] preparePush.fastpath 153ms hit
[s3-sync] commitPush 23344ms noop      # ~23s: the ~10.7MB monolith going up over ~1.1MiB/s

So commitPush reports noop (no shard diff) yet spends ~23s — consistent with the full .datastore-index.json PUT happening unconditionally rather than being skipped on v2. We're not blocked (the raised timeout is a fine mitigation for our batch writers, and hot-path writers are on a local FS store), so please treat this purely as a data point on whether the v2 monolith-skip landed as intended — you may already know it's staged for a later increment.

We also noticed the compact-JSON change for the S3 index upload — nice, that's real headroom below the ceiling. Thank you again for all of it. Happy to share the full SWAMP_S3_SYNC_TRACE=1 captures or test any candidate build against this setup whenever useful.

mgreten commented 7/8/2026, 10:58:16 PM

Correction to my previous ripple — I owe you one, and it was my error, not a gap in your fix. Apologies.

I claimed the v2 monolith-skip "did not land" and cited a slow commitPush. That was wrong on two counts, and the root cause is entirely on our side:

Our repo is not actually on shard-first v2. I checked all six of our local cache roots and every one is still monolith-format — no _index/_meta.json, just a .datastore-index.json. Our environment line said "shard-first v2 migrated 2026-07-08," but that migration evidently never completed (or got rolled back during a same-day cache purge we did chasing #1033). So readPartitionMeta() returns null and commitPush correctly takes the commitPushMonolith branch — which re-uploads the monolith as it should for a monolith repo. I never exercised commitPushShardFirst, so I had no business reporting on whether it skips the monolith. Reading the shipped .3 code, the skip is right there: commitPush → if (meta && meta.version === 2) commitPushShardFirst(...), and that path writes shards + _meta.json with no monolith PUT. It landed exactly as described.

The ~23–32s I measured was also misattributed. It was the pushed === 0 && deleted === 0 fast-path (the noop you saw in the trace) doing a pullIndex({ forceRemote: true }) + localHasAllRemoteEntries() — a full remote-index download and verify, not a monolith upload. Different operation entirely.

So: the configurable timeout, the trace points, and the compact upload are all working, and the v2 monolith-skip is present and correct — we simply hadn't migrated to v2 to benefit from it. We're running swamp datastore migrate-index on our end now. Thank you for the fix and sorry for the noise on an unmigrated repo; if anything, the "Run 'swamp datastore migrate-index'" hint in the monolith path is exactly what should have tipped me off sooner. Appreciate your patience.

Sign in to post a ripple.