Skip to main content
← Back to list
01Issue
BugOpenExtensions
AssigneesNone

Relationships

#1008 datastore migrate-index aborts on hardcoded 30s S3 requestTimeout for large shards

Opened by mgreten · 7/7/2026

Summary

swamp datastore migrate-index (monolithic → shard-first) aborts on a large namespace because a single shard's putObject exceeds a hardcoded 30 s request timeout with no override. This makes the migration — which is the intended way to get the shard-first per-write benefit — unusable on exactly the namespaces that need it most.

Steps to reproduce

  1. Have an @swamp/s3-datastore 2026.07.02.1 namespace whose local index is still monolithic (.swamp/data/_catalog.db, no local _index/ dir) and large — here ~124k latest rows / ~630k all-version entries / 95 MB, with one model holding ~62k index entries in a single shard.
  2. Run swamp datastore migrate-index --json (I set SWAMP_DATASTORE_SYNC_TIMEOUT_MS=1800000 first, expecting it to cover the operation).

Observed:

[s3-sync] Migrating monolithic index to shard-first: 630739 entries → 289 shard(s)
{ "error": "S3 putObject failed timed out after 30000ms TimeoutError —
            @smithy/node-http-handler ... exceeded the configured 30000 ms requestTimeout." }

The migration rolls back cleanly (datastore stays healthy, _catalog.db intact — no corruption), but it never completes.

Root cause (from reading the bundle)

The per-request S3 timeout is a hardcoded constant DEFAULT_REQUEST_TIMEOUT_MS = 3e4 in the datastore bundle, and I couldn't find a Deno.env.get that reads it — so SWAMP_DATASTORE_SYNC_TIMEOUT_MS (which governs the overall sync) does not apply to the individual S3 request. The oversized shard is one model with ~62k entries (an adw-analytics session stream, lifetime: infinite); writing it as a single object to a MinIO backend over a tailnet exceeds 30 s.

Suggested fixes (either would unblock)

  1. Make the request timeout overridable — an env var, or a --timeout flag on migrate-index (and ideally other heavy one-off sync ops). Even honoring SWAMP_DATASTORE_SYNC_TIMEOUT_MS for the per-request timeout during migrate-index would do it.
  2. Chunk the shard upload (multipart, or split an oversized shard into sub-objects) so no single putObject has to carry 60k+ entries at once.

Why it matters

Without a completable migrate-index, a namespace stuck on the monolithic local index keeps paying a whole-index sync on every write (I measured ~20 min per uncontended write on this namespace — see swamp Lab #829 for that thread). The migration is the fix; it just can't run at this scale today.

Environment

  • @swamp/s3-datastore 2026.07.02.1
  • swamp 20260706.015952.0-sha.a3f4a37c
  • MinIO backend over tailnet, single namespace (~95 MB / 630k index entries, one ~62k-entry shard)

Happy to run a patched build against this namespace — it's a good stress case. Thanks for all the datastore work lately.

02Bog Flow
◉OPEN○TRIAGED○IN PROGRESS○SHIPPED+ 1 MOREASSIGNED

Open

7/7/2026, 2:17:38 AM

Click a lifecycle step above to view its details.

03Sludge Pulse
keeb assigned keeb7/7/2026, 4:08:36 AM
keeb unassigned keeb7/7/2026, 4:09:15 AM
Editable. Press Enter to edit.

keeb commented 7/7/2026, 4:09:43 AM

I don't have an s3 setup so this is better off left to someone else.

Sign in to post a ripple.