Skip to main content
← Back to list
01Issue
BugShippedExtensionsPublic
Assigneesstack72

Relationships

#4557 @swamp/gcs-datastore (and @swamp/s3-datastore): a 16 MiB chunk put times out after a flat 30 s, so a slow serve-to-GCS link can never store it

Opened by stack72 · 11/9/2026· Shipped 11/9/2026

What happens

In team mode, serve stores each carried chunk through GcsChunkStorage.put (datastore/gcs/extensions/datastores/_lib/gcs_storage.ts). Every attempt is one simple upload (uploadType=media) under the client's per-request timeout, requestTimeoutMs. That defaults to 30 s (gcs_client.ts, DEFAULT_REQUEST_TIMEOUT_MS), and retryWithBackoff makes 3 attempts.

Chunks are up to 16 MiB. Below about 560 KB/s between serve and GCS, no attempt can finish: every one times out at 30 s and the put fails after about 92 s. Serve then fails the whole commit.

Not yet seen on the in-region dogfood. It matters for serve deployments with a slow or remote link to the bucket.

The same flat timeout covers every request, from a 16 MiB upload to a small metadata call, so raising it to suit chunks also slows failure detection everywhere else.

Found while reproducing swamp-club#4542 locally (GCS emulator, serve-to-bucket link throttled). Serve logged GCS putObject timed out after 30000ms from GcsChunkStorage.put ← ChunkBatch.put, and the member re-sent the batch until the command failed. The member-side resend is bounded in swamp by #4542.

Workaround today

Set requestTimeoutMs in the datastore config (up to 600 s), or set SWAMP_GCS_REQUEST_TIMEOUT_MS. Either one raises the timeout for every GCS request, not only chunk puts.

Asks

  1. A put timeout that scales with size. A chunk put's deadline becomes the configured timeout plus the body's bytes at a floor rate (swamp uses 256 KiB/s for serve requests). Or, above some size, use a resumable upload that commits only when complete, so a slow link makes progress instead of restarting. Other requests keep the flat timeout.
  2. No copy of the body per put. put copies the chunk (bytes.slice()) so a caller's later change cannot reach the upload. Serve now stores up to 4 chunks per request at once, under a 128 MiB serve-wide budget for chunks read but not yet stored (#4542). The copy doubles that to about 256 MiB held during uploads. Swamp's ChunkBatch never changes a buffer after handing it to put, so the copy can go, or be limited to callers that ask for it.
  3. Check @swamp/s3-datastore for the same two problems. Its chunk put probably has both. s3_client.ts defaults to a flat 30 s request timeout, and S3ChunkStorage.put copies the body with bytes.slice() (s3_storage.ts). Apply the same fixes there if they hold.
  4. A test: a 16 MiB put through a delayed emulator link, longer than the flat timeout but within the scaled deadline, succeeds in one attempt.

Not in scope

The member-to-serve side: request deadlines and serve's concurrent chunk stores are fixed in swamp (#4542).

02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 5 MOREREVIEW+ 7 MOREPR_MERGED+ 2 MORESESSION_SUMMARIZED

Shipped

11/9/2026, 7:31:31 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack7211/9/2026, 5:57:14 PM
Editable. Press Enter to edit.

system commented 11/9/2026, 5:56:22 PM

Classified automatically when this issue was filed.

  • Source: Extensions

If you feel this classification is incorrect, add a ripple to tell us so.

Sign in to post a ripple.