Relationships
#4557 @swamp/gcs-datastore (and @swamp/s3-datastore): a 16 MiB chunk put times out after a flat 30 s, so a slow serve-to-GCS link can never store it
Opened by stack72 · 11/9/2026· Shipped 11/9/2026
What happens
In team mode, serve stores each carried chunk through GcsChunkStorage.put
(datastore/gcs/extensions/datastores/_lib/gcs_storage.ts). Every attempt is
one simple upload (uploadType=media) under the client's per-request timeout,
requestTimeoutMs. That defaults to 30 s (gcs_client.ts,
DEFAULT_REQUEST_TIMEOUT_MS), and retryWithBackoff makes 3 attempts.
Chunks are up to 16 MiB. Below about 560 KB/s between serve and GCS, no attempt can finish: every one times out at 30 s and the put fails after about 92 s. Serve then fails the whole commit.
Not yet seen on the in-region dogfood. It matters for serve deployments with a slow or remote link to the bucket.
The same flat timeout covers every request, from a 16 MiB upload to a small metadata call, so raising it to suit chunks also slows failure detection everywhere else.
Found while reproducing swamp-club#4542 locally (GCS emulator, serve-to-bucket
link throttled). Serve logged GCS putObject timed out after 30000ms from
GcsChunkStorage.put ← ChunkBatch.put, and the member re-sent the batch
until the command failed. The member-side resend is bounded in swamp by #4542.
Workaround today
Set requestTimeoutMs in the datastore config (up to 600 s), or set
SWAMP_GCS_REQUEST_TIMEOUT_MS. Either one raises the timeout for every GCS
request, not only chunk puts.
Asks
- A put timeout that scales with size. A chunk put's deadline becomes the configured timeout plus the body's bytes at a floor rate (swamp uses 256 KiB/s for serve requests). Or, above some size, use a resumable upload that commits only when complete, so a slow link makes progress instead of restarting. Other requests keep the flat timeout.
- No copy of the body per put.
putcopies the chunk (bytes.slice()) so a caller's later change cannot reach the upload. Serve now stores up to 4 chunks per request at once, under a 128 MiB serve-wide budget for chunks read but not yet stored (#4542). The copy doubles that to about 256 MiB held during uploads. Swamp'sChunkBatchnever changes a buffer after handing it toput, so the copy can go, or be limited to callers that ask for it. - Check
@swamp/s3-datastorefor the same two problems. Its chunk put probably has both.s3_client.tsdefaults to a flat 30 s request timeout, andS3ChunkStorage.putcopies the body withbytes.slice()(s3_storage.ts). Apply the same fixes there if they hold. - A test: a 16 MiB put through a delayed emulator link, longer than the flat timeout but within the scaled deadline, succeeds in one attempt.
Not in scope
The member-to-serve side: request deadlines and serve's concurrent chunk stores are fixed in swamp (#4542).
Shipped
Click a lifecycle step above to view its details.
system commented 11/9/2026, 5:56:22 PM
Classified automatically when this issue was filed.
- Source: Extensions
If you feel this classification is incorrect, add a ripple to tell us so.
Sign in to post a ripple.