Skip to main content
← Back to list
01Issue
BugClosedSwamp CLIPublic
AssigneesNone

Relationships

#3763 v3 on a bucket: after a commit fails because the bucket is unreachable, the closing sync still waits its full 60 s before the command exits

Opened by stack72 · 10/22/2026

Correction: this was first filed as an unreleased regression (75 s on main against 16 s on the release). It is not one. Both binaries take about 75 s when the Litestream sidecar is alive, and about 16 s when the sidecar is dead when the command ends. The 16 s release run happened right after a sidecar had died. bfde0792..main does not touch the code involved.

What happens

On a v3 S3 datastore with the bucket unreachable, a write whose commit needs the bucket (an output large enough to be stored as chunks) fails its commit on the chunk check after about 10 s:

S3 headObject failed key=<prefix>/_blobs/_default/sha256/<hash> AggregateError

The command then waits about 60 s more before it exits 1. Teardown's closing Litestream sync (closeV3Datastores, DEFAULT_SYNC_DEADLINE_MS = 60_000 in src/infrastructure/datastore_v3/litestream/v3_replication.ts) runs on every outcome. A live sidecar that cannot reach the bucket uses up the whole deadline, even though the command already failed because the bucket is unreachable. The sync's own error is then hidden behind the first one.

Timeline from OTel spans (t=0 is CLI start):

  • 0.2–1.7 s: the method runs.
  • 1.8–9.7 s: chunk headObject retries fail, and the commit fails.
  • 9.7–69.8 s: the closing sync waits out its 60 s deadline.
  • 69.8–74.6 s: dropped-work recovery and the lease release fail.
  • About 76 s: exit 1.

Expected

Once a commit has failed because the bucket cannot be reached, the closing sync should not wait a full minute for a bucket the command already knows is down. Serve already uses a short deadline for this (SERVE_REQUEST_SYNC_DEADLINE_MS, 10 s). Commits kept in commits.db replicate with the next write either way.

Fixed together with swamp-club#3759, which introduces the error that names this case.

02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS◉CLOSEDLINKED+ 1 MOREUNLINKED

Closed

10/22/2026, 2:36:38 AM

No activity in this phase yet.

03Sludge Pulse
stack72 linked sibling of #375910/22/2026, 1:58:36 AM
stack72 removed sibling of #375910/22/2026, 2:36:35 AM
Editable. Press Enter to edit.

stack72 commented 10/22/2026, 2:36:38 AM

Closing: this was filed in error. It is not a regression, and the fix (a 10 s closing-sync deadline after a commit fails because the bucket is unreachable) ships as part of swamp-club#3759.

Sign in to post a ripple.