Relationships
#3763 v3 on a bucket: after a commit fails because the bucket is unreachable, the closing sync still waits its full 60 s before the command exits
Opened by stack72 · 10/22/2026
Correction: this was first filed as an unreleased regression (75 s on main against 16 s on the release). It is not one. Both binaries take about 75 s when the Litestream sidecar is alive, and about 16 s when the sidecar is dead when the command ends. The 16 s release run happened right after a sidecar had died. bfde0792..main does not touch the code involved.
What happens
On a v3 S3 datastore with the bucket unreachable, a write whose commit needs the bucket (an output large enough to be stored as chunks) fails its commit on the chunk check after about 10 s:
S3 headObject failed key=<prefix>/_blobs/_default/sha256/<hash> AggregateErrorThe command then waits about 60 s more before it exits 1. Teardown's closing Litestream sync (closeV3Datastores, DEFAULT_SYNC_DEADLINE_MS = 60_000 in src/infrastructure/datastore_v3/litestream/v3_replication.ts) runs on every outcome. A live sidecar that cannot reach the bucket uses up the whole deadline, even though the command already failed because the bucket is unreachable. The sync's own error is then hidden behind the first one.
Timeline from OTel spans (t=0 is CLI start):
- 0.2–1.7 s: the method runs.
- 1.8–9.7 s: chunk
headObjectretries fail, and the commit fails. - 9.7–69.8 s: the closing sync waits out its 60 s deadline.
- 69.8–74.6 s: dropped-work recovery and the lease release fail.
- About 76 s: exit 1.
Expected
Once a commit has failed because the bucket cannot be reached, the closing sync should not wait a full minute for a bucket the command already knows is down. Serve already uses a short deadline for this (SERVE_REQUEST_SYNC_DEADLINE_MS, 10 s). Commits kept in commits.db replicate with the next write either way.
Fixed together with swamp-club#3759, which introduces the error that names this case.
Closed
No activity in this phase yet.