Relationships
#4929 v3 bucket writer intermittently refuses with a false 'another user or host runs this datastore's Litestream sidecar' when a stale presence record from another kernel or pid namespace meets its own just-launched supervisor
Opened by zack · 11/20/2026· Shipped 11/20/2026
Summary
A v3 bucket writer intermittently refuses to start replication with a false "another user or host runs this datastore's Litestream sidecar" (exit 1, datastore_v3_replication_unavailable). It happens when the checkout's presence.json was left by a sidecar that died uncleanly on another kernel, boot or pid namespace. The "other host" is the supervisor this same command just launched: the command polls between the moment the new supervisor takes supervisor.lock and the moment it rewrites presence.json, sees a held lock next to the stale foreign record, and refuses.
This is the cause of the intermittent swamp-uat failure in tests/cli/datastore/v3_datastore_test.ts, "swamp takes over the v3 writer lease of a writer on another machine only after its TTL, and a writer stopped there is fenced when it resumes", step "a writer stopped on another machine" (the run's body and sidecar never both ran:). It failed this way in Datastore UAT runs 44638384964 (s3-v3), 44402217301 (gcs-v3), 44361964892 (s3-v3) and 44218904319 (s3-v3).
Error
{
"error": "swamp cannot replicate this v3 datastore: another user or host runs this datastore's Litestream sidecar. Nothing was changed.",
"code": "datastore_v3_replication_unavailable"
}Root cause (main at df967e324)
- A sidecar dies without cleanup on another kernel or pid namespace (in the UAT, step 1's
docker killof the "other host" container). Itspresence.jsonstays in the checkout's shared dir (.commits.db-replication/), naming<hostname>#pid:[<ns>]and its boot id. - The next writer finds
supervisor.lockfree and launches a supervisor (LitestreamReplicator.#findOrStart,litestream_replicator.ts~411-445), then polls every 50ms. - The supervisor takes the flock first (
replication_supervisor.ts~253,takeSupervisorLock), then awaitslocalProcess(), and only then writes its own presence (superviseHeld, ~275). - A poll that lands in that gap runs
#observeAndDecide(~496):isOwnHost→sameKernelgives false (different pid namespace, or a different boot id after a reboot), sootherOwner = true. decideSidecarAction(domain/datastore_v3/litestream_replication.ts~243-251) returnsrefuseimmediately for lock held +otherOwner. Unlike the!supervisorVerifiedbranch below it, it does not wait forwaitedPastDeadline, and#findOrStartdoes not take its ownlaunchedflag into account on this path.
A stale record from the same kernel and namespace doesn't trigger it: the dead pid fails isSupervisorFor, so the decision is wait. It needs a record whose identity is foreign. It clears itself on the next command, once the new supervisor has written its own presence.
Reproduction
Host only, no Docker (build 20261119.232644.0-sha.ac5bbfb7, ministack S3):
swamp init, thenswamp datastore setup extension @swamp/s3-datastore --format 3on a bucket, then any write, so a sidecar starts.- SIGKILL the supervisor and its Litestream (the pids in
.swamp/_datastore_v3_copies/<hash>/.commits.db-replication/presence.json). - Rewrite
presence.jsonwith a differentprocess.bootId. This is what a reboot leaves. - Run a write, e.g.
swamp model create command/shell x --json. - Repeat steps 2-4 under CPU load: the window is a few ms and widens under load.
| Planted record | CPU load | Writes refused |
|---|---|---|
| Different boot id | idle | 0 / 30 |
| Different boot id | 64 busy loops on 32 cores | 13 / 40 |
| The dead supervisor's own record, untouched (control) | 64 busy loops | 0 / 40 |
The swamp-uat test itself, patched locally to print await writer.done when its poll gives up, failed 2 of 16 runs (8 in parallel) with exactly this error. In each case the writer exited about 300ms after the lease showed as held, with only swamp and the new __swamp_v3_replication_supervisor ever seen in docker top. So Litestream never started; it was not a slow restore or the 15s start timeout.
Impact
Any user whose sidecar died uncleanly on another kernel, boot or pid namespace can hit it on their next write: after a reboot, or with a dev container restarted over a bind-mounted checkout. They get exit 1 with a misleading message saying another user or host owns the sidecar, though nothing else is running. Retrying works.
Possible fix directions
- In
#findOrStart, once this command has launched a supervisor, treat lock held +otherOwneraswaituntil the start deadline, as the unverified-supervisor case already does. Or givedecideSidecarActionalaunchedinput. - Or have the supervisor take the lock and write its presence with no await in between. That narrows the window but does not close it.
A real foreign sidecar holding the lock should still be refused, at the latest once the start deadline passes.
Shipped
Click a lifecycle step above to view its details.
Sign in to post a ripple.