Skip to main content
← Back to list
01Issue
BugShippedSwamp CLIPublic
Assigneeszack

Relationships

#4929 v3 bucket writer intermittently refuses with a false 'another user or host runs this datastore's Litestream sidecar' when a stale presence record from another kernel or pid namespace meets its own just-launched supervisor

Opened by zack · 11/20/2026· Shipped 11/20/2026

Summary

A v3 bucket writer intermittently refuses to start replication with a false "another user or host runs this datastore's Litestream sidecar" (exit 1, datastore_v3_replication_unavailable). It happens when the checkout's presence.json was left by a sidecar that died uncleanly on another kernel, boot or pid namespace. The "other host" is the supervisor this same command just launched: the command polls between the moment the new supervisor takes supervisor.lock and the moment it rewrites presence.json, sees a held lock next to the stale foreign record, and refuses.

This is the cause of the intermittent swamp-uat failure in tests/cli/datastore/v3_datastore_test.ts, "swamp takes over the v3 writer lease of a writer on another machine only after its TTL, and a writer stopped there is fenced when it resumes", step "a writer stopped on another machine" (the run's body and sidecar never both ran:). It failed this way in Datastore UAT runs 44638384964 (s3-v3), 44402217301 (gcs-v3), 44361964892 (s3-v3) and 44218904319 (s3-v3).

Error

{
  "error": "swamp cannot replicate this v3 datastore: another user or host runs this datastore's Litestream sidecar. Nothing was changed.",
  "code": "datastore_v3_replication_unavailable"
}

Root cause (main at df967e324)

  1. A sidecar dies without cleanup on another kernel or pid namespace (in the UAT, step 1's docker kill of the "other host" container). Its presence.json stays in the checkout's shared dir (.commits.db-replication/), naming <hostname>#pid:[<ns>] and its boot id.
  2. The next writer finds supervisor.lock free and launches a supervisor (LitestreamReplicator.#findOrStart, litestream_replicator.ts ~411-445), then polls every 50ms.
  3. The supervisor takes the flock first (replication_supervisor.ts ~253, takeSupervisorLock), then awaits localProcess(), and only then writes its own presence (superviseHeld, ~275).
  4. A poll that lands in that gap runs #observeAndDecide (~496): isOwnHost → sameKernel gives false (different pid namespace, or a different boot id after a reboot), so otherOwner = true.
  5. decideSidecarAction (domain/datastore_v3/litestream_replication.ts ~243-251) returns refuse immediately for lock held + otherOwner. Unlike the !supervisorVerified branch below it, it does not wait for waitedPastDeadline, and #findOrStart does not take its own launched flag into account on this path.

A stale record from the same kernel and namespace doesn't trigger it: the dead pid fails isSupervisorFor, so the decision is wait. It needs a record whose identity is foreign. It clears itself on the next command, once the new supervisor has written its own presence.

Reproduction

Host only, no Docker (build 20261119.232644.0-sha.ac5bbfb7, ministack S3):

  1. swamp init, then swamp datastore setup extension @swamp/s3-datastore --format 3 on a bucket, then any write, so a sidecar starts.
  2. SIGKILL the supervisor and its Litestream (the pids in .swamp/_datastore_v3_copies/<hash>/.commits.db-replication/presence.json).
  3. Rewrite presence.json with a different process.bootId. This is what a reboot leaves.
  4. Run a write, e.g. swamp model create command/shell x --json.
  5. Repeat steps 2-4 under CPU load: the window is a few ms and widens under load.
Planted record CPU load Writes refused
Different boot id idle 0 / 30
Different boot id 64 busy loops on 32 cores 13 / 40
The dead supervisor's own record, untouched (control) 64 busy loops 0 / 40

The swamp-uat test itself, patched locally to print await writer.done when its poll gives up, failed 2 of 16 runs (8 in parallel) with exactly this error. In each case the writer exited about 300ms after the lease showed as held, with only swamp and the new __swamp_v3_replication_supervisor ever seen in docker top. So Litestream never started; it was not a slow restore or the 15s start timeout.

Impact

Any user whose sidecar died uncleanly on another kernel, boot or pid namespace can hit it on their next write: after a reboot, or with a dev container restarted over a bind-mounted checkout. They get exit 1 with a misleading message saying another user or host owns the sidecar, though nothing else is running. Retrying works.

Possible fix directions

  • In #findOrStart, once this command has launched a supervisor, treat lock held + otherOwner as wait until the start deadline, as the unverified-supervisor case already does. Or give decideSidecarAction a launched input.
  • Or have the supervisor take the lock and write its presence with no await in between. That narrows the window but does not close it.

A real foreign sidecar holding the lock should still be refused, at the latest once the start deadline passes.

02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 5 MOREREVIEW+ 6 MOREPR_LINKED+ 2 MORESESSION_SUMMARIZED

Shipped

11/20/2026, 3:17:33 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
zack assigned zack11/20/2026, 2:41:34 PM

Sign in to post a ripple.