Relationships
#4993 On a v3 datastore, one slow CLI command blocks every other writer in the repo, which fails after 60 s
Opened by skunk-ape · 11/22/2026
What happens
On a v3 datastore, a writing CLI command takes the datastore's writer lease when it opens the repo. It keeps the lease until the command ends. A swamp workflow run or swamp behavior run that is slow, for example one step waiting on an API that never answers, keeps every other writing process out for that whole time. This counts writers on other behaviors that have nothing to do with the slow one.
The others wait up to 60 s for the lease. If the slow command is still running then, they fail with exit 75, datastore_v3_writer_busy.
On a v2 datastore, and on v3 when commands go through swamp serve, only the slow step's own behavior lock is busy. Writes to other behaviors go through right away.
We found this while looking into a user report of a hung stagecraft publish.
Steps to reproduce
Generic version. By the code this should reproduce. We measured the stagecraft version below.
swamp repo init
swamp datastore setup filesystem --path ~/ds3 --format 3
swamp behavior create command/shell slow --json # set globalArguments.run: "sleep 120" in the printed file
swamp behavior create command/shell quick --json # set globalArguments.run: "true"
swamp behavior run slow execute &
sleep 3
time swamp behavior run quick executeMeasured version: a stagecraft repo, with the Linear tracker pointed at an endpoint that accepts requests and never answers. Each tracker request times out at 30 s.
swamp workflow run @swamp/stagecraft/publish --input tracker=linear --json & # 4 work items pending, takes 123 s
sleep 5
swamp behavior run linear round --input kind=pending --input run=probe-1 # same tracker
swamp behavior run @swamp/stagecraft/work-item advance <key> --input transition=submit ... # a work item on a different trackerExpected vs actual
Expected: the write to the unrelated behavior finishes in under a second, as it does on v2 (0.4 s measured) and on v3 through serve.
Actual, v3 without serve:
- The publish sweep ran for 123 s and held the writer lease the whole time.
- A tracker round probe failed after 60.2 s with exit 75,
datastore_v3_writer_busy: "Another swamp process is writing the v3 datastore ... A v3 datastore takes one writer at a time." - A write on a work item using a different (built-in) tracker waited 60.5 s.
- With two stalled single writes running, a built-in tracker write waited 58 s.
Cause
The lease is held for the whole command on purpose, and the docs say so. Their cost is that one slow command blocks the whole repo.
requireInitializedRepoUnlockedandrequireInitializedRepocallholdV3Writerbefore any other lock:src/cli/repo_context.ts:1604-1619andsrc/cli/repo_context.ts:1321-1334.workflow runopens the repo this way atsrc/cli/commands/workflow_run.ts:396-399, andbehavior rundoes too atsrc/cli/commands/model_method_run.ts:415-421.takeV3WriterHoldtakes the lease and waitsresolveLockTimeoutMs()for it:src/infrastructure/datastore_v3/v3_repo_context.ts:1041-1044. The default is 60 s (DEFAULT_LOCK_TIMEOUT_MS,src/domain/datastore/datastore_config.ts:106), andSWAMP_LOCK_TIMEOUT_MSchanges it.- The release is put in the command's teardown list (
src/infrastructure/datastore_v3/v3_repo_context.ts:1184-1198). It runs incloseV3Datastores(v3_repo_context.ts:2119), which the CLI calls at the end of the command (src/cli/mod.ts:2745-2749). - A busy lease becomes
V3WriterBusyError, exit 75:src/infrastructure/datastore_v3/v3_writer_lock.ts:69-84,:601-609, andsrc/domain/datastore_v3/writer_lease.ts:58-62. - The design doc states the scope: "A writing CLI command (
requireInitializedRepo,requireInitializedRepoUnlocked) joins the hold before it takes any other lock and keeps it until teardown; serve holds it from start-up to shutdown" (design/enablers/datastores.md:5885-5894). - Serve does not have the problem because it is one process. It takes the lease once for its whole life. Every request in that process shares the hold: the lease is reference-counted per process (
v3_writer_lock.ts:138,:153-178). Per-behavior locks then order the writes, as on v2. - The lease lasts 30 s for a CLI writer (
DEFAULT_LOCK_TTL_MS,src/infrastructure/persistence/file_lock.ts:49) and renews every TTL/3 (v3_writer_lock.ts:704-707). A live holder never lets it expire, so a slow command keeps it as long as it runs.
So this is a design limitation, not a bug in the lease. The whole-command hold also protects things the docs describe: the checkout's files are newer than its last commit while a writer runs, and the writer marker and recovery rely on the hold (datastores.md:5898-5906, :7904-7905). Any fix has to keep that.
Suggested fix
Choose one direction:
- Narrow the hold. Take the lease for each commit (or each unit's write batch), not for the whole command, so a step waiting on the network holds no lease. This is the larger change. It has to keep the checkout protection, the writer marker and interrupted-writer recovery, which assume one writer per checkout for the whole command.
- Route through a local holder. When a long-running local process already holds the lease, send CLI writes to it, the way
--serverdoes today, so the CLI never contends. This could be a runningswamp serve, or a small local holder started on demand.
Smaller steps that help either way:
- Document in the v3 docs that one slow command blocks all writers in the repo, not only writers on its own behaviors, and point to
swamp serveas the setup for concurrent writers. - While a command waits for the lease, say so and name the holder's command, not just its pid, so a user can tell which run is holding everyone up.
Version measured
swamp 20261120.161258 (sha 7b758e05), local filesystem datastores, format 2 and format 3. Code references are to main at e52c839ba.
Open
No activity in this phase yet.
Sign in to post a ripple.