Relationships
#4994 swamp run doctor takes the v3 writer lease even without --fix, so it waits behind any writer and fails while serve runs
Opened by skunk-ape · 11/22/2026
What happens
On a v3 datastore, local swamp run doctor opens the repo as a writer, so it takes the datastore's writer lease. It does this even without --fix, when it only reports.
So:
- While another command writes,
run doctorwaits for it to finish. It fails after 60 s with exit 75,datastore_v3_writer_busy. - While
swamp serveruns on the datastore, serve holds the lease for its whole life. Localrun doctorandrun doctor --fixthen fail every time after 60 s.
run doctor does work through serve: it accepts --server <url>, and it also picks up SWAMP_SERVE_URL, SWAMP_SERVER_URL or .swamp.yaml's serverAddress. But the busy error only says "run this command through serve" and doesn't give the flag. People reach for run doctor when something looks stuck, so a plain diagnosis should not need the lease at all.
We found this while looking into a user report of a hung stagecraft publish.
Steps to reproduce
swamp repo init
swamp datastore setup filesystem --path ~/ds3 --format 3
swamp serve & # or any long-running writing command
swamp run doctor --json # local route, no --serverExpected vs actual
Expected: swamp run doctor without --fix reports runs right away, even while a writer or serve holds the lease. --fix may need the lease. When the lease holder is serve, the error should tell you to add --server.
Actual (measured, local route):
- While serve ran:
run doctorandrun doctor --fixeach failed after 60.2 s with exit 75,datastore_v3_writer_busy, naming serve's pid. This happened in every run of the scenario: 9 of 9 attempts, including 3 with--fix. - With a slow local writer:
run doctorwaited 28 s, until the writer finished. - With serve stopped:
run doctortook 0.7 s.
We did not measure run doctor --server.
Cause
- Local
run doctoropens the repo withrequireInitializedRepoUnlocked, with or without--fix:src/cli/commands/run.ts:488-491. - On v3, that function always calls
holdV3Writer:src/cli/repo_context.ts:1604-1619. It waitsresolveLockTimeoutMs()for the lease, 60 s by default:src/infrastructure/datastore_v3/v3_repo_context.ts:1041-1044andsrc/domain/datastore/datastore_config.ts:106. - Serve holds the lease from start-up to shutdown (
design/enablers/datastores.md:5889-5894, andsrc/cli/commands/serve.ts:9409-9415). So the local route can never get it while serve runs. run historyalready opens the repo read-only, so it never takes the lease:src/cli/commands/run.ts:198-203. The docs say a read-only command takes the lease only to recover a dead writer's checkout, without waiting (datastores.md:5895-5897).- Without
--fix,diagnoseLocalRunschanges nothing in the datastore. Every reap and settle is behindfix(run.ts:287,:301-303,:322,:341-343,:358). The one write it can do isverifyIndexes(run.ts:314), which can rebuild the local run index (src/infrastructure/persistence/yaml_workflow_run_repository.ts:2162-2200). - The
--serverroute exists and supports--fix:run.ts:387-420, routing precedence insrc/cli/remote_run.ts:118-137, the serve handler atsrc/serve/connection.ts:4181-4192. The busy message is generic:src/infrastructure/datastore_v3/v3_writer_lock.ts:72-78.
Suggested fix
- Without
--fix, open the repo read-only, asrun historydoes. Check that the run index repair inverifyIndexesstays a local cache write and is never staged as a commit (a read-only context refuses staged changes). Otherwise skip the repair on the read-only path. - With
--fix, keep taking the lease. When it is busy and the holder is serve, sayswamp run doctor --fix --server <url>in the error, not only "run this command through serve". - Optional: when a serve address is configured and serve holds the lease, route
run doctorthere without asking.
Version measured
swamp 20261120.161258 (sha 7b758e05), local filesystem datastore, format 3. run doctor --server is in that build and on main. Code references are to main at e52c839ba.
Open
No activity in this phase yet.
Sign in to post a ripple.