Skip to main content
← Back to list
01Issue
BugOpenSwamp CLIPublic
AssigneesNone

Relationships

#4994 swamp run doctor takes the v3 writer lease even without --fix, so it waits behind any writer and fails while serve runs

Opened by skunk-ape · 11/22/2026

What happens

On a v3 datastore, local swamp run doctor opens the repo as a writer, so it takes the datastore's writer lease. It does this even without --fix, when it only reports.

So:

  • While another command writes, run doctor waits for it to finish. It fails after 60 s with exit 75, datastore_v3_writer_busy.
  • While swamp serve runs on the datastore, serve holds the lease for its whole life. Local run doctor and run doctor --fix then fail every time after 60 s.

run doctor does work through serve: it accepts --server <url>, and it also picks up SWAMP_SERVE_URL, SWAMP_SERVER_URL or .swamp.yaml's serverAddress. But the busy error only says "run this command through serve" and doesn't give the flag. People reach for run doctor when something looks stuck, so a plain diagnosis should not need the lease at all.

We found this while looking into a user report of a hung stagecraft publish.

Steps to reproduce

swamp repo init
swamp datastore setup filesystem --path ~/ds3 --format 3
swamp serve &            # or any long-running writing command
swamp run doctor --json  # local route, no --server

Expected vs actual

Expected: swamp run doctor without --fix reports runs right away, even while a writer or serve holds the lease. --fix may need the lease. When the lease holder is serve, the error should tell you to add --server.

Actual (measured, local route):

  • While serve ran: run doctor and run doctor --fix each failed after 60.2 s with exit 75, datastore_v3_writer_busy, naming serve's pid. This happened in every run of the scenario: 9 of 9 attempts, including 3 with --fix.
  • With a slow local writer: run doctor waited 28 s, until the writer finished.
  • With serve stopped: run doctor took 0.7 s.

We did not measure run doctor --server.

Cause

  • Local run doctor opens the repo with requireInitializedRepoUnlocked, with or without --fix: src/cli/commands/run.ts:488-491.
  • On v3, that function always calls holdV3Writer: src/cli/repo_context.ts:1604-1619. It waits resolveLockTimeoutMs() for the lease, 60 s by default: src/infrastructure/datastore_v3/v3_repo_context.ts:1041-1044 and src/domain/datastore/datastore_config.ts:106.
  • Serve holds the lease from start-up to shutdown (design/enablers/datastores.md:5889-5894, and src/cli/commands/serve.ts:9409-9415). So the local route can never get it while serve runs.
  • run history already opens the repo read-only, so it never takes the lease: src/cli/commands/run.ts:198-203. The docs say a read-only command takes the lease only to recover a dead writer's checkout, without waiting (datastores.md:5895-5897).
  • Without --fix, diagnoseLocalRuns changes nothing in the datastore. Every reap and settle is behind fix (run.ts:287, :301-303, :322, :341-343, :358). The one write it can do is verifyIndexes (run.ts:314), which can rebuild the local run index (src/infrastructure/persistence/yaml_workflow_run_repository.ts:2162-2200).
  • The --server route exists and supports --fix: run.ts:387-420, routing precedence in src/cli/remote_run.ts:118-137, the serve handler at src/serve/connection.ts:4181-4192. The busy message is generic: src/infrastructure/datastore_v3/v3_writer_lock.ts:72-78.

Suggested fix

  1. Without --fix, open the repo read-only, as run history does. Check that the run index repair in verifyIndexes stays a local cache write and is never staged as a commit (a read-only context refuses staged changes). Otherwise skip the repair on the read-only path.
  2. With --fix, keep taking the lease. When it is busy and the holder is serve, say swamp run doctor --fix --server <url> in the error, not only "run this command through serve".
  3. Optional: when a serve address is configured and serve holds the lease, route run doctor there without asking.

Version measured

swamp 20261120.161258 (sha 7b758e05), local filesystem datastore, format 3. run doctor --server is in that build and on main. Code references are to main at e52c839ba.

02Bog Flow
◉OPEN○TRIAGED○IN PROGRESS○SHIPPED

Open

11/22/2026, 4:26:37 PM

No activity in this phase yet.

03Sludge Pulse

Sign in to post a ripple.