Relationships
#4995 swamp datastore lock status shows a dead process's lock as held, and no doctor command checks locks
Opened by skunk-ape · 11/22/2026
What happens
When a process holding a behavior lock is killed (SIGKILL), its lock file stays on disk. Swamp handles this fine on one host: the next process that wants the lock sees the holder's pid is gone and takes it in about 3 ms. So there is no real orphan.
But swamp datastore lock status still lists the lock as held: true, with the dead pid. Nothing says the holder is gone. A user chasing a stuck run sees a "held" lock and may think it is the cause.
No doctor command checks locks either:
swamp run doctorlooks at the run tracker, run records and method run outputs. It finds the dead process's orphaned runs, and--fixsettles them, but it never looks at locks.swamp doctor datastoreshas no lock check.
The way out exists, swamp datastore lock release --force --behavior <name>, but no diagnostic points to it.
We found this while looking into a user report of a hung stagecraft publish.
Steps to reproduce
swamp repo init
swamp datastore setup filesystem --path ~/ds3 --format 3 # also happens on the default v2 datastore
# start any slow behavior method run or workflow run that holds a behavior lock
swamp workflow run <slow-workflow> --json &
sleep 4
kill -9 %1
swamp datastore lock status --json
swamp run doctor --json
swamp doctor datastores --jsonExpected vs actual
Expected: lock status says the holder is gone and the lock will be taken by the next process, or reports it as stale and suggests swamp datastore lock release --force --behavior <name>. Doctor reports the stale lock.
Actual (measured, v3 local, after SIGKILL of a publish sweep):
lock statusshowed the behavior lockheld: true,pid: <killed pid>, with nothing more.- The next run on that behavior took the lock "after 1 retries (3ms)".
run doctorreported 2 orphaned workflow runs and 1 orphaned method run, and--fixsettled them. It said nothing about locks.
Cause
- Behavior locks on a filesystem datastore:
lock statuswalks everydata/**/.lockfile and reports each one asheld: truewith the file's contents. It does not check the pid or TTL:src/libswamp/datastores/lock.ts:181-191and:359-376. The renderer prints the pid without judging it (src/presentation/renderers/datastore_lock.ts:36-52). - The lock itself gets this right.
FileLock.isStaletreats a holder as stale when its pid is gone or its last heartbeat is older than its TTL (src/infrastructure/persistence/file_lock.ts:651-660).acquireclears a stale holder and retries at once (file_lock.ts:381-400). That is the 3 ms takeover. - The v3 writer lease has the same blind spot.
v3WriterLockHandle.inspectreports every lease that is notreleased, and that includes one judgedexpiredbecause its process is gone (src/infrastructure/datastore_v3/v3_writer_lock.ts:562-567,src/domain/datastore_v3/writer_lease.ts:370-382). We did not see this in the measurement because a waiting command took the lease over and released it right after the kill. By the code, a dead writer's lease shows as held until the next writer takes it. run doctor(diagnoseLocalRuns,src/cli/commands/run.ts:266-385) reads the run tracker, workflow run records and method run outputs. It does not read locks.doctor datastoreschecks health, unmigrated data, namespace contamination, unpushed files, catalog completeness, mixed-kind names, unreadable versions, retention, and on v3 content, kept lineages, replica restore and filters. None of these looks at locks:src/cli/commands/doctor_datastores.ts:299-427.- The release command exists:
swamp datastore lock release --force --behavior <name>(src/cli/commands/datastore_lock.ts:178-200).
Holder on another machine (from the code, not measured)
- v3 writer lease: a record from another host, pid namespace or boot is judged "unknown", never "gone" (
src/domain/datastore_v3/process_liveness.ts:22-23,:75-79). Takeover then waits for the TTL plus 10 s clock skew after the last renewal (writer_lease.ts:42,:188-194): 40 s for a CLI writer (30 s TTL), 25 s for serve (15 s TTL,writer_lease.ts:48). Renewal runs every TTL/3 (v3_writer_lock.ts:704-707). - Behavior file locks: TTL 30 s, with a heartbeat every TTL/3 that rewrites
acquiredAt(file_lock.ts:49,:723-748). The pid check here does not compare hostnames, unless a lock setssameHostPidCheck, and only the v3 writer lock's legacy lock does (file_lock.ts:60-65,:655-657;v3_writer_lock.ts:802). So on a datastore directory shared between hosts, a remote holder is cleared at once if no local process has its pid number, and otherwise after 30 s. On v3, the writer lease already admits only one writing process across machines, so this only matters for v2 on a shared mount. It may deserve its own look. - Bucket datastores from extensions: behavior locks come from the provider's
createLock(src/cli/repo_context.ts:2011-2013), so their expiry is the extension's own code, which we did not check.
Suggested fix
- In
lock status, judge each holder the wayFileLock.isStaleand the writer lease already do. Mark a lock whose holder pid is gone on this host, or whose heartbeat is past its TTL, as stale (for exampleheld: true, stale: true, reason: "process-gone"). For the writer lease, report itsexpiredstate, not a plainheld. For a stale lock, say that the next locker will take it, and nameswamp datastore lock release --force --behavior <name>as the manual way out. - Add a lock check to
swamp doctor datastoresthat lists stale locks with the same wording. Optionally, haverun doctormention a stale lock held by the same dead pid as the orphaned runs it reports.
Version measured
swamp 20261120.161258 (sha 7b758e05), local filesystem datastores, format 2 and format 3, one host. Code references are to main at e52c839ba.
Open
No activity in this phase yet.
Sign in to post a ripple.