Relationships
#4658 A crashed FileLock holder's stale lock can be reclaimed by two waiters at once, so two runs hold one model's lock
Opened by zack · 11/11/2026· Shipped 11/11/2026
Summary
When a FileLock holder dies (SIGKILL, OOM, power loss) or stops heartbeating past its TTL, two waiters can both reclaim the stale lock and both end up holding it. Two method runs then execute on the same model at the same time, which the per-model lock exists to prevent. Present on main (4fa5f6e1e).
Cause
FileLock.clearStaleHolder (src/infrastructure/persistence/file_lock.ts) reads the lock file, decides it is stale, and then deletes whatever file is at the path:
const state = await readLockFileState(this.lockPath, this.ttlMs); // read
...
const isStale = pidDead || Date.now() - acquiredAt > existing.ttlMs;
if (!isStale) return existing;
await Deno.remove(this.lockPath); // delete whatever is there now
return "cleared"; // caller retries createNewThe read and the delete are not atomic. With a crashed holder H and two waiters W1 and W2:
| W1 | W2 | lock file | |
|---|---|---|---|
| t0 | reads: H's pid is dead, so stale | reads: H's pid is dead, so stale | H's (stale) |
| t1 | deletes it | none | |
| t2 | createNew succeeds, step starts |
W1's | |
| t3 | deletes the file, acting on its t0 read | none (W1's live lock is gone) | |
| t4 | still running | createNew succeeds, step starts |
W2's |
Nonce fencing in the heartbeat (extend) only makes W1 stop heartbeating once it sees W2's nonce. W1's step is not interrupted, so the two runs overlap for the rest of W1's step.
Reproduction
Filesystem datastore, a command/shell model, and a workflow that fans out N steps on that one model (forEach over a list of items, each step running sleep "$D"):
- Start a run that takes the model's lock with a long step, then
kill -9it once.swamp/data/<type>/<id>/.lockexists. The lock file is left behind with a dead pid. - Immediately start the fan-out with 4–5 steps of a few seconds each.
- Compare the steps'
startedAtandcompletedAttimes in the--jsonoutput. Serialized steps complete one step length apart; overlapping steps complete within milliseconds of each other.
Observed on the main binary, in one process (5 steps × 8 s, then 4 steps × 3 s): two steps overlapped in 2 of 5 trials. In one trial, s-1 and s-3 both ran 8.07 s from the same start instant and finished 4 ms apart.
Across processes (4 separate workflow runs with one step each, started together after the SIGKILL): overlap in 1 of 4 trials.
Impact
- Two method runs on one model overlap. They can act on the same external resource concurrently, and their data writes race.
- Only after a crash or a missed TTL, when two or more waiters notice the stale lock at about the same time. Parallel steps on one model make this the usual case, since every sibling step is polling.
- This hits the per-model locks and anything else built on FileLock's stale reclaim On a filesystem datastore that includes the datastore-wide
.datastore.lock(createDatastoreLock) and workflow-run claims.
Suggested direction
Make the delete apply only to the file that was judged stale. Re-reading the nonce just before Deno.remove narrows the window but does not close it. An atomic rename of the stale file to a unique name, then checking that the renamed file still carries the stale nonce, lets only one waiter win a given stale file. A waiter whose rename captured a fresh lock instead has to restore it, so this needs care and tests: a property or interleaving test that runs several waiters against one stale lock and asserts at most one holder.
The S3/GCS extension locks have their own stale-steal path, which hasn't been checked for the same race.
Shipped
Click a lifecycle step above to view its details.
Sign in to post a ripple.