Skip to main content
← Back to list
01Issue
BugShippedSwamp CLIPublic
Assigneeszack

Relationships

#4658 A crashed FileLock holder's stale lock can be reclaimed by two waiters at once, so two runs hold one model's lock

Opened by zack · 11/11/2026· Shipped 11/11/2026

Summary

When a FileLock holder dies (SIGKILL, OOM, power loss) or stops heartbeating past its TTL, two waiters can both reclaim the stale lock and both end up holding it. Two method runs then execute on the same model at the same time, which the per-model lock exists to prevent. Present on main (4fa5f6e1e).

Cause

FileLock.clearStaleHolder (src/infrastructure/persistence/file_lock.ts) reads the lock file, decides it is stale, and then deletes whatever file is at the path:

const state = await readLockFileState(this.lockPath, this.ttlMs); // read
...
const isStale = pidDead || Date.now() - acquiredAt > existing.ttlMs;
if (!isStale) return existing;
await Deno.remove(this.lockPath);                                 // delete whatever is there now
return "cleared";                                                 // caller retries createNew

The read and the delete are not atomic. With a crashed holder H and two waiters W1 and W2:

W1 W2 lock file
t0 reads: H's pid is dead, so stale reads: H's pid is dead, so stale H's (stale)
t1 deletes it none
t2 createNew succeeds, step starts W1's
t3 deletes the file, acting on its t0 read none (W1's live lock is gone)
t4 still running createNew succeeds, step starts W2's

Nonce fencing in the heartbeat (extend) only makes W1 stop heartbeating once it sees W2's nonce. W1's step is not interrupted, so the two runs overlap for the rest of W1's step.

Reproduction

Filesystem datastore, a command/shell model, and a workflow that fans out N steps on that one model (forEach over a list of items, each step running sleep "$D"):

  1. Start a run that takes the model's lock with a long step, then kill -9 it once .swamp/data/<type>/<id>/.lock exists. The lock file is left behind with a dead pid.
  2. Immediately start the fan-out with 4–5 steps of a few seconds each.
  3. Compare the steps' startedAt and completedAt times in the --json output. Serialized steps complete one step length apart; overlapping steps complete within milliseconds of each other.

Observed on the main binary, in one process (5 steps × 8 s, then 4 steps × 3 s): two steps overlapped in 2 of 5 trials. In one trial, s-1 and s-3 both ran 8.07 s from the same start instant and finished 4 ms apart.

Across processes (4 separate workflow runs with one step each, started together after the SIGKILL): overlap in 1 of 4 trials.

Impact

  • Two method runs on one model overlap. They can act on the same external resource concurrently, and their data writes race.
  • Only after a crash or a missed TTL, when two or more waiters notice the stale lock at about the same time. Parallel steps on one model make this the usual case, since every sibling step is polling.
  • This hits the per-model locks and anything else built on FileLock's stale reclaim On a filesystem datastore that includes the datastore-wide .datastore.lock (createDatastoreLock) and workflow-run claims.

Suggested direction

Make the delete apply only to the file that was judged stale. Re-reading the nonce just before Deno.remove narrows the window but does not close it. An atomic rename of the stale file to a unique name, then checking that the renamed file still carries the stale nonce, lets only one waiter win a given stale file. A waiter whose rename captured a fresh lock instead has to restore it, so this needs care and tests: a property or interleaving test that runs several waiters against one stale lock and asserts at most one holder.

The S3/GCS extension locks have their own stale-steal path, which hasn't been checked for the same race.

02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 8 MOREREVIEW+ 6 MOREPR_LINKED+ 2 MORESESSION_SUMMARIZED

Shipped

11/11/2026, 7:33:51 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
zack assigned zack11/11/2026, 6:21:44 PM

Sign in to post a ripple.