Relationships
#4983 A hung stagecraft publish holds a tracker's datastore lock 15+ min; after the holder dies the lock stays orphaned and run doctor reports the datastore healthy with no way to release it
Opened by kneel · 11/21/2026
Summary
A @swamp/stagecraft/publish workflow holds the target tracker's per-model datastore lock for the duration of the workflow. On an Azure-blob (v3) datastore we've measured holds of 15+ min. While held, all other writes to that tracker block and reads degrade (a fetch_issue took 71s under contention). If the holder process dies mid-hold, the lock is orphaned.
swamp run doctor detects orphaned run records (reported "2 orphaned workflow runs + 1 orphaned method run whose owner is gone" during the incident) and reaps them with --fix, but reports the datastore itself healthy and provides no detection or release path for an orphaned datastore lock. The tracker stays wedged until manual intervention, with no indication of what to clear.
Observed
- 2026-11-20: concurrent sessions on one build tracker; a
publishworkflow plus multiplerecord_evidenceruns failed against it;fetch_issuetook 71s;run doctorshowed orphaned runs (owners gone) while reporting the datastore healthy.run doctor --fixreaped 3 orphaned workflow runs + cancelled 1 orphaned method run and unwedged the tracker, but that only reaps run records, not the datastore lock itself. - 2026-11-19: a hung
publishon one ticket froze an unrelated sibling ticket on the same tracker for 12+ min.
Distinct from #4658
#4658 (filesystem FileLock double-reclaim, shipped) is a different failure. This is a long single-holder hold on a v3/Azure datastore lock, plus the absence of an orphaned-datastore-lock release path in run doctor.
Direction
- Hold the tracker lock per-write or time-bounded rather than for the whole publish workflow, so a hang wedges the tracker for less time.
- Have
run doctorsurface orphaned datastore locks (holder pid dead / heartbeat past TTL) as it does orphaned runs, with--fixreleasing them.
Environment
swamp CLI 20261121.015048.0-sha.7129f14c, Azure-blob (v3) datastore, @swamp/stagecraft/publish.
In Progress
Click a lifecycle step above to view its details.