Skip to main content
← Back to list
01Issue
BugIn ProgressPlatformPublicTeam
Assigneesskunk-ape

Relationships

#4983 A hung stagecraft publish holds a tracker's datastore lock 15+ min; after the holder dies the lock stays orphaned and run doctor reports the datastore healthy with no way to release it

Opened by kneel · 11/21/2026

Summary

A @swamp/stagecraft/publish workflow holds the target tracker's per-model datastore lock for the duration of the workflow. On an Azure-blob (v3) datastore we've measured holds of 15+ min. While held, all other writes to that tracker block and reads degrade (a fetch_issue took 71s under contention). If the holder process dies mid-hold, the lock is orphaned.

swamp run doctor detects orphaned run records (reported "2 orphaned workflow runs + 1 orphaned method run whose owner is gone" during the incident) and reaps them with --fix, but reports the datastore itself healthy and provides no detection or release path for an orphaned datastore lock. The tracker stays wedged until manual intervention, with no indication of what to clear.

Observed

  • 2026-11-20: concurrent sessions on one build tracker; a publish workflow plus multiple record_evidence runs failed against it; fetch_issue took 71s; run doctor showed orphaned runs (owners gone) while reporting the datastore healthy. run doctor --fix reaped 3 orphaned workflow runs + cancelled 1 orphaned method run and unwedged the tracker, but that only reaps run records, not the datastore lock itself.
  • 2026-11-19: a hung publish on one ticket froze an unrelated sibling ticket on the same tracker for 12+ min.

Distinct from #4658

#4658 (filesystem FileLock double-reclaim, shipped) is a different failure. This is a long single-holder hold on a v3/Azure datastore lock, plus the absence of an orphaned-datastore-lock release path in run doctor.

Direction

  • Hold the tracker lock per-write or time-bounded rather than for the whole publish workflow, so a hang wedges the tracker for less time.
  • Have run doctor surface orphaned datastore locks (holder pid dead / heartbeat past TTL) as it does orphaned runs, with --fix releasing them.

Environment

swamp CLI 20261121.015048.0-sha.7129f14c, Azure-blob (v3) datastore, @swamp/stagecraft/publish.

02Bog Flow
✓OPEN✓TRIAGED◉IN PROGRESS○SHIPPED+ 1 MOREASSIGNED+ 5 MOREREVIEW+ 3 MOREVERIFICATION_STARTED

In Progress

11/21/2026, 10:36:35 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
kneel moved this from a private collective to public11/21/2026, 7:41:39 PM
skunk-ape assigned skunk-ape11/21/2026, 9:44:34 PM

Sign in to post a ripple.