Relationships
#3096 A nested structural swamp still waits on its run's lock when the lock holder is not a same-host ancestor (cross-host worker, --server into a non-ancestor serve)
Opened by hammz · 10/6/2026· Shipped 10/6/2026
Follow-up to swamp-club#2982 and swamp-club#2983. Both fixes leave a known limit behind, and the two limits have the same cause, so this ticket covers them together.
What still fails
A nested structural swamp (for example swamp data gc) waits on a lock held for its own run until SWAMP_LOCK_TIMEOUT_MS, then fails with exit code 75, in these cases:
- Cross-host worker (left by swamp-club#2983). A step is dispatched to a remote worker on another machine that shares the datastore (for example over NFS). The orchestrator holds the step lock. The dispatch carries the lock holder (pid, hostname, nonces), but the worker applies it only when the hostname is its own.
- Local run between the step and a --server client (left by swamp-club#2982). A local swamp workflow run holds a step lock, its shell step runs a swamp command with --server, and the serve it calls runs a nested structural swamp. Serve adopts only locks it holds itself, so the local run's lock is waited on.
- --server into a different swamp process on the same datastore (left by swamp-club#2982). Same as 2, where the caller holding the lock is another serve or CLI process, possibly on another host.
- Locks held above the orchestrator (noted in the swamp-club#2983 review). A swamp workflow run holds a lock, its shell step starts a second swamp that dispatches to a same-host worker. The hand-off names only the second swamp's own locks, so the first one's lock is waited on.
Shared cause
The drain in waitForPerModelLocks skips a lock only when LockHolderMarker.lockRelation finds all of: the lock file pid is one of this process's ancestors, the lock file hostname is this host, and the lock nonce is listed for that pid in SWAMP_LOCK_HOLDER_TOKENS. Whenever the run crosses a boundary that is not a fork from the lock holder (a dispatch to a worker, a --server request), the pid and hostname tests cannot be met across hosts and cannot be trusted when a client supplies them. swamp-club#2982 says so directly in its known limit: skipping those would mean trusting pids a client supplied.
Proposed direction
Treat the handed-down nonce as the proof that a lock is held for the run that started this process, with no pid or hostname test. A nonce is a random UUID written only to the lock file and to the hand-down chain, and a lock that is released and taken again gets a new nonce, so a stale nonce matches nothing. One change to the skip rule would then cover all four cases, and the dispatch and --server paths would forward the full inherited nonce list rather than only the forwarding process's own entry.
Questions the design must settle
- Is a nonce strong enough proof when it comes from a --server client? A client with read access to the datastore can read nonces from lock files. Compare with what such a client can already do (a shell step can already set the lock variables for its own child).
- The skip is safe only because the holder keeps the lock until the nested swamp exits. Confirm that holds for every forwarding path, including a cancelled dispatch, a dropped worker connection and a detached serve run.
- The fail-fast wait-cycle detection from swamp-club#2981 builds on inheritedLockIds, which filters by ancestor pid. It needs to treat nonce-only matches the same way.
- The ancestor-other-run relation and its timeout message are keyed on the ancestor pid. Decide what a nonce-only lineage reports there.
- Mixed versions: an older nested swamp still applies the pid and hostname rule.
- design/enablers/datastores.md, Parent-Process Lock Awareness: the hostname rule and the Known limits list would be rewritten once instead of once per case.
Why one ticket
swamp-club#2982 and swamp-club#2983 each added a narrow hand-off (runAdopting for --server, lockHolder for dispatch) on top of the pid and hostname rule. Fixing the leftovers separately would change the same rule in lock_holder_marker.ts and the same design paragraph twice.
Shipped
Click a lifecycle step above to view its details.
Sign in to post a ripple.