Relationships
#3372 datastore lock status reports held: false for model locks on S3 and GCS datastores
Opened by stack72 · 10/13/2026· Shipped 10/13/2026
Background (read this first)
swamp is reworking its datastore layer (tracking: swamp-club#2865). Before Phase 5 (serve leader, commit feed and model leases), swamp-uat is pinning today's v2 behaviour for:
- concurrent runs from two clones (swamp-uat#488);
- an unreachable remote (swamp-uat#489);
- multi-instance serve (swamp-uat#491).
Those tests found this bug in today's v2 datastores. v2 stays the default until Phase 7, so fix it now; don't wait for Phase 5.
How v2 works:
- every repository writes files into the checkout's
.swamp/(or the datastore cache dir for S3 and GCS) and marks them dirty; - on S3 and GCS, a sync service from the
@swamp/s3-datastoreor@swamp/gcs-datastoreextension pushes dirty files and pulls peers' changes; - a filesystem datastore is a shared directory with no sync service: every checkout and serve instance reads and writes the same files;
- per-model locks (
acquireModelLocks,src/cli/repo_context.ts:2773) serialise runs of one model across clones. Each lock is aDistributedLock(src/domain/datastore/distributed_lock.ts):FileLockon filesystem, and the extensions'S3LockandGcsLockon buckets; swamp datastore lock release --force [--model …]is the breakglass for a stuck lock.
swamp-uat commits the scenario as ignore: true with a link to this issue. Un-ignoring it is swamp-uat's follow-up once this ships. The fix must not need any other swamp-uat change.
Rules (from AGENTS.md)
- Unit tests are in-process only: no subprocesses and no process-global mutation; use
withMockedEnv. - Integration tests in
integration/wire real components on a temp filesystem or port-0 mock servers, and never spawn the CLI. - Use injected clocks and
waitForfrom@swamp-club/swamp-testing, never fixed sleeps. - Keep the change to what this issue needs.
The bug
On S3 and GCS datastores, swamp datastore lock status reports held: false while a model lock is held. Meanwhile, a run blocked by that same lock names the holder in its error. Users checking whether a lock is stuck before reaching for the breakglass get the wrong answer.
Cause
createDatastoreLockStatusDeps (src/libswamp/datastores/lock.ts, around line 355) sets datastorePath only for filesystem datastores and passes "" otherwise. scanModelLocks (around line 306) then walks that local directory for *.lock files. On a bucket there's nothing to walk, so no model lock is ever reported. The global lock is reported correctly, because it goes through the DistributedLock.
What to do
- For custom (S3, GCS, extension) datastores, report model locks through the datastore's own lock implementation, the way a blocked run already sees them. Enumerate the repo's models and
inspect()each model's lock key through the same factoryacquireModelLocksuses. - Keep the output identical: same
--jsonshape and same log lines. Filesystem behaviour doesn't change. - Lock keys: if per-key inspection needs a lock key builder that isn't exported, export it from the existing module rather than duplicating the key format.
- Cost: one
inspect()per model on a large repo. Keep it bounded and concurrent, and note the cost in the PR. - Other repos: if the extensions turn out to need a new method, stop and say so in the issue. That part would be filed for swamp-extensions as a swamp-club Lab issue.
Tests
- Unit/integration on the in-memory remote (
packages/testing's fake): a held model lock is reported with its holder, and a released one isn't. - Check the test is red without the fix: confirm it fails on main.
Acceptance
- On S3 and GCS,
datastore lock statusreports a held model lock (holder, pid, model), in both log and--jsonmodes. - swamp-uat#488's lock-status scenario passes once un-ignored.
Blocked by: nothing. Runs in parallel with Phase 4 (swamp-club#3357, #3359).
Merge note: swamp-club#3359 changes createDatastoreLockStatusDeps's parameter type in the same file. Whichever lands second rebases.
Shipped
Click a lifecycle step above to view its details.
Sign in to post a ripple.