#3373 S3/GCS: a run whose model lock was force-released still pushes, and one run's write is lost

Background (read this first)

swamp is reworking its datastore layer (tracking: swamp-club#2865). Before Phase 5 (serve leader, commit feed and model leases), swamp-uat is pinning today's v2 behaviour for:

• concurrent runs from two clones (swamp-uat#488);
• an unreachable remote (swamp-uat#489);
• multi-instance serve (swamp-uat#491).

Those tests found this bug in today's v2 datastores. v2 stays the default until Phase 7, so fix it now; don't wait for Phase 5.

How v2 works:

• every repository writes files into the checkout's .swamp/ (or the datastore cache dir for S3 and GCS) and marks them dirty;
• on S3 and GCS, a sync service from the @swamp/s3-datastore or @swamp/gcs-datastore extension pushes dirty files and pulls peers' changes;
• a filesystem datastore is a shared directory with no sync service: every checkout and serve instance reads and writes the same files;
• per-model locks (acquireModelLocks, src/cli/repo_context.ts:2773) serialise runs of one model across clones. Each lock is a DistributedLock (src/domain/datastore/distributed_lock.ts): FileLock on filesystem, and the extensions' S3Lock and GcsLock on buckets;
• swamp datastore lock release --force [--model …] is the breakglass for a stuck lock.

swamp-uat commits the scenario as ignore: true with a link to this issue. Un-ignoring it is swamp-uat's follow-up once this ships. The fix must not need any other swamp-uat change.

Rules (from AGENTS.md)

• Unit tests are in-process only: no subprocesses and no process-global mutation; use withMockedEnv.
• Integration tests in integration/ wire real components on a temp filesystem or port-0 mock servers, and never spawn the CLI.
• Use injected clocks and waitFor from @swamp-club/swamp-testing, never fixed sleeps.
• Keep the change to what this issue needs.

The bug (data loss)

On S3 and GCS:

1. clone A runs a model method and holds its model lock;
2. while it runs, someone runs swamp datastore lock release --force --model <model>;
3. clone B then runs the same model.

Both runs exit 0, but only one data version survives. The other run's acknowledged write is gone.

Elsewhere:

• a filesystem v2 datastore keeps both versions;
• v3 refuses the old holder's write (its writer is fenced).

What's known

• The lock side. The extensions' S3Lock and GcsLock heartbeats already detect the loss: the lock object carries another nonce. They stop heartbeating and print "Lock … was lost while the operation was still running". But core never asks, so A goes on to push its changes.

• Probable mechanism, to confirm first:

  • each clone allocates the next data version from its own cache;
  • A and B both write the same version key;
  • the later push overwrites the earlier one.

  On a filesystem datastore both write the one shared directory, which is why both versions survive there.

Trace the mechanism before fixing. If the overwrite happens inside the extension's sync, say so in the issue.

What to do

The holder that lost its lock must not push, and must fail. Matching v3, it exits non-zero with a clear error.

• Before pushing under a model lock, ModelLockResult.push (src/cli/repo_context.ts) checks that each lock it holds still names this process. Use the lock's inspect() and compare holder, hostname and pid with this process's own, since core doesn't learn the nonce.
  • If a lock is absent or names another process, refuse the push and throw a UserError. Give it a new code (e.g. model_lock_lost), and have the message say the lock was force-released or taken over, that nothing was pushed, and how to re-run.
• This narrows the window but doesn't close it: there's a race between the check and the push. Say so in the PR; Phase 5's model leases close it properly. If tracing shows a cheap way to stop the overwrite itself in core, such as a create-only put for version keys, propose it in the PR. Don't build it without asking.
• Filesystem and v3 don't change. Don't run the check on v3; it has its own fence. On filesystem, run it only if it changes nothing observable.

Tests

• On the in-memory remote:
  • a holder whose model lock is force-released, then taken by another process, refuses to push, fails with the new code, and leaves the other run's version intact;
  • a holder whose lock is still its own pushes as today.
• Check the test is red without the fix: confirm it fails on main.

Acceptance

• On S3 and GCS, after a breakglass, no run that exits 0 has lost its write. The old holder fails with a clear error, and the other run's version survives.
• swamp-uat#488's breakglass scenario passes once un-ignored. If it asserts "both exit 0", swamp-uat follows the new non-zero exit; say so in the PR.

Blocked by: nothing. Runs in parallel with Phase 4 (swamp-club#3357, #3359).