#3485 S3/GCS: a run refused for losing its model lock keeps its write dirty when the refresh can't reach the datastore, so a later sync pushes it

Background (read this first)

swamp is reworking its datastore layer (tracking: swamp-club#2865). Before Phase 5 (serve leader, commit feed and model leases), swamp-uat is pinning today's v2 behaviour:

• concurrent runs (swamp-uat#488);
• an unreachable remote (swamp-uat#489);
• multi-instance serve (swamp-uat#491).

swamp-uat#489 found this bug. It came with the swamp-club#3373 fix.

How v2 works:

• on S3 and GCS, repositories write into the datastore cache dir and mark files dirty, and the extension's sync service (@swamp/s3-datastore, @swamp/gcs-datastore) pushes them;
• per-model locks (acquireModelLocks, src/cli/repo_context.ts) serialise runs of one model across clones. Each is a DistributedLock (src/domain/datastore/distributed_lock.ts), backed on buckets by the extensions' S3Lock and GcsLock, and kept alive by a heartbeat with a 30 s TTL;
• since #3373, before each push assertModelLocksHeld (around line 3130 of repo_context.ts) reads each lock with inspect(). If one no longer names this run's holder, the run refuses to push, pulls the lost models so the cache holds the datastore's copy, and throws ModelLockLostError (model_lock_lost, exit 1).

The bug

A run that exits 1 saying "Nothing was pushed" can still have its write pushed later.

1. Clone A starts a mutating method run, which takes the model's lock.
2. The datastore becomes unreachable while the method body runs.
3. The run decides its lock is lost and refuses to push:

   This run lost its lock on @test/side-effect/ (now released) before pushing: the lock was force-released or taken over by another process, which may have written the same data version. Nothing was pushed, and this host's copy of the model's data was refreshed from the datastore, which may replace what this run wrote. Run the method or workflow again.

4. The refresh pull fails, because the datastore is still unreachable. It is best effort, so the failure is only logged at debug. The run's data version stays in A's cache, still marked dirty.
5. The datastore comes back. A's next swamp datastore sync pushes that version, and clone B pulls it. Both clones now hold data from a run that reported failure.

The same gap exists after a real takeover (a breakglass with lock release --force) whenever the refresh pull fails for any reason. A later push then uploads the refused version over the other run's, which is the lost write #3373 was meant to stop.

Cause

The refusal relies on the refresh pull to overwrite the refused versions in the cache. When that pull can't run, nothing removes them from the cache or from the dirty set, and the next push (a later datastore sync, a workflow's end-of-run push, serve's sync) uploads them.

A contributing cause is in the GCS extension, and will be filed separately as a swamp-club Lab issue. GcsLock.inspect() returns null for any read failure, a timeout included (readLockWithGeneration catches everything), so an unreachable bucket looks like a released lock. S3Lock returns null only when the lock object is absent, and throws otherwise. On S3, the same outage therefore fails the push as an ordinary push error: the write stays dirty and is pushed later, which is v2's normal offline behaviour.

This issue is the core side. It has to hold whatever any lock backend returns, since third-party registry locks may also report errors as "absent".

What to do

Make "Nothing was pushed" true without depending on the refresh pull. When a run refuses its push because a lock was lost:

• remove the run's own refused data versions from the local cache and from the dirty set before throwing ModelLockLostError;
• then try the refresh pull, as today, best effort. A later sync pulls whatever the datastore holds, so the cache converges either way.
• The run knows what it wrote: the push is given the data versions the work under the locks wrote (ModelLockResult push / flush(written)). Remove those, never versions this run didn't write.
• Keep the error code and exit code. Adjust the message only if it has to match what now happens: the run's write was discarded, and the model's data is refreshed from the datastore when it can be reached.
• Filesystem and v3 don't change. v2 S3 and GCS pushes that keep their lock don't change.

If tracing shows the refused versions can't be told apart from other dirty files in the cache (for example a version directory shared with an earlier run), stop and say so on this issue before widening the fix.

Tests

• On the in-memory remote (packages/testing), with failure injection:
  • lock lost while the remote is unreachable: the run's lock read returns "absent" and the refresh pull fails. The run exits with model_lock_lost, its version is gone from the cache and the dirty set, and a later sync pushes nothing of it;
  • breakglass with a failed refresh: another holder's version survives a later sync from the old holder;
  • unchanged: a holder whose lock is still its own pushes as today, and a refusal whose refresh pull succeeds behaves as today.
• Check the tests fail without the fix: confirm each fails on main.

Acceptance

• A run that exits with model_lock_lost leaves its data on no clone, ever, whether or not the datastore is reachable when it exits.
• swamp-uat#489's scenario "swamp model method run cut off from its gcs datastore mid-run ends with its data on every clone or on none, never a partial result or a hang [gcs]" passes once un-ignored, with no other swamp-uat change.

Evidence:

• swamp 20261013.200158.0-sha.35be0b9d;
• swamp-uat CI run 38473633290, job "Datastore: gcs (3/4)": AssertionError: a failed run must leave no data; actual ["15517086-…"], expected [];
• reproduced locally on macOS against fake-gcs-server.

Blocked by: nothing. Runs in parallel with Phase 4 (swamp-club#3482).