#3493 @swamp/gcs-datastore: GcsLock.inspect() reports a failed read as no lock, so a GCS outage looks like a lost model lock

Background

swamp's v2 S3 and GCS datastores use the extensions' S3Lock and GcsLock (a DistributedLock) for per-model locks. Since swamp-club#3373, core reads each held lock with inspect() before pushing. A lock that is absent, or that names another holder, means the run lost it, and the run refuses to push (model_lock_lost).

swamp-extensions has GitHub issues disabled, so this is tracked here. The core side of the same incident is swamp-club#3485.

The bug

GcsLock.inspect() reports any failed read as "no lock".

• readLockWithGeneration (datastore/gcs/extensions/datastores/_lib/gcs_lock.ts) catches every error from getObject and returns null: a timeout, a network failure and access denied included.
• During a GCS outage, core therefore sees the lock as released. It refuses the run's push with "This run lost its lock … (now released)", although nobody took the lock.

S3Lock gets this right: readLockEntry returns null only when isAbsent(error), and throws otherwise. On S3 the same outage fails the push as an ordinary push error, and the write is kept and pushed later, which is v2's normal offline behaviour.

Found by swamp-uat#489 ("swamp model method run cut off from its gcs datastore mid-run …", CI run 38473633290, job "Datastore: gcs (3/4)").

What to do

• In GcsLock, treat only a missing object (404 / not found) as absent, and rethrow every other error from the read, as S3Lock does. Check every caller of readLockWithGeneration / readLock:
  • inspect() and forceRelease() must throw on a failed read;
  • the heartbeat's own read keeps its "inconclusive, try again until the TTL" handling;
  • acquire()'s best-effort reads may keep returning null where a failure is harmless, as S3Lock.readLockOrNull does. Name them explicitly.
• No change to S3.

Tests

• Against fake-gcs-server, or a stub GCS client:
  • inspect() returns null for a missing lock;
  • it returns the info for a held lock;
  • it rejects on a timeout or a 5xx;
  • forceRelease() rejects on a failed read and deletes nothing.
• Check the tests fail without the fix.

Acceptance

• During a GCS outage, a run holding its model lock is not refused as model_lock_lost. Its push fails as an ordinary push error, as on S3.

Blocked by: nothing.