Skip to main content
← Back to list
01Issue
BugOpenExtensionsPublic
AssigneesNone

Relationships

#2333 gcs-datastore: a failed lock read is reported as "unlocked" — #2298 in the GCS backend

Opened by hammz · 9/21/2026

Summary

#2298 is fixed in @swamp/s3-datastore: a read that fails tells you nothing about ownership, so it must not be reported as "no lock here". @swamp/gcs-datastore still has the original bug, so the same double-execution is reachable on GCS today.

Where

datastore/gcs/extensions/datastores/_lib/gcs_lock.ts:395:

  private async readLockWithGeneration(): Promise<
    { info: LockInfo; generation: string | undefined } | null
  > {
    try {
      const { data, generation } = await this.gcs.getObject(this.lockKey);
      return { info: decodeLockInfo(data), generation };
    } catch {
      return null;
    }
  }

Every failure — transport drop, 403, 500, corrupt body — collapses to null.

Impact

  • GcsLock.inspect() (gcs_lock.ts:294) reports an unreachable bucket as "nobody holds it", which reads as an invitation to proceed.
  • GcsLock.release() (gcs_lock.ts:261) cannot tell absent from unreadable.
  • The heartbeat cannot distinguish "lock genuinely taken" from "could not reach GCS" — the exact path in #2298, where two hosts ran the same model and both exited 0 with no warning.

The two backends now document contradictory contracts for the same interface: datastore/s3/.../interfaces.ts states that inspect() returns null only when the lock is genuinely unheld and throws otherwise, while datastore/gcs/.../interfaces.ts:72 still carries the old comment. Core callers written against the new S3 semantics are wrong on GCS.

Reproduction

Same shape as #2298, with GCS in place of S3: start a long-running method on hostA, interrupt the GCS endpoint for longer than one heartbeat tick, then run the same model on hostB while hostA is still executing. hostB acquires the lock and runs concurrently; neither host warns.

Fix

Port the S3 changes:

  1. readLockWithGeneration() returns null only for a genuine 404; every other error throws.
  2. The heartbeat treats an unreadable lock, a throwing write, or an unconfirmed write as inconclusive (hold and retry), not as loss.
  3. That grace is bounded by the TTL — once ttlMs passes with no confirmed write, the object is stealable by any contender on the same staleness rule, so the hold ends rather than running on indefinitely.
  4. Update datastore/gcs/.../interfaces.ts to match the S3 contract.

Reference implementation and tests: datastore/s3/extensions/datastores/_lib/s3_lock.ts and s3_lock_test.ts on the swamp-extensions branch issue-2298.

02Bog Flow
◉OPEN○TRIAGED○IN PROGRESS○SHIPPED

Open

9/21/2026, 11:33:49 PM

No activity in this phase yet.

03Sludge Pulse

Sign in to post a ripple.