Relationships
#2333 gcs-datastore: a failed lock read is reported as "unlocked" — #2298 in the GCS backend
Opened by hammz · 9/21/2026
Summary
#2298 is fixed in @swamp/s3-datastore: a read that fails tells you nothing
about ownership, so it must not be reported as "no lock here".
@swamp/gcs-datastore still has the original bug, so the same
double-execution is reachable on GCS today.
Where
datastore/gcs/extensions/datastores/_lib/gcs_lock.ts:395:
private async readLockWithGeneration(): Promise<
{ info: LockInfo; generation: string | undefined } | null
> {
try {
const { data, generation } = await this.gcs.getObject(this.lockKey);
return { info: decodeLockInfo(data), generation };
} catch {
return null;
}
}Every failure — transport drop, 403, 500, corrupt body — collapses to null.
Impact
GcsLock.inspect()(gcs_lock.ts:294) reports an unreachable bucket as "nobody holds it", which reads as an invitation to proceed.GcsLock.release()(gcs_lock.ts:261) cannot tell absent from unreadable.- The heartbeat cannot distinguish "lock genuinely taken" from "could not reach GCS" — the exact path in #2298, where two hosts ran the same model and both exited 0 with no warning.
The two backends now document contradictory contracts for the same
interface: datastore/s3/.../interfaces.ts states that inspect() returns
null only when the lock is genuinely unheld and throws otherwise, while
datastore/gcs/.../interfaces.ts:72 still carries the old comment. Core
callers written against the new S3 semantics are wrong on GCS.
Reproduction
Same shape as #2298, with GCS in place of S3: start a long-running method on hostA, interrupt the GCS endpoint for longer than one heartbeat tick, then run the same model on hostB while hostA is still executing. hostB acquires the lock and runs concurrently; neither host warns.
Fix
Port the S3 changes:
readLockWithGeneration()returnsnullonly for a genuine 404; every other error throws.- The heartbeat treats an unreadable lock, a throwing write, or an unconfirmed write as inconclusive (hold and retry), not as loss.
- That grace is bounded by the TTL — once
ttlMspasses with no confirmed write, the object is stealable by any contender on the same staleness rule, so the hold ends rather than running on indefinitely. - Update
datastore/gcs/.../interfaces.tsto match the S3 contract.
Reference implementation and tests:
datastore/s3/extensions/datastores/_lib/s3_lock.ts and s3_lock_test.ts
on the swamp-extensions branch issue-2298.
Open
No activity in this phase yet.
Sign in to post a ripple.