#2298 s3-datastore: a transient S3 read error during a heartbeat silently abandons the lock; a second host then runs the same model concurrently

Summary

A single transient S3 read failure during one heartbeat tick makes a lock holder silently abandon its lock while its method keeps running. ~30 s later the lock is stale, another host acquires it, and two processes run the same model concurrently. Neither run prints a warning; both exit 0.

Environment

• swamp 20260916.001346.0-sha.600af419 (also read against latest 20260918.211634.0-sha.bcaa9695 — code path unchanged)
• @swamp/s3-datastore 2026.09.17.1 (latest at time of filing)
• Backend: MinIO RELEASE.2025-04-22T22-12-26Z, path-style, namespaced datastore
• Two separate Linux containers ("hostA", "hostB"), separate hostnames and separate caches, same repo, same bucket

Reproduction (≈ 100 s)

1. hostA: swamp model method run m2 execute --input run='sleep 100; echo A-finished' (command/shell model)
2. At +5 s stop the S3 endpoint; at +19 s start it again (a 14 s outage — long enough to cover one 10 s heartbeat tick).
3. At +66 s, while hostA's method still has ~35 s to run, hostB: swamp model method run m2 execute --input run='echo B-ran-concurrently'

Observed

• hostB acquired the per-model lock in 1.4 s and ran to completion, rc=0, while hostA's method was still executing.
• hostA finished at +101 s, rc=0. Its output contains no warning, no lock message, nothing.
• Control: the same two commands with no outage serialize correctly (hostB waits for hostA's release), so the lock works until the blip.

Expected

A transient S3 error during a heartbeat must not be treated as loss of ownership. At minimum, losing the lock while a method is running should be loud.

Cause (from the extension source, datastores/_lib/s3_lock.ts)

• readLock() wraps getObject in try { … } catch { return null; } — every error, including a network error, becomes null.
• extend() does const current = await this.readLock(); if (!current || current.nonce !== this.nonce) { this.held = false; this.stopHeartbeat(); return; }. So "S3 was unreachable for this one read" is indistinguishable from "someone else took my lock", and the heartbeat is stopped permanently for the rest of the run.
• With no heartbeat, LastModified + ttlMs passes 30 s later and any other process steals the lock via the stale path in acquire().
• startHeartbeat() also swallows failures (this.extend().catch(() => {}), "Heartbeat failure is non-fatal — lock will expire via TTL"), so nothing surfaces.

The exposed workload is long-running methods: any run longer than ~40 s that overlaps a brief endpoint blip (VIP failover, daemon restart, a dropped connection) loses mutual exclusion for the remainder of the run.

Suggested direction

1. In extend(), distinguish "lock object read OK and nonce differs / object absent (404)" from "read failed". On a failed read, keep held, keep the heartbeat, and retry on the next tick.
2. When ownership genuinely is lost mid-run, log at warn/error and ideally signal the running operation, rather than silently continuing.

Related, read in source but NOT reproduced

extend() verifies the nonce with a read and then writes with an unconditional putObject. If the lock is stolen between that read and that write, the old holder's heartbeat overwrites the new holder's lock body; the new holder's next extend() then sees a foreign nonce and silently stops its own heartbeat. Mentioned only because it is the same silent-loss shape.

Upstream repository: https://github.com/systeminit/swamp-extensions

Environment

• Extension: @swamp/s3-datastore@2026.09.17.1
• swamp: 20260916.001346.0-sha.600af419
• OS: linux (x86_64)
• Deno: 2.9.6
• Shell: /bin/bash