Skip to main content
← Back to list
01Issue
BugTriagedExtensionsPublic
Assigneeshammz

Relationships

#2298 s3-datastore: a transient S3 read error during a heartbeat silently abandons the lock; a second host then runs the same model concurrently

Opened by randybias · 9/20/2026

Summary

A single transient S3 read failure during one heartbeat tick makes a lock holder silently abandon its lock while its method keeps running. ~30 s later the lock is stale, another host acquires it, and two processes run the same model concurrently. Neither run prints a warning; both exit 0.

Environment

  • swamp 20260916.001346.0-sha.600af419 (also read against latest 20260918.211634.0-sha.bcaa9695 — code path unchanged)
  • @swamp/s3-datastore 2026.09.17.1 (latest at time of filing)
  • Backend: MinIO RELEASE.2025-04-22T22-12-26Z, path-style, namespaced datastore
  • Two separate Linux containers ("hostA", "hostB"), separate hostnames and separate caches, same repo, same bucket

Reproduction (≈ 100 s)

  1. hostA: swamp model method run m2 execute --input run='sleep 100; echo A-finished' (command/shell model)
  2. At +5 s stop the S3 endpoint; at +19 s start it again (a 14 s outage — long enough to cover one 10 s heartbeat tick).
  3. At +66 s, while hostA's method still has ~35 s to run, hostB: swamp model method run m2 execute --input run='echo B-ran-concurrently'

Observed

  • hostB acquired the per-model lock in 1.4 s and ran to completion, rc=0, while hostA's method was still executing.
  • hostA finished at +101 s, rc=0. Its output contains no warning, no lock message, nothing.
  • Control: the same two commands with no outage serialize correctly (hostB waits for hostA's release), so the lock works until the blip.

Expected

A transient S3 error during a heartbeat must not be treated as loss of ownership. At minimum, losing the lock while a method is running should be loud.

Cause (from the extension source, datastores/_lib/s3_lock.ts)

  • readLock() wraps getObject in try { … } catch { return null; } — every error, including a network error, becomes null.
  • extend() does const current = await this.readLock(); if (!current || current.nonce !== this.nonce) { this.held = false; this.stopHeartbeat(); return; }. So "S3 was unreachable for this one read" is indistinguishable from "someone else took my lock", and the heartbeat is stopped permanently for the rest of the run.
  • With no heartbeat, LastModified + ttlMs passes 30 s later and any other process steals the lock via the stale path in acquire().
  • startHeartbeat() also swallows failures (this.extend().catch(() => {}), "Heartbeat failure is non-fatal — lock will expire via TTL"), so nothing surfaces.

The exposed workload is long-running methods: any run longer than ~40 s that overlaps a brief endpoint blip (VIP failover, daemon restart, a dropped connection) loses mutual exclusion for the remainder of the run.

Suggested direction

  1. In extend(), distinguish "lock object read OK and nonce differs / object absent (404)" from "read failed". On a failed read, keep held, keep the heartbeat, and retry on the next tick.
  2. When ownership genuinely is lost mid-run, log at warn/error and ideally signal the running operation, rather than silently continuing.

extend() verifies the nonce with a read and then writes with an unconditional putObject. If the lock is stolen between that read and that write, the old holder's heartbeat overwrites the new holder's lock body; the new holder's next extend() then sees a foreign nonce and silently stops its own heartbeat. Mentioned only because it is the same silent-loss shape.

Upstream repository: https://github.com/systeminit/swamp-extensions

Environment

  • Extension: @swamp/s3-datastore@2026.09.17.1
  • swamp: 20260916.001346.0-sha.600af419
  • OS: linux (x86_64)
  • Deno: 2.9.6
  • Shell: /bin/bash
02Bog Flow
✓OPEN◉TRIAGED○IN PROGRESS○SHIPPED+ 1 MOREASSIGNEDCLASSIFICATION

Triaged

9/21/2026, 5:17:03 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
hammz assigned hammz9/21/2026, 5:15:16 PM

Sign in to post a ripple.