Skip to main content
← Back to list
01Issue
BugOpenExtensionsPublic
AssigneesNone

Relationships

#2299 s3-datastore: documented offline behaviour is unreachable — lock acquire fails closed before pull, while a mid-run outage exits 0 with the data stranded

Opened by randybias · 9/20/2026

Summary

The documented offline behaviour of the S3 datastore cannot be reached. design/enablers/datastores.md ("Offline Behavior") says: "If S3 is unreachable, pull and push warn and continue. The command runs against the local cache. Data is pushed on the next successful connection." In practice every write command acquires a lock before pull, and lock acquisition hard-fails when the endpoint is unreachable, so the command never runs.

The result is an inconsistent pair of behaviours for the same outage, depending only on when it starts:

endpoint goes down… behaviour
before the command exit 1 in <1 s: S3OperationError: S3 putObjectConditional failed key=<ns>/data/<type>/<id>/.lock — connect ECONNREFUSED. The method body never executes.
during the method method completes, push fails with one [ERR] … push failed line, and the command exits 0 "succeeded". The data stays in the local cache and reaches the bucket only when that host happens to run another write command; nothing retries.

Environment

  • swamp 20260916.001346.0-sha.600af419; lock.acquire() in src/infrastructure/persistence/datastore_sync_coordinator.ts is unchanged in latest 20260918.211634.0-sha.bcaa9695
  • @swamp/s3-datastore 2026.09.17.1; MinIO backend, namespaced

Reproduction

  1. Configure the S3 datastore, run one method successfully (control: "Committed N file(s) to datastore index").
  2. Stop the endpoint. swamp model method run <m> execute … → exit 1 at the lock, method not run. Read-only commands still work from cache, as documented.
  3. Start the endpoint, start a 12 s method, stop the endpoint at +4 s → exit 0, [ERR] push failed, data only in the local cache. A second host cannot see the result after datastore sync --pull; it appears only after the first host's next write command.

The same holds through swamp serve: with the endpoint down, [HOST-1] fails with method_execution_failed (S3OperationError) … putObjectConditional … .lock, while reads are served from cache.

Expected (either is fine, but one of them)

  • Make the code match the doc: a documented, explicit degraded mode, or
  • Make the doc match the code: state that write commands fail closed when the endpoint is unreachable, and that SWAMP_DATASTORE=filesystem:<path> is the supported way to operate through an outage.

Separately, a run whose push failed arguably should not exit 0 without at least a machine-readable signal (--json field or distinct exit code): automation reading the exit status reports a run that no other participant can see. Related: #1770.

Using SWAMP_DATASTORE=filesystem:<the S3 cache dir>/<namespace> during an outage works and sees prior data, but writes made that way are lost to the shared record: filesystem datastores do not call markDirty, so after recovery swamp datastore sync reports N pulled, 0 pushed and the pull overwrites the cache's version index, leaving the outage-time version as unreferenced files. Not a bug as such, but worth a sentence in the docs since pointing the override at the warm cache is the obvious thing to try.

Related: #2298 (a transient read error silently abandons the lock) — same "S3 briefly unreachable" area, different failure.

Upstream repository: https://github.com/systeminit/swamp-extensions

Environment

  • Extension: @swamp/s3-datastore@2026.09.17.1
  • swamp: 20260916.001346.0-sha.600af419
  • OS: linux (x86_64)
  • Deno: 2.9.6
  • Shell: /bin/bash
02Bog Flow
◉OPEN○TRIAGED○IN PROGRESS○SHIPPED

Open

9/20/2026, 11:26:35 PM

No activity in this phase yet.

03Sludge Pulse

Sign in to post a ripple.