Relationships
#2299 s3-datastore: documented offline behaviour is unreachable — lock acquire fails closed before pull, while a mid-run outage exits 0 with the data stranded
Opened by randybias · 9/20/2026
Summary
The documented offline behaviour of the S3 datastore cannot be reached. design/enablers/datastores.md ("Offline Behavior") says: "If S3 is unreachable, pull and push warn and continue. The command runs against the local cache. Data is pushed on the next successful connection." In practice every write command acquires a lock before pull, and lock acquisition hard-fails when the endpoint is unreachable, so the command never runs.
The result is an inconsistent pair of behaviours for the same outage, depending only on when it starts:
| endpoint goes down… | behaviour |
|---|---|
| before the command | exit 1 in <1 s: S3OperationError: S3 putObjectConditional failed key=<ns>/data/<type>/<id>/.lock — connect ECONNREFUSED. The method body never executes. |
| during the method | method completes, push fails with one [ERR] … push failed line, and the command exits 0 "succeeded". The data stays in the local cache and reaches the bucket only when that host happens to run another write command; nothing retries. |
Environment
- swamp
20260916.001346.0-sha.600af419;lock.acquire()insrc/infrastructure/persistence/datastore_sync_coordinator.tsis unchanged in latest20260918.211634.0-sha.bcaa9695 @swamp/s3-datastore2026.09.17.1; MinIO backend, namespaced
Reproduction
- Configure the S3 datastore, run one method successfully (control: "Committed N file(s) to datastore index").
- Stop the endpoint.
swamp model method run <m> execute …→ exit 1 at the lock, method not run. Read-only commands still work from cache, as documented. - Start the endpoint, start a 12 s method, stop the endpoint at +4 s → exit 0,
[ERR] push failed, data only in the local cache. A second host cannot see the result afterdatastore sync --pull; it appears only after the first host's next write command.
The same holds through swamp serve: with the endpoint down, [HOST-1] fails with method_execution_failed (S3OperationError) … putObjectConditional … .lock, while reads are served from cache.
Expected (either is fine, but one of them)
- Make the code match the doc: a documented, explicit degraded mode, or
- Make the doc match the code: state that write commands fail closed when the endpoint is unreachable, and that
SWAMP_DATASTORE=filesystem:<path>is the supported way to operate through an outage.
Separately, a run whose push failed arguably should not exit 0 without at least a machine-readable signal (--json field or distinct exit code): automation reading the exit status reports a run that no other participant can see. Related: #1770.
Related observation
Using SWAMP_DATASTORE=filesystem:<the S3 cache dir>/<namespace> during an outage works and sees prior data, but writes made that way are lost to the shared record: filesystem datastores do not call markDirty, so after recovery swamp datastore sync reports N pulled, 0 pushed and the pull overwrites the cache's version index, leaving the outage-time version as unreferenced files. Not a bug as such, but worth a sentence in the docs since pointing the override at the warm cache is the obvious thing to try.
Related: #2298 (a transient read error silently abandons the lock) — same "S3 briefly unreachable" area, different failure.
Upstream repository: https://github.com/systeminit/swamp-extensions
Environment
- Extension:
@swamp/s3-datastore@2026.09.17.1 - swamp:
20260916.001346.0-sha.600af419 - OS:
linux(x86_64) - Deno:
2.9.6 - Shell:
/bin/bash
Open
No activity in this phase yet.
Sign in to post a ripple.