Skip to main content
← Back to list
01Issue
BugShippedSwamp CLIPublic
Assigneesstack72

Relationships

#4788 v2 S3/GCS: a mutating run that can't confirm its model lock at push exits 1 but keeps its data, and the next sync publishes it to every clone

Opened by stack72 · 11/16/2026· Shipped 11/16/2026

Summary

On a v2 S3/GCS datastore, a mutating swamp model method run whose push cannot confirm the run's model lock (the datastore is unreachable at push time) exits 1, but keeps its data on the host. The next swamp datastore sync (or write) then publishes that "failed" run's data to every clone.

So a command reports failure while its write lands everywhere later. Exit 1 should mean the run left no data, or the data should be reported as saved and pending (as v3 does with exit 75).

Regressed in #3773 (swamp-club#4673), swamp 26c82aa0. Before it, the same push error was swallowed and the run exited 0. Since then, swamp-uat's tests/cli/datastore/offline_test.ts ("swamp model method run cut off from its {s3,gcs} datastore mid-run ends with its data on every clone or on none…") has failed on every Datastore UAT run (since 2026-11-12, about 30 consecutive runs, s3 and gcs):

AssertionError: Values are not equal: a failed run must leave no data
-   [ "<executionId>" ]
+   []

Example run: https://github.com/swamp-club/swamp-uat/actions/runs/43810775226 (swamp 20261116.052640.0-sha.4d3ccd8c)

Steps to reproduce

  1. Two clones on one v2 S3 or GCS datastore.
  2. On clone A, start a mutating method that holds its model lock and takes a few seconds (the UAT uses a local-only method that sleeps 3s, then writes a resource).
  3. While the method body runs, make the datastore unreachable (the UAT uses docker compose pause on the emulator).
  4. The run exits 1 with datastore_push_failed: "This command's changes are saved on this host but did not reach the datastore (…). Run 'swamp datastore sync --push' to publish them."
  5. Restore the datastore and run swamp datastore sync on A, then on B.
  6. The failed run's execution is present on A and B.

Or run the UAT directly against a build at or after 26c82aa0: deno test -A --filter "/cut off from its s3 datastore mid-run/" tests/cli/datastore/offline_test.ts

Code path (at 4d3ccd8c)

  1. The method body succeeds, so runSucceeded = true (src/cli/commands/model_method_run.ts:1060).
  2. The push calls assertModelLocksHeld (src/cli/repo_context.ts:3899), which calls lock.inspect(). With the datastore unreachable, inspect() throws ("a push that cannot confirm its lock does not go ahead", line 3906).
  3. Only the lock-lost branch discards the run's versions (repo_context.ts:3915–3947, discardRefusedVersions → ModelLockLostError: "Nothing was pushed: this run's data was discarded from this host"). A lock that can't be read throws before that branch, so the versions stay dirty in the cache.
  4. model_method_run.ts:838–857 rethrows the push error, since the run succeeded. src/cli/command_root_unit.ts:130–134 wraps it with toPushFailure into DatastorePushFailedError (datastore_push_failed). exitCodeForError (src/presentation/output/error_output.ts) has no case for that code, so the run exits 1.
  5. The next sync publishes the dirty versions.

Impact

  • Automation that retries on a non-zero exit re-runs a mutating method that actually succeeded, so its side effects outside swamp run twice and there are two executions.
  • A non-zero exit no longer means "nothing happened": the run's data shows up on every clone later.
  • The three outcomes give inconsistent guidance:
    • lock lost: data discarded, "run it again"
    • lock unreadable: data kept, "sync --push"
    • read/list publish failure (datastore_publish_failed, model_method_run.ts:800): "on this host only; run the method again"
  • Open question, not verified: the push was refused because the lock couldn't be confirmed, yet the user is told to sync --push. If sync --push doesn't re-check the per-model lock, a clone that took the lock during the outage and wrote the same version number could be overwritten. That is the case ModelLockLostError's discard exists to prevent.

The message itself is accurate, so a person reading it does the right thing. The harm is to callers that act on the exit code.

Suggested direction

Make the data's fate match the exit code. Either:

  • A. Treat "lock can't be confirmed at push" like "lock lost": discard the run's versions and exit 1 with a "nothing was pushed" message. Strict, consistent with the comment at repo_context.ts:3906, and needs no UAT change. A successful run loses its data to a brief outage.
  • B. Keep the data, but report a distinct "saved, pending publish" outcome, as v3's exit 75 / V3_REPLICATION_UNCONFIRMED_CODE does, and make sure the later publish re-checks the model lock. swamp-uat's exit-75 branch would then be extended to v2.

Note: exitCodeForError defines 75 as "nothing changed or a retry is safe", so a plain switch of datastore_push_failed to 75 doesn't fit that definition on its own. B needs a deliberate "pending publish" contract.

Also align the datastore_publish_failed (read/list) wording with whichever option is chosen.

02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 5 MOREREVIEW+ 16 MOREPR_LINKED+ 2 MORESESSION_SUMMARIZED

Shipped

11/16/2026, 1:22:29 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack7211/16/2026, 8:06:16 AM

Sign in to post a ripple.