Skip to main content
← Back to list
01Issue
BugShippedSwamp CLIPublic
Assigneeshammz

Relationships

#2919 workflow approve racing workflow cancel loses the cancel: the run returns to suspended after cancel reported cancelled

Opened by hammz · 10/1/2026· Shipped 10/5/2026

Description

swamp workflow approve and swamp workflow cancel can both run against the same suspended run at once, and the cancel can be lost. Both commands exit 0, and cancel prints "status": "cancelled". The run record then ends up suspended again, with the gate approved, because approve writes back the copy of the run it read before the cancel was saved.

Steps to reproduce

Use a workflow whose job is a gate and then a step:

jobs:
  - name: main
    steps:
      - name: gate
        task: {type: manual_approval, prompt: "go?"}
      - name: post
        task: {type: model_method, modelIdOrName: sh, methodName: execute, inputs: {run: "echo post"}}
        dependsOn: [{step: gate, condition: {type: succeeded}}]
  1. swamp workflow run gated2 --no-supersede --json. The run suspends at gate. Call its id $R.
  2. Start approve and cancel together:
    swamp workflow approve gated2 gate --run $R --json &
    swamp workflow cancel  gated2 --run $R --json &
    wait
  3. swamp workflow history get $R --json.

The race is timing dependent. In loops of 20, the cancel was lost 3 and 8 times with release 20261001.190402.0-sha.543aad7e, and 1 and 4 times with the swamp-club#2895 branch (8ae02e19).

Actual

  • approve prints {"approved":true,"allGatesDecided":true,…} and exits 0.
  • cancel prints {"previousStatus":"suspended","status":"cancelled","reason":"Cancelled by user"} and exits 0.
  • The stored run is suspended, with main=running, main/gate=succeeded and main/post=pending. cancel_reason is not recorded.

The user has been told the run is cancelled, but it is still resumable, and workflow resume would run post.

Expected

The two commands serialize on the run, so one of them wins and the other sees the result:

  • If cancel wins, approve is refused with not suspended (status: cancelled).
  • If approve wins, cancel then cancels and settles the approved run: gate=succeeded, post failed cancelled.

A command must never report success for a write that is then overwritten.

Cause (from reading the code)

Both commands open the repo with requireInitializedRepoUnlocked: src/cli/commands/workflow_approve.ts:138 and src/cli/commands/workflow_cancel.ts:495. Each then does a read-modify-write of workflow-runs/<wf>/workflow-run-<id>.yaml with no lock and no version check. The same pattern probably affects reject against cancel, and approve against a supersede. Those were not tested.

Environment: Linux x86_64, filesystem datastore. Found while end-to-end validating swamp-club#2895. It is not a regression from that change: the pre-change binary loses more cancels.

02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 5 MOREREVIEW+ 23 MOREPR_MERGED+ 2 MORESESSION_SUMMARIZED

Shipped

10/5/2026, 5:25:20 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
hammz assigned hammz10/5/2026, 3:44:01 PM

Sign in to post a ripple.