Lab #2919: workflow approve racing workflow cancel loses the cancel: the run returns to suspended after cancel reported cancelled

Description

swamp workflow approve and swamp workflow cancel can both run against the same suspended run at once, and the cancel can be lost. Both commands exit 0, and cancel prints "status": "cancelled". The run record then ends up suspended again, with the gate approved, because approve writes back the copy of the run it read before the cancel was saved.

Steps to reproduce

Use a workflow whose job is a gate and then a step:

jobs:
  - name: main
    steps:
      - name: gate
        task: {type: manual_approval, prompt: "go?"}
      - name: post
        task: {type: model_method, modelIdOrName: sh, methodName: execute, inputs: {run: "echo post"}}
        dependsOn: [{step: gate, condition: {type: succeeded}}]

swamp workflow run gated2 --no-supersede --json. The run suspends at gate. Call its id $R.

Start approve and cancel together:

swamp workflow approve gated2 gate --run $R --json &
swamp workflow cancel  gated2 --run $R --json &
wait

swamp workflow history get $R --json.

The race is timing dependent. In loops of 20, the cancel was lost 3 and 8 times with release 20261001.190402.0-sha.543aad7e, and 1 and 4 times with the swamp-club#2895 branch (8ae02e19).

Actual

approve prints {"approved":true,"allGatesDecided":true,…} and exits 0.
cancel prints {"previousStatus":"suspended","status":"cancelled","reason":"Cancelled by user"} and exits 0.
The stored run is suspended, with main=running, main/gate=succeeded and main/post=pending. cancel_reason is not recorded.

The user has been told the run is cancelled, but it is still resumable, and workflow resume would run post.

Expected

The two commands serialize on the run, so one of them wins and the other sees the result:

If cancel wins, approve is refused with not suspended (status: cancelled).
If approve wins, cancel then cancels and settles the approved run: gate=succeeded, post failed cancelled.

A command must never report success for a write that is then overwritten.

Cause (from reading the code)

Both commands open the repo with requireInitializedRepoUnlocked: src/cli/commands/workflow_approve.ts:138 and src/cli/commands/workflow_cancel.ts:495. Each then does a read-modify-write of workflow-runs/<wf>/workflow-run-<id>.yaml with no lock and no version check. The same pattern probably affects reject against cancel, and approve against a supersede. Those were not tested.

Environment: Linux x86_64, filesystem datastore. Found while end-to-end validating swamp-club#2895. It is not a regression from that change: the pre-change binary loses more cancels.

A nested structural swamp still waits on its run's lock when the lock holder is not a same-host ancestor (cross-host worker, --server into a non-ancestor serve)

Deliver workflow signals through a write-once outcome record instead of writing the run record

Forward held-lock identity over --server so a loopback nested swamp does not wait on its caller's lock

A nested structural swamp on a same-host remote worker waits on its own step's lock held by swamp serve

Fail fast when nested structural commands under parallel runs wait on each other's locks

Docs: remove followUpActions from the MethodResult reference in the extension model manual

signal_test: already-settled test fails by chance when a random UUID spells the sender name

Add a wait_for_signal workflow step that pauses a run for a JSON message

run doctor through serve interrupts a live run of another serve instance when no heartbeats are recorded

Flaky test: WalSink replayed segments are delivered before events written after replay leaves one WAL segment

Remove followUpActions: no model or extension can produce them

Workflow schedule watcher and workflow edit symlink lookup ignore the managed config workflows dir

main is red: repository_dirty_coverage_test fails for UnifiedData.collectGarbage(orphaned deferred write)

isProcessGone always reports alive on Linux under the test task permissions, failing 9 tests on main

serve: relative tls cert-file/key-file and --config resolve against the working directory, not the repository

findBySpec/findByTag lose a workflow step's output after a version is deleted, pruned or rolled back

Run tracker keeps some interrupted workflow rows forever: retention waits for markSettled, which several paths never call

run doctor reads every run record on each run to rebuild the run indexes

Serve audit: two sinks with the same name share one emitter cursor, so the second never receives events

Serve audit: a durable sink write that times out is retried while the first call is still pending, so the store can hold duplicate sequences

Flaky test: ModelResolver data accessors for a renamed model return the earlier id's record from findBySpec

data query --limit returns fewer live records than exist and reports limited: false when stale catalog rows fall inside the limit

data get: help example passes --run latest, which is not a recognised run id

workflow reject leaves sibling gates waiting_approval and dependents pending in a failed, retryable run

workflow cancel cannot cancel a suspended run whose workflow file was deleted, and cancel --all silently skips it

A job alone in its level starts after the abort, and its unstarted step is recorded as a real failure instead of settledByAbort

workflow approve racing workflow cancel loses the cancel: the run returns to suspended after cancel reported cancelled

data query: does not follow data rename forwarding that data get follows

Unit tests share fixed /tmp catalog paths, so concurrent runs corrupt them and fail every later run

data query: renamed data items are not found by their old name

data query: model lookup by id and orphan recovery parity with data get

workflow approve on a run that is not suspended prints a fatal error with a stack trace

data query: no way to target a workflow's latest run

data query: binary content lacks contentEncoding/base64 parity with data get

data query: no fallback from spec name to data instance name

data query: no definitionHash or garbageCollection fields in query results

A nested structural swamp spawned by swamp serve skips every lock serve holds, including other runs' locks

data query: confirm catalog parity with data get for custom datastores

data query: single-result --json mode for scripts migrating from data get

data get --workflow silently returns an arbitrary step's data when several steps share a data name

swamp doctor workflows misses nested and .yml extension workflows that the loader reads

auth whoami says the stored API key is no longer valid when auth.json only holds an env-key identity cache

Docs: swamp skill says cancelling a parent run cancels its nested child runs; the child is left suspended

Namespace advice in the slow-lock warning is wrong for a single repo on a local datastore

Method-run records left running by a dead owner are never reaped

Flaky: usecase_sync_characterization_data_test 'data gc (serve)' asserts markDirty order that varies under parallel load

Worker dispatch rebuilds the definition with Definition.create, so a legacy-named model cannot run on a worker

A cancelled workflow exits before its steps stop, leaving their method-run records at running

Nested swamp structural command in a shell step times out on its parent's per-model lock (SWAMP_LOCK_HOLDER_PID is stripped)

Auth gate blocks nested swamp in workflow shell steps when the credential is in SWAMP_API_KEY

swamp workflow create crashes with an [FTL] ZodError stack trace on an invalid (uppercase) workflow name

Relationships

#2919 workflow approve racing workflow cancel loses the cancel: the run returns to suspended after cancel reported cancelled

Description

Steps to reproduce

Actual

Expected

Cause (from reading the code)

Shipped