#2498 workflow resume of a suspended run crashes mid-run when the workflow was edited during the approval window

Problem

swamp workflow resume <wf> --run <id> on a suspended run walks the current workflow YAML against the stored run. If the workflow was edited during the approval window, the resume does not detect it. The mismatch is only found after work has started. The run is then left with steps and jobs stuck in running, and nothing clears them. In the moved shape, a real side effect has already happened when the resume crashes.

This was split out of swamp-club#2433, which fixes the same gap for resuming a failed run (--from and the #2409 retry). The approach for failed runs (a structure check before any change) does not carry over directly: suspended runs hold unfinished records that are legitimate, and some edits that work today have to keep working.

Reproduction

Installed swamp 20260924.141756.0-sha.5b8a2673. Job main runs prep → gate (manual_approval) → deploy → notify, all in one job. Run it; it suspends at gate with deploy and notify pending and main running. Edit the YAML, then run swamp workflow approve <wf> gate --run <id> followed by swamp workflow resume <wf> --run <id>.

1. Added step. Add lint to main, depending on gate. The resume fails with Error: Workflow resume failed: Step run not found: lint and exits 1. History: the run is failed and main is running. deploy is left running with no completedAt, although it never ran. notify is pending.
2. Removed step. Remove notify. The resume exits 0 and the run succeeded, with notify left pending. This works today and must keep working; only the leftover pending record is cosmetic.
3. Moved step. The original YAML also has job post (dependsOn main) with step announce. Move notify from main into post. deploy runs (its data artifacts are written), then the resume fails with Step run not found: notify and exits 1. History: main succeeded with notify pending. post and announce are left running.

Scripts: ~/swamp/repro/issue-2433/rerun.sh suspended-added|suspended-removed|suspended-moved on the triager's machine. They build a fresh repo, run, swap the YAML, approve and resume.

Cause

resume() (src/domain/workflows/execution_service.ts) walks every job of the current workflow whose stored record is not terminal. runJob then walks every step of the current job and looks up that step's record in the same stored job. A step with no record there throws Step run not found mid-run. A record the workflow no longer puts in that job stays pending, and the job and the run still complete as succeeded.

Design constraints, carried over from the #2433 adversarial review

• Tell removed apart from moved. A pending record whose step exists in no current job (removed) must not be refused, because the removed shape succeeds today. A record whose step the current workflow places in another job (moved) must be refused. (#2433 ADV-1)
• Narrowing a forEach at resume must keep working. resume --input can shrink a forEach collection. Iterations that were already expanded stay pending, and today the job and the run succeed. A blanket "no success while a step is pending" rule would break this. (#2433 ADV-2)
• Refuse in the resolver. Serve registers the run and charges the principal's cap before resume() runs (src/serve/resume_launcher.ts), so the check has to run in resolveResumableRun and the suspended resolver. A refusal also leaves the run suspended and awaiting resume, so the message has to name the way out: swamp workflow cancel --run <id>, or a new run that supersedes it. (#2433 ADV-4)
• recover() re-enters through this path. recover() calls resetUnknownStepsForRecovery(), which sets the run to suspended, and saves the run before it calls resume(). A check on the suspended path has to run before that mutation, or accept that recovery is refused afterwards. The fingerprint check usually prevents this, but runs started with --last-evaluated record no run plan. (#2433 ADV-11)
• Allow evaluated names. Runs started with --last-evaluated, and legacy forEach records, carry no forEachTemplate and have expression-named templates. Their unfinished records must not be refused. (#2433 ADV-3)