Lab #2430: workflow resume: Ctrl-C leaves the run 'running' with nothing driving it, and it can no longer be resumed

Problem

Pressing Ctrl-C during swamp workflow resume exits the process with code 130 but leaves the run record at running. The step that was executing and its dependents stay pending, and the run tracker row stays running. Nothing is driving the run, and it cannot be resumed again: resume --run <id> and resume --from <step> --run <id> both refuse a running run.

This affects every local resume: a suspended run after approval, --from on a failed run, and the retry of a failed run added by swamp-club#2409. The retry makes it easier to hit, since every failed run now prints a swamp workflow resume <wf> --run <id> command to run next.

A fresh swamp workflow run handles the same Ctrl-C: it records the run as cancelled and exits 1.

The only way out today is swamp workflow cancel <wf> --run <id>, which moves the run to cancelled. A cancelled run cannot be retried or resumed with --from, so the steps that already succeeded are lost and the operator has to start a new run.

Cause

workflow run suppresses the datastore sync coordinator's Deno.exit(130) on SIGINT (suppressSyncExitOnSignal()), and when its abort signal fires it saves the run as cancelled before exiting. workflow resume (src/cli/commands/workflow_resume.ts) registers a shutdown handler that aborts its controller, but it does not suppress that exit, so the process exits before resume() can save anything. It also has no fallback that saves a terminal status for the run after an abort.

Expected

Ctrl-C during a resume leaves the run in a terminal state, and the tracker row agrees with it. workflow run uses cancelled. For a resume, consider whether an operator interrupt should leave the run retryable (for example failed, with the interrupted step recorded), because cancelled discards a run the operator explicitly chose to continue.

Fix approach

Give the local resume path the same SIGINT handling as workflow run: keep the process alive after SIGINT, let resume() unwind on the abort signal, save the chosen terminal status if the generator did not, and complete the tracker row before exiting non-zero. Check the remote path (resume --server) separately.

Reproduction (Linux)

Create a workflow with a step s that runs sleep 6; test -f /tmp/repro/ok and a step t that depends on s.
swamp workflow run slow: s fails. Note the run id.
touch /tmp/repro/ok, then swamp workflow resume slow --run <id> (or --from s --run <id>, or approve a suspended run and swamp workflow resume <wf>).
Press Ctrl-C while s is running. The process exits 130.
swamp workflow history get slow --json: status running, s and t pending. swamp run history --active still lists the run as running.
swamp workflow resume slow --run <id> refuses: Run <id> is not suspended or failed (status: running).

Reproduced on main at c811f8ce for the --from and post-approval resumes, and on the swamp-club#2409 branch for all three.

Related: swamp-club#2420 (a resume keeps the original run's process identity, so workflow cancel cannot stop a live resume).

A nested structural swamp still waits on its run's lock when the lock holder is not a same-host ancestor (cross-host worker, --server into a non-ancestor serve)

Deliver workflow signals through a write-once outcome record instead of writing the run record

Forward held-lock identity over --server so a loopback nested swamp does not wait on its caller's lock

A nested structural swamp on a same-host remote worker waits on its own step's lock held by swamp serve

Fail fast when nested structural commands under parallel runs wait on each other's locks

Docs: remove followUpActions from the MethodResult reference in the extension model manual

signal_test: already-settled test fails by chance when a random UUID spells the sender name

Add a wait_for_signal workflow step that pauses a run for a JSON message

run doctor through serve interrupts a live run of another serve instance when no heartbeats are recorded

Flaky test: WalSink replayed segments are delivered before events written after replay leaves one WAL segment

Remove followUpActions: no model or extension can produce them

Workflow schedule watcher and workflow edit symlink lookup ignore the managed config workflows dir

main is red: repository_dirty_coverage_test fails for UnifiedData.collectGarbage(orphaned deferred write)

isProcessGone always reports alive on Linux under the test task permissions, failing 9 tests on main

serve: relative tls cert-file/key-file and --config resolve against the working directory, not the repository

findBySpec/findByTag lose a workflow step's output after a version is deleted, pruned or rolled back

Run tracker keeps some interrupted workflow rows forever: retention waits for markSettled, which several paths never call

run doctor reads every run record on each run to rebuild the run indexes

Serve audit: two sinks with the same name share one emitter cursor, so the second never receives events

Serve audit: a durable sink write that times out is retried while the first call is still pending, so the store can hold duplicate sequences

Flaky test: ModelResolver data accessors for a renamed model return the earlier id's record from findBySpec

data query --limit returns fewer live records than exist and reports limited: false when stale catalog rows fall inside the limit

data get: help example passes --run latest, which is not a recognised run id

workflow reject leaves sibling gates waiting_approval and dependents pending in a failed, retryable run

workflow cancel cannot cancel a suspended run whose workflow file was deleted, and cancel --all silently skips it

A job alone in its level starts after the abort, and its unstarted step is recorded as a real failure instead of settledByAbort

workflow approve racing workflow cancel loses the cancel: the run returns to suspended after cancel reported cancelled

data query: does not follow data rename forwarding that data get follows

Unit tests share fixed /tmp catalog paths, so concurrent runs corrupt them and fail every later run

data query: renamed data items are not found by their old name

data query: model lookup by id and orphan recovery parity with data get

workflow approve on a run that is not suspended prints a fatal error with a stack trace

data query: no way to target a workflow's latest run

data query: binary content lacks contentEncoding/base64 parity with data get

data query: no fallback from spec name to data instance name

data query: no definitionHash or garbageCollection fields in query results

A nested structural swamp spawned by swamp serve skips every lock serve holds, including other runs' locks

data query: confirm catalog parity with data get for custom datastores

data query: single-result --json mode for scripts migrating from data get

data get --workflow silently returns an arbitrary step's data when several steps share a data name

swamp doctor workflows misses nested and .yml extension workflows that the loader reads

auth whoami says the stored API key is no longer valid when auth.json only holds an env-key identity cache

Docs: swamp skill says cancelling a parent run cancels its nested child runs; the child is left suspended

Namespace advice in the slow-lock warning is wrong for a single repo on a local datastore

Method-run records left running by a dead owner are never reaped

Flaky: usecase_sync_characterization_data_test 'data gc (serve)' asserts markDirty order that varies under parallel load

Worker dispatch rebuilds the definition with Definition.create, so a legacy-named model cannot run on a worker

A cancelled workflow exits before its steps stop, leaving their method-run records at running

Nested swamp structural command in a shell step times out on its parent's per-model lock (SWAMP_LOCK_HOLDER_PID is stripped)

Auth gate blocks nested swamp in workflow shell steps when the credential is in SWAMP_API_KEY

swamp workflow create crashes with an [FTL] ZodError stack trace on an invalid (uppercase) workflow name

swamp model create crashes with an [FTL] ZodError stack trace on an invalid (uppercase) model name

model cancel SIGTERMs a running swamp serve process when the method run belongs to a serve-run workflow

model cancel --all on a running workflow leaves its method-run records at running, or records them failed instead of cancelled

model cancel SIGKILLs the owner 2 s after SIGTERM, before the step executor's own 3 s kill grace, so the run never records its own cancellation

workflow cancel and supersede of a suspended run leave its jobs running in the cancelled record

A force-exited workflow run stays running with a dead pid: resume, recover and run doctor --fix cannot clear it

A nested workflow that suspends on manual approval forwards its suspended event into the parent run's stream

workflow cancel SIGKILLs the run 2 s after SIGTERM, cutting off always/completed cleanup jobs mid-step

Warn when datastore setup keeps managedConfig but the new datastore has no config tier

serve: a relative grants-dir resolves against the working directory, not the repository

workflow resume cleanup mode can run an always-gated teardown before a suspended job's approved work

managedConfig: extension writes update the shared lockfile from a stale cache and overwrite peers' entries

datastore setup extension overwrites the shared config tier with the repo-local copy and drops managedConfig from .swamp.yaml

serve daemon enable writes a unit that crash-loops when serve args are invalid (e.g. missing --admins)

Concurrent embedded deno extraction races: stale temp-file cleanup deletes another process's in-flight .deno.tmp file (chmod ENOENT)

Telemetry bridge drops a parent step's invocation when a nested workflow step has the same job and step names

serve daemon: unit sets SWAMP_HOME, which relocates the config dir — token/oauth daemons can't find auth.json and crash-loop

Workflow step that times out on the model lock leaves its output and run-tracker row stuck at 'running'

An extension upgrade that hits a type collision is rolled back to no version: roll back to the prior version with a commit/rollback install handle

Telemetry bridge attributes a parent run's later method invocations to its nested child run

sensitive_data_delivery_test fails where /bin/sh is bash: the final ps is exec'd and reports its own argv

build-attestation rejects every run: evaluated workflows now carry sensitiveFormat, which the provenance check treats as an uncommitted field

DuplicateTypeError names the incoming extension as the existing claimant and reports a false ghost row when it sorts first

Install extensions by stage and swap, with a journal and crash recovery

workflow resume ignores SWAMP_MAX_CONCURRENT_STEPS for job-level concurrency

Tidy up tests and helpers from the serve suspended-run cancel

Concurrent extension installs and removals can interleave in one checkout: add a pulled-extensions lock and LockfileRepository.refresh()

deno run audit prints No description available for every advisory

Add a literal() CEL function so one string can mix swamp expressions with another service template syntax

Property test for deferred bindings fails intermittently on a proto key