#3587 v3: a crashed workflow run and its completed steps' data are deleted by recovery before workflow recover or serve's reaper can see them (bug hunt group 6)

Background

swamp is reworking its datastore layer (tracking: swamp-club#2865). On a v3 datastore a workflow run's root commits only when the run ends; nothing checkpoints per step. When a writer dies without tearing down (SIGKILL, OOM, a crash), the next command's interrupted-writer recovery (swamp-club#3299) replays the commit log and removes every uncommitted file the dead writer left. design/enablers/datastores.md (~:4098, "Recovery removes, it does not commit") documents that, and it's right for ordinary uncommitted writes.

For a workflow run it contradicts what the rest of swamp promises:

• design/primitives/workflows.md: a crash mid-level keeps that level's completed steps; workflow recover settles a dead owner's running record;
• design/enablers/run-tracker.md: workflow recover and run doctor --fix settle a dead owner's run;
• design/primitives/serve.md:176: "Runs survive a process restart" (filesystem datastores).

v2 keeps the run.

The bug (V3-20, S2)

A workflow run that is SIGKILLed or crashes mid-run (CLI or serve) is deleted by v3 recovery before workflow recover or serve's reaper can see it, and the outputs of its completed steps are deleted too.

Reproduced (a bug hunt on 2026-10-16, verified on main d3b76490c, unchanged by swamp-club#3564). A child process runs workflow s1 → s2 through executeWorkflowWithLocks: s1 writes resource s1out, s2 hangs. The parent SIGKILLs the child, reopens the repo and lists the workflow's runs.

• v2: the run is there as running, with s1: succeeded and s2: running; workflow recover finds and settles it.
• v3: findAllByWorkflowId returns []. The run YAML and s1's data are gone; only the run's .log remains. workflow recover exits 1: "No interrupted runs found".

Why: the next context open runs recoverInterruptedWriter → the read model's recover / #sweepUnowned (sqlite_read_model.ts), which removes the unowned run YAML and step data. That happens before settleDeadOwnerRun (workflow_recover.ts) or reapOrphanedWorkflowRuns (serve) run, so they have nothing to settle.

Trigger: OOM, kill -9, a CI timeout, a host crash, or a serve crash during a multi-step workflow on a v3 datastore. Everyday events, and the user loses both the run's history and the completed steps' outputs.

Decision (made): keep crashed runs recoverable, as v2 does

The workflows, run-tracker and serve docs are right; datastores.md's rule needs an exception for workflow runs. Dropping crashed runs on v3 would be a behaviour regression from v2 that users rely on (workflow recover, serve's reaper, run doctor --fix).

What to do

1. Recovery keeps a dead owner's run record and marks it interrupted instead of removing it. It commits it (status interrupted, or running for the reaper to settle; say which, and match what v2's recover and reaper expect) under the recovering writer's lease, so every checkout sees the same record. Removing it from one checkout only would make checkouts diverge.
2. Completed steps' data survives. Options, say which:
   • (Recommended) checkpoint the run's root after each completed step (or level), so its data and the run's progress are committed as they happen. root.checkpoint() exists (swamp-club#3053). Measure the per-step cost against the uat#496 baseline: a workflow with many short steps pays one commit per step.
   • or have recovery commit the dead run's completed steps' data from the checkout files, identified through the run record. This is harder to make exact: it must not commit a half-written step.
3. The sweep must not remove what step 1 or 2 keeps. Make #sweepUnowned treat a dead owner's run record and checkpointed data as owned. Coordinate with the recovery-sweep scope issue (bug hunt group 8, not filed yet), which changes the same sweep.
4. workflow recover, run doctor --fix and serve's reapOrphanedWorkflowRuns then find and settle the run on v3 as on v2.
5. Docs, in this PR: add the workflow-run exception to datastores.md's "Recovery removes, it does not commit", and say how a crashed run is kept. Check serve.md:176's "Runs survive a process restart" holds for filesystem v3, and qualify it if not.

Tests

Each must fail on main first.

• Integration (child process, SIGKILL): a workflow s1 → s2 on filesystem v3, killed during s2. After reopening:
  • the run is listed, as running or interrupted;
  • s1's data is readable;
  • workflow recover settles it;
  • a second checkout sees the same run and data.
• Serve: serve killed mid-run on v3; after restart, the reaper settles the run.
• v2 unchanged: the same scenario on v2 behaves as today.
• Ordinary uncommitted writes are still removed: a killed model method run (not a workflow) keeps today's v3 recovery behaviour.
• swamp-uat passes unchanged. If a swamp-uat test pins the v3 deletion, list it in the PR.

Severity: S2. Everyday crashes lose run history and completed outputs on v3, contrary to the documented contract and to v2.

Blocked by: nothing; swamp-club#3564 has shipped. It runs in parallel with the outbox correctness issue (bug hunt group 3), which also touches interrupted-writer recovery; expect a small rebase. The recovery-sweep scope issue (group 8) follows this one.