← Back to list
01Issue
BugShippedSwamp CLIPublic
Assigneesstack72
Relationships
#4579 Team mode: a member's recovery after an interrupted command fails with 'Cannot apply commit N of partition run--…' (from #4374 run 5)
Opened by stack72 · 11/10/2026· Shipped 11/10/2026
Found by #4559's analysis of team-mode dogfood run 5 (swamp-club#4374). A ship blocker that gates dogfood run 7. After a member command is interrupted (for example, during a serve failover), the next command's recovery of the member checkout fails:
Recovering /home/…/test after a swamp process ended without finishing its v3 datastore writes failed: Cannot apply commit 5 of partition run--…
- Where it was seen: several run 5 c05 (leader killed) rounds, with the failing recoveries taking 45–143 s. Release
20261108.111402.0-sha.37c27525. - Why it matters: a recovery that fails can leave a member checkout unable to make progress until someone intervenes. It also makes failover rounds look slow, which hides whether #4559's failover fix works.
Evidence
- Run 5 data extracted by #4559:
/private/tmp/claude-501/-Users-stack72--herdr-worktrees-swamp-4559/0195134b-0bae-4398-9a35-65526cd0554e/scratchpad/evidence/(probes-4559.json.gz,results.jsonl,orchestrator.log). - The dogfood controller's
~/dogfood/infra/evidence.log. - Members'
~/load/fail/and~/load/ops.jsonlon the dogfood VMs (ask the #4374 session). - A local failover repro from #4559:
/private/tmp/claude-501/-Users-stack72--herdr-worktrees-swamp-4559/0195134b-0bae-4398-9a35-65526cd0554e/scratchpad/repro/failover.ts(fake-gcs and two serves behind a TCP proxy;DEBUG_MEMBERS=1keeps member debug logs).
Work
- Reproduce it: kill the leader while a member's workflow is committing to a
run--…partition, then run the member's next command and capture the debug log. Find which commit fails to apply and why. Candidates: a commit the member staged or sent that serve later rebuilt or refused; a partition seq gap after a fenced or lapsed lease; the outbox holding a commit serve already landed under another seq. - Fix the cause so recovery completes. If the commit genuinely can't apply, recovery must settle it (drop or mark it, as the outbox does for refused entries) and carry on, not fail the whole recovery.
- The error carries a code, and names the commit and the reason it can't apply.
Tests
- The repro as a test, red on current main: an interrupted member write during a leader kill, then the next command's recovery completes.
- A commit that can't apply is settled, and recovery finishes.
Must not regress
- No acknowledged write lost.
- #4512's recovery behaviour: stoppable and resumable.
- #4559's failover behaviour.
Done when
The test passes, and dogfood run 7's failover rounds show no failed member recoveries.
02Bog Flow
Shipped
Click a lifecycle step above to view its details.
03Sludge Pulse
Sign in to post a ripple.