#3452 Concurrent run writers leave finished runs stuck as running in the run index: repairIndex returns early once counts agree, so lost updates are never healed (regression from #3105)

Summary

Since #3105 (c5038e5b, swamp-club#3443 / swamp-club#3407), when two or more processes write runs of the same workflow concurrently, a finished run can stay running in that workflow's run index (.runs-index.json) while its record says succeeded. The stale entry persists: workflow run search, workflow.run.search / workflow.run.summary in serve, and the dashboard keep showing the run as running until something forces a real record/entry count mismatch, or run get <id> / run doctor touches it.

Cause

Index read-modify-write is serialized only within one process (withIndexQueue in src/infrastructure/persistence/workflow_run_index.ts), so two writer processes can overwrite each other's entry update. That race predates #3105 (it already causes occasional missing entries on 3999f85e).

Before #3105, any reader that counted a record mismatch (which happens transiently on every save: the record is written before its entry) did a full rebuildIndex, re-reading every record — which incidentally healed lost status updates.

Since #3105, repairIndex (src/infrastructure/persistence/yaml_workflow_run_repository.ts:1314) re-checks the count inside the index queue and returns early when it now matches:

if (!isIndexStale(result, count)) return result.entries;

so the fingerprint check (findSuspectRecords / reconcileSuspects) that would catch the rewritten record never runs, and nothing heals the lost update. The entry's stored record fingerprint no longer matches the record file, so the stat-based check would find it.

Reproduction

Repo with one workflow (shell step, ~0.5 s/run) and ~1,750 existing runs. Run 4 CLI writer loops (swamp workflow run <wf>) against it for 150 s, no serve. Then compare every index entry's status with its record's status.

With serve running alongside mixed old/new writers: 0 of 5 all-old trials vs 4 of 6 trials with any new process (1, 2, 10, 1 stale entries). Adding one record by hand forces a real count mismatch and heals them (4 → 0).

Suggested fix

When repairIndex finds the counts agree inside the queue, still run the stat-only findSuspectRecords + reconcileSuspects before returning (one stat per record, and only after a reader has seen a mismatch — still far cheaper than the old full rebuild). Longer term, a cross-process lock on index updates removes the underlying race.

Environment

Linux x86_64, Deno 2.9.7, ext4 (NVMe). Found during adversarial e2e verification of #3105.