Relationships
#3455 Run index updates race across processes: concurrent writer processes can lose each other's index entry updates
Opened by zack · 10/14/2026
Summary
Each workflow's run index (.runs-index.json) is updated by a read-modify-write that is serialized only within one process (withIndexQueue in src/infrastructure/persistence/workflow_run_index.ts). Two processes saving runs of the same workflow at once can overwrite each other's entry update: an entry is left with the status and record fingerprint of an earlier write of its record (a finished run shown as running), or a new run's entry goes missing.
Current mitigation
Readers correct this after the fact: a read that sees a record/entry count mismatch stats every record against its entry's fingerprint and fixes the ones that differ (restored for the agreeing-count case by swamp-club#3452), and swamp run doctor verifies every index. Lost updates from the last saves of a burst still stay wrong until a later read sees a mismatch, run doctor runs, or the run itself is read.
Suggested fix
A cross-process lock (or another atomic update scheme) around index read-modify-write, so no update is lost in the first place.
Reproduction
Four concurrent loops of swamp workflow run against one workflow for 150 s leave a handful of entries with index status running while their records say succeeded (see swamp-club#3452).
Closed
No activity in this phase yet.
zack commented 10/14/2026, 5:59:08 PM
Closing as won't do: this will be fixed by the datastore rework, so it gets no separate fix.
Triage confirmed the bug and reproduced it with 4 concurrent workflow run loops for 150 s. Of 536 runs, 8 index entries were left at running while their records said succeeded. Until the rework lands, read-time index repair and run doctor still correct these entries after the fact.
Sign in to post a ripple.