Relationships
↔ sibling #4793#4819 Flaky under load: workflow report race, team-mode read-lock and recovery chunk-read integration tests
Opened by stack72 · 11/16/2026· Shipped 11/17/2026
Three integration test files fail intermittently when verify-build runs under load, and pass alone.
Tests and evidence
Counts are from past verify-build logs in the main checkout's .swamp data, across every session's branches:
- integration/serve_upstream_workflow_report_race_test.ts ("workflow report race: two full-grant members: overlapping runs of one workflow all succeed, with the shared report's versions consecutive", and three siblings): 41 failed of 320 runs.
- integration/serve_upstream_method_run_read_lock_test.ts ("team-mode read run: serve's own read method run during a member's leased write, on a datastore serve claimed / did not claim, waits for it, ..."): 33 failed of 272 runs.
- integration/serve_v3_recovery_refused_read_test.ts ("serve recovery: a connection lost while the recovery reads a chunk keeps its storage-unreachable code, as before"): 34 failed of 278 runs; the failing run hits the 2-minute test timeout.
On 2026-11-16 all seven failed in one verify-build shard (load average about 10, run-tests took 12 minutes against the usual 7-8), then the three files passed 13 of 13 twice in a row run alone on the same commit.
Likely cause
Timing under load: these tests hold two runs at a barrier or wait on a lock hand-off between serve and a member, and the recovery test waits on a dropped connection, so a slow host lets a wait or deadline pass first. Each should wait on the state change it asserts rather than on time, as the swamp-club#4785/#4786 fixes did for the owned-sidecar replication test.
Shipped
Click a lifecycle step above to view its details.