#4787 Test flake: fan-out rollout 'maxFailures 25% over 8 lanes halts on the third failure mid-batch' sometimes dispatches one lane more than expected
Opened by stack72 · 11/16/2026· Shipped 11/16/2026
integration/fleet_fan_out_rollout_test.ts:228 "fan-out rollout: maxFailures 25% over 8 lanes halts on the third failure mid-batch; the running lane finishes and the rest are held" fails intermittently in verify-build.
Seen: 2 failures in 321 recorded verify-build runs (counted from the main checkout's workflow-run logs, 2026-11-16). The latest was on the swamp-club#4784 branch (run de31fd98), which doesn't touch workflows or the fleet. On a separate run (swamp-club#4783, run 9cbbff91) the same test stalled for over 4 minutes under heavy host load (load average about 35) until run-tests hit its 15-minute limit; it normally takes 1-12 s.
Failure: the per-worker dispatch count differs by one lane:
"w-5": 1,
- "w-6": 1, (actual)
+ "w-6": 0, (expected)
"w-7": 0,at integration/fleet_fan_out_rollout_test.ts:255.
Likely cause: the halt on the third failure races the next lane's dispatch. When the third failure is recorded after w-6 has already been dispatched, one extra lane runs. Either the halt must be decided before a dispatch goes out, or the test must allow the lane that was already in flight. The 4-minute stall suggests a wait in the same path with no bound under load.
Shipped
Click a lifecycle step above to view its details.