Skip to main content
← Back to list
01Issue
BugShippedSwamp CLIPublic
Assigneesstack72

Relationships

↔ sibling #4786↔ sibling #4785

#4787 Test flake: fan-out rollout 'maxFailures 25% over 8 lanes halts on the third failure mid-batch' sometimes dispatches one lane more than expected

Opened by stack72 · 11/16/2026· Shipped 11/16/2026

integration/fleet_fan_out_rollout_test.ts:228 "fan-out rollout: maxFailures 25% over 8 lanes halts on the third failure mid-batch; the running lane finishes and the rest are held" fails intermittently in verify-build.

Seen: 2 failures in 321 recorded verify-build runs (counted from the main checkout's workflow-run logs, 2026-11-16). The latest was on the swamp-club#4784 branch (run de31fd98), which doesn't touch workflows or the fleet. On a separate run (swamp-club#4783, run 9cbbff91) the same test stalled for over 4 minutes under heavy host load (load average about 35) until run-tests hit its 15-minute limit; it normally takes 1-12 s.

Failure: the per-worker dispatch count differs by one lane:

  "w-5": 1,
- "w-6": 1,   (actual)
+ "w-6": 0,   (expected)
  "w-7": 0,

at integration/fleet_fan_out_rollout_test.ts:255.

Likely cause: the halt on the third failure races the next lane's dispatch. When the third failure is recorded after w-6 has already been dispatched, one extra lane runs. Either the halt must be decided before a dispatch goes out, or the test must allow the lane that was already in flight. The 4-minute stall suggests a wait in the same path with no bound under load.

02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 3 MOREISSUE_LINKED+ 5 MOREREVIEW+ 7 MOREPR_LINKED+ 2 MORESESSION_SUMMARIZED

Shipped

11/16/2026, 11:42:59 AM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack7211/16/2026, 7:34:43 AM
stack72 linked sibling of #478611/16/2026, 7:36:03 AM
stack72 linked sibling of #478511/16/2026, 7:36:06 AM

Sign in to post a ripple.