#4740 Flaky: command_routes_rules_test's completion-callback test gets no answer from its full-tree worker within 120 s in verify-build
Opened by stack72 · 11/14/2026· Shipped 11/16/2026
What fails
- Test:
integration/command_routes_rules_test.ts, "command routes: a completion callback's routes in a fresh module graph resolve each completion from the same command, to the same values, as the full tree". - Error: the
assert(fullTree.response.ok, …)after the batchedinWorkercalls fails withno answer within 120000ms. The worker loading the full command tree in a fresh module graph doesn't answer in time.
Where it was seen
It failed in 2 of 65 verify-build runs in the main checkout's workflow history, on two unrelated branches:
| Branch | Commit | Date |
|---|---|---|
| 4679 | c07dd6f793 | 2026-11-13 |
| 4702 | 7344249b89 (run ee1cf8a0-32bf-4258-a3e1-5f0ecf44da5f) | 2026-11-14 |
The 4702 run was a rerun of verify-build alone. The previous run of the same commit passed this test. In that rerun the test ran in its own shard (shard 2, one file), as swamp-club#4711 arranged, so a separate test process alone doesn't prevent the timeout. The host's 15-minute load average was about 10 on 18 cores.
Not established
- Whether a cold module graph (deno resolving and loading the whole CLI tree in a new Worker) can simply exceed 120 s on a busy host, or whether one batch of 4 workers starves the full-tree worker.
- A deterministic reproduction, for example limiting the process's CPUs while the batch runs.
Expected
The full-tree worker answers within the bound under verify-build's normal parallel load, or the bound is derived from a measured cold load instead of fixed.
Shipped
Click a lifecycle step above to view its details.
stack72 commented 11/14/2026, 1:56:41 AM
Same file, second case: on the next rerun of verify-build alone for 4702 at 7344249b89 (run c48e[REDACTED-PHONE]-8918-b097f4952537), "command routes: each route loads alone in a fresh module graph with the full tree's schema and error handlers" failed after 4m52s. Other sessions were running verification on the host at the same time (one verify-reviews run was active, and the 15-minute load average was about 11-13 on 18 cores). That run's type check took 109 s and its tests 677 s, against roughly 50 s and 440 s normally. Both command_routes cases load the CLI tree in fresh-module-graph workers against fixed bounds, so they look like one root cause: cold module-graph loads under host contention.
Sign in to post a ripple.