Relationships
#4492 verify-build can hang for up to an hour when test workers stop loading, and type checks during tests add heavy CPU load
Opened by skunk-ape · 11/7/2026· Shipped 11/7/2026
In some full test runs, new test workers stop loading partway through. Worker-based tests then fail at their 1 or 2 minute limits, while a v3 restore: test waits forever because its harness has no deadline. The verify-build test step has no time limit, so the run stays silent for 25–54 minutes until someone kills it. Killing it loses Deno's final error output. A normal run takes about 5 minutes.
The test step also type-checks during the run: about 36 full checks as workers load, plus an initial whole-suite check that delays the first test by 46 s. On a set of 10 worker-heavy test files with a cold cache, this was 412 s of CPU with checks and 154 s without. On macOS in Flaky Test Detection, the initial check has run out of memory at about 3.5 GB of heap.
Changes
- Give the worker calls in
integration/datastore_v3_litestream_restore_test.tsa deadline, as the sibling harnesses have, and make sure teardown doesn't wait behind a stuck call. - Cap the
verify-buildtest step at 15 minutes using the command/shelltimeoutargument. - Run the
verify-buildand Flaky Test Detection test steps with--no-check. The static-analysis step still type-checks everything and still gatescompile. Localdeno run testis unchanged.
Acceptance
- A worker that never answers fails its restore test instead of hanging it, and teardown completes.
- A stuck test step fails at the cap and reports as failed.
- A type error still fails
verify-build. - The Flaky Test Detection macOS job no longer runs out of memory during type checking.
Out of scope: the root cause of the worker-load stall, which is still unknown; a host-wide lock for concurrent verification; and trimming the slowest tests.
Shipped
Click a lifecycle step above to view its details.
Sign in to post a ripple.