Skip to main content
← Back to list
01Issue
BugShippedSwamp CLIPublic
Assigneesskunk-ape

Relationships

#4492 verify-build can hang for up to an hour when test workers stop loading, and type checks during tests add heavy CPU load

Opened by skunk-ape · 11/7/2026· Shipped 11/7/2026

In some full test runs, new test workers stop loading partway through. Worker-based tests then fail at their 1 or 2 minute limits, while a v3 restore: test waits forever because its harness has no deadline. The verify-build test step has no time limit, so the run stays silent for 25–54 minutes until someone kills it. Killing it loses Deno's final error output. A normal run takes about 5 minutes.

The test step also type-checks during the run: about 36 full checks as workers load, plus an initial whole-suite check that delays the first test by 46 s. On a set of 10 worker-heavy test files with a cold cache, this was 412 s of CPU with checks and 154 s without. On macOS in Flaky Test Detection, the initial check has run out of memory at about 3.5 GB of heap.

Changes

  1. Give the worker calls in integration/datastore_v3_litestream_restore_test.ts a deadline, as the sibling harnesses have, and make sure teardown doesn't wait behind a stuck call.
  2. Cap the verify-build test step at 15 minutes using the command/shell timeout argument.
  3. Run the verify-build and Flaky Test Detection test steps with --no-check. The static-analysis step still type-checks everything and still gates compile. Local deno run test is unchanged.

Acceptance

  • A worker that never answers fails its restore test instead of hanging it, and teardown completes.
  • A stuck test step fails at the cap and reports as failed.
  • A type error still fails verify-build.
  • The Flaky Test Detection macOS job no longer runs out of memory during type checking.

Out of scope: the root cause of the worker-load stall, which is still unknown; a host-wide lock for concurrent verification; and trimming the slowest tests.

02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 2 MOREREVIEW+ 10 MOREPR_LINKED+ 2 MORESESSION_SUMMARIZED

Shipped

11/7/2026, 3:42:29 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
skunk-ape assigned skunk-ape11/7/2026, 2:41:25 AM

Sign in to post a ripple.