Skip to main content
← Back to list
01Issue
FeatureShippedSwamp CLIPublic
Assigneesstack72

Relationships

#2282 Eliminate test flakiness: detect flaky tests with deno --repeats, track and fix them

Opened by skunk-ape · 9/18/2026· Shipped 9/23/2026

Split out from #2270. That issue proposed a merge gate running the full suite against the PR merge result with deno task test --retry 1. #2270 closed without that gate: .github/workflows/ci.yml has no test job, and nothing in the repo passes --retry or --repeats. Tests run in two places today — the run-tests step of verification/workflow-verify-build.yaml (plain deno task test, run locally, gating the attestation) and post-merge in windows-compat.yml.

Why

Every flake in verify-build fails the attestation and forces a full re-run. AGENTS.md "Timing and flakiness rules" prevent new flakes; they do not find the ones already there. Deno 2.9.7 (pinned in .tool-versions) has --repeats N, which runs each test N extra times and requires every repetition to pass — the hunting tool.

Decisions

  • verify-build stays strict — no --retry. It runs with an agent in session, so a flake costs a re-run, not a stalled PR. Adding --retry there would let a PR that introduces a new flaky test pass on retry and be attested.
  • Detection runs as a scheduled GitHub Actions job, off the merge path. Not per-merge: flakes are nondeterministic, not commit-local, so a post-merge run blames whichever PR happened to land.
  • Alerts go to the swamp-club lab, not GitHub issues — auto-response.yml moves every GitHub issue into the lab.

Scope of this issue

New workflow .github/workflows/flaky-tests.yml:

  • deno task test --repeats 3 on ubuntu-latest and macos-latest, every 6 hours plus workflow_dispatch.
  • Not a required check; no push or pull_request trigger; runs never overlap.
  • On failure, the run summary lists the failing tests per OS (or the log tail if the run died early).
  • On failure, an alert job opens one lab issue titled "Scheduled flaky-test detection is failing", or comments on it if it is already open.

Acceptance

  • The workflow runs from main via workflow_dispatch on both OSes and the summary names the failing tests.
  • The first red run opens one lab issue; a second red run comments on it instead of opening another.
  • verify-build still runs deno task test without --retry.

Out of scope — decided after the first CI run

Fixing the tests the job reports. A local macOS --repeats 3 run already shows 28 failures (mostly tests that only fail when repeated in the same process, via the process-wide type registries, plus one parallel flake in integration/otel_logs_worker_correlation_test.ts); the CI run's Linux and macOS list is the authoritative input.

02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 5 MOREREVIEW+ 10 MOREPR_MERGED+ 2 MORESESSION_SUMMARIZED

Shipped

9/23/2026, 11:21:47 AM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack729/23/2026, 7:18:03 AM

Sign in to post a ripple.