Skip to main content
← Back to list
01Issue
FeatureOpenSwamp CLIPublic
AssigneesNone

Relationships

#2282 Eliminate test flakiness: detect flaky tests with deno --repeats, track and fix them

Opened by skunk-ape · 9/18/2026

Split out from #2270. That issue adds a merge gate that runs the full suite against the PR merge result, using deno task test --retry 1 so an intermittent failure does not block auto-merge. The retry is a shock absorber, not a fix — it stops flakes from stalling PRs, but it also means flakes can accumulate unnoticed unless we go after them deliberately.

Why now

Before #2270 a flaky test cost almost nothing: verification runs locally with the agent in session, and a re-run is immediate. Once the suite gates merges, every flake is a PR that needs a human to notice and re-run, so the tolerance for them drops sharply.

AGENTS.md already carries hard-won rules in its "Timing and flakiness rules" section — poll with waitFor instead of a fixed setTimeout, never assert on measured wall-clock durations, set mtime explicitly with Deno.utime rather than sleeping, generate IDs with crypto.randomUUID not Date.now, restore env vars with an undefined check. Those exist because this suite has had timing flakes. They prevent new ones; they do not find the ones already there.

What the tooling gives us

Deno 2.9.7 has two relevant flags, confirmed against the pinned toolchain:

  • --retry N re-runs only the failing tests, and the summary line reports a flaky count, e.g. ok | 2 passed | 0 failed | 1 flaky. Individual tests are marked retrying (attempt 1 failed). This is what #2270 uses, and it makes flakes greppable rather than invisible.
  • --repeats N runs each test N additional times and requires every repetition to pass. This is the hunting tool.

Suggested work

  1. Run deno task test --repeats 3 (and higher) to build an initial list of tests that do not pass deterministically. Run it on Linux as well as macOS — AGENTS.md requires the suite to pass on Linux, macOS and Windows, and timing flakes are often platform-specific.
  2. Triage the list against the AGENTS.md rules above; most hits are likely to be one of those five patterns.
  3. Fix them, and where a fix reveals a convention not yet written down, add it to the AGENTS.md section.
  4. Consider a scheduled job running the suite with --repeats so regressions surface without a human remembering to look. Keep it off the merge path — this is detection, not gating.
  5. Once the list is near zero, revisit whether #2270 gate still needs --retry 1, and whether the flaky count it reports should escalate from a warning to a failure.

Acceptance

The suite passes --repeats 3 on Linux and macOS, and the merge gate introduced by #2270 stops reporting a non-zero flaky count on ordinary PRs.

02Bog Flow
◉OPEN○TRIAGED○IN PROGRESS○SHIPPED

Open

9/18/2026, 5:11:03 PM

No activity in this phase yet.

03Sludge Pulse

Sign in to post a ripple.