Skip to main content
← Back to list
01Issue
BugShippedSwamp CLIPublic
Assigneesstack72

Relationships

↔ sibling #4793

#4819 Flaky under load: workflow report race, team-mode read-lock and recovery chunk-read integration tests

Opened by stack72 · 11/16/2026· Shipped 11/17/2026

Three integration test files fail intermittently when verify-build runs under load, and pass alone.

Tests and evidence

Counts are from past verify-build logs in the main checkout's .swamp data, across every session's branches:

  • integration/serve_upstream_workflow_report_race_test.ts ("workflow report race: two full-grant members: overlapping runs of one workflow all succeed, with the shared report's versions consecutive", and three siblings): 41 failed of 320 runs.
  • integration/serve_upstream_method_run_read_lock_test.ts ("team-mode read run: serve's own read method run during a member's leased write, on a datastore serve claimed / did not claim, waits for it, ..."): 33 failed of 272 runs.
  • integration/serve_v3_recovery_refused_read_test.ts ("serve recovery: a connection lost while the recovery reads a chunk keeps its storage-unreachable code, as before"): 34 failed of 278 runs; the failing run hits the 2-minute test timeout.

On 2026-11-16 all seven failed in one verify-build shard (load average about 10, run-tests took 12 minutes against the usual 7-8), then the three files passed 13 of 13 twice in a row run alone on the same commit.

Likely cause

Timing under load: these tests hold two runs at a barrier or wait on a lock hand-off between serve and a member, and the recovery test waits on a dropped connection, so a slow host lets a wait or deadline pass first. Each should wait on the state change it asserts rather than on time, as the swamp-club#4785/#4786 fixes did for the owned-sidecar replication test.

02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 10 MOREPR_LINKED+ 1 MORESESSION_SUMMARIZED

Shipped

11/17/2026, 8:23:26 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack7211/16/2026, 9:45:06 PM
stack72 linked sibling of #479311/17/2026, 1:04:09 PM

Sign in to post a ripple.