← Back to list
01Issue
BugShippedSwamp CLIPublic
Assigneesstack72
Relationships
↔ sibling #4793#4806 Flaky under load: team forwarding test 'a leader that left two forwards in a row unanswered is refused at once' gets a may-have-run reply
Opened by stack72 · 11/16/2026· Shipped 11/17/2026
The test "team forwarding: a leader that left two forwards in a row unanswered is refused at once, never sent, until a new leader takes over" (integration/serve_v3_team_forwarding_test.ts, also matched in src/serve/leader_link_test.ts) fails intermittently under load and passes alone.
Evidence
- Past verify-build logs in the main checkout's .swamp data: 17 failures in 126 runs that ran it (2026-11-13: 4 of 28; 2026-11-14: 3 of 12; 2026-11-16: 10 of 84), across several sessions' branches, starting before any of them touched serve's forwarding.
- On 2026-11-16 it failed in a full local run of integration/ plus the datastore_v3, persistence and cli unit suites (12,294 passed, 3 failed), then passed 3 of 3 alone on the same commit.
- The failing assertion: the reply was serve_not_leader with "This swamp serve node forwarded the request to the leader, which changed or stopped answering before it replied. The request may already have run" and inFlight: true, where the test expects the refusal it gets once the leader is known to have left two forwards unanswered (refused at once, never sent).
Likely cause
A timing dependency: under load the second forward is still in flight, or not yet counted as unanswered, when the test's next request arrives, so serve forwards it and answers "may already have run" instead of refusing it without sending. The test should wait on the state change (the leader marked as having left two forwards unanswered) rather than on timing.
02Bog Flow
Shipped
Click a lifecycle step above to view its details.
03Sludge Pulse