Skip to main content
← Back to list
01Issue
BugShippedSwamp CLIPublic
Assigneesstack72

Relationships

#4567 Team mode: a member's workflow run fails when another member's overlapping run commits the shared workflow report version first ('version N is already live'), about 1 run in 8 under load (F40 from #4374)

Opened by stack72 · 11/10/2026· Shipped 11/10/2026

Found by the team-mode dogfood on real GCS (swamp-club#4374), run 6 part A, 2026-11-09. Release 20261109.214205.0-sha.2c24fe25, @swamp/gcs-datastore 2026.11.09.1. A ship blocker: under normal team load, one member's workflow runs fail about 1 in 8.

What happens

Three members each ran a mixed load loop for 30 minutes. bob's swamp workflow run nightly-report --json failed 3 times out of 24, with exit 1, no code, and this on stderr:

Data report-swamp-workflow-summary version 30 is already live; only an append re-records it; 1 workflow run record(s) were held back, since they may name data a refused partition did not record (run--f0af6571-…--e25bdf42-ecc9-48b5-88b8-42c94a25f3e2)

The other two failures were identical, with versions 43 and 63. alice (21 of 21) and carol (27 of 27) had none. The same refusal appeared in dogfood run 1 (F5); nothing has targeted it since.

The workflow: nightly-report is one job with two steps: health (behavior ops-healthcheck), then report (behavior report-gen). Every run, by any member, also writes a new version of one shared data item, the workflow report report-swamp-workflow-summary.

bob's failed run (UTC) Run id Other members' overlapping runs
23:19:19–23:19:37 e25bdf42-ecc9-48b5-88b8-42c94a25f3e2 carol 23:19:16–23:19:31, succeeded
23:24:47–23:25:16 c98fa187-216d-4236-9add-99094c6ed912 alice 23:24:44–23:24:59 and carol 23:24:45–23:25:08, both succeeded
23:33:48–23:34:09 3c89323b-95e0-40ed-97a2-bd833906eced carol 23:33:46–23:33:59, succeeded
  • Each failure overlapped another member's run of the same workflow that started 2–3 s earlier and succeeded.
  • But overlap alone isn't enough: alice's and carol's runs at 23:24:44–45 overlapped, and both succeeded.
  • bob's grants differ:
    • bob: read on everything; run on workflow:* and model:*; write on model:ops-* and data:*. No write on workflow:*.
    • alice: everything;
    • carol: read, run on everything; write on data:*; approve on workflow:*.
  • The held-back run records landed later, and the runs show as failed in bob's workflow history.
  • No GCS 429 was involved (#4522's head-object retries were seen in the window, but none coincided with these failures).

Likely cause (to confirm)

The report data's next version is taken from the member's copy ("next is N"), and a concurrent run committed N first. Serve then refuses bob's write as a rewrite of a version already live. Since alice's and carol's overlapping runs both succeed, their report writes are probably appends that serve sequences, while bob's takes a different path: a write of a specific version. The obvious difference is bob's missing write on workflow:*; the workflow summary's write may be authorised or routed differently for him. #4525 removed per-step catch-ups, which may make a member's copy staler and the race more frequent.

Fix

  1. Find why bob's report write takes a different path, and make every member's take the same one, whatever their grants.
  2. Allocate the version of shared run-summary data at commit time: an append that serve sequences, never a version taken from the member's copy. Two members' overlapping runs both succeed, with consecutive versions.
  3. If a refusal still happens, it carries a stable code, so scripts and agents can classify it, and the run's step outputs aren't lost. Check whether health's and report's outputs were kept in the failed runs.

Tests

  • Two members, one with bob's grant set, run the same workflow with starts 0–3 s apart, many times. Every run succeeds, the summary versions are consecutive and unique, and neither is refused. Red on current main.
  • The same with two full-grant members (stays green).
  • A forced refusal carries a code, and the step outputs survive.

Must not regress

  • No acknowledged write lost, and no version silently overwritten.
  • #4454's initiator check and grant checks: bob still can't write what his grants don't allow.
  • #4525's commit and request counts per workflow.

Evidence

  • bob's failures: member-2 ~/load/fail/20261109T231937-wf-nightly.err, …T232516…, …T233409…;
  • ~/load/ops.jsonl on every member;
  • the controller's ~/dogfood/results/results.jsonl (the a1-load rows);
  • spans and logs in each member's /var/lib/otel/spans.jsonl and logs.jsonl (services swamp-alice, swamp-bob, swamp-carol).

Done when

The tests pass, and the next dogfood shows every member's nightly-report at 93% or better under load, with no F40 refusals.

02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 5 MOREREVIEW+ 12 MOREPR_LINKED+ 2 MORESESSION_SUMMARIZED

Shipped

11/10/2026, 3:44:43 AM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack7211/10/2026, 12:01:40 AM

Sign in to post a ripple.