Skip to main content
← Back to list
01Issue
BugShippedSwamp CLIPublic
Assigneesstack72

Relationships

#4574 Team mode speed: far-member workflows within 1.5x v2, fewer commits per step and workflow, and no slow fresh clone, cold read or teardown after large outputs (F41, F42 from #4374 run 6)

Opened by stack72 · 11/10/2026· Shipped 11/10/2026

From the team-mode dogfood on real GCS (swamp-club#4374), run 6 parts B and C, release 20261109.214205.0-sha.2c24fe25 (with #4525 and #4539). This is the remaining speed work before team mode ships. It follows #4525, ideally with the same owner, since it built the commit batching. The full report is ~/Desktop/4374-run6-partBC-report.md.

Where we are

Measure Run 6 Target Verdict
Two-step workflow, near median 9.1 s, 1.33× v2 (v2 6.86 s) ≤ 1.5× v2 pass
Two-step workflow, far median 12.6 s, 1.84× v2 ≤ 1.5× v2 (≤ 10.3 s) fail
Fresh member clone and data list 62–64 s, 94 MB (run 4: 37–44 s; v2: 3.8–6.2 s) within 2× v2 fail, and a regression since run 4
Cold read after 6 min idle 7.7–8.6 s (warm 3.0–6.1 s) within 1.5× warm fail
Teardown after large outputs 2.6–7.7 s (small commands: 64–80 ms) no run over 3 s fail

The two-step workflow, per step (means of 10, ms)

Near Far v2
Wall 10,646 12,600 7,101
Before the workflow (catch-up) 895 (catch-up 434) 1,408 (catch-up 1,022) 457
Step health: commits / commit time / lock wait / requests 3 / 1,681 / 274 / 4 3 / 1,880 / 901 / 6 0 (75 GCS calls)
Step report: commits / commit time / requests 3 / 2,220 / 6 4 / 3,486 / 9 0 (67 GCS calls)
Workflow-level, outside the steps 4 commits, 2,118 4 commits, 2,315 222
  • Per commit (health, near): Litestream sync 213 ms, publish_head 209 ms, after-commit 155 ms, lease check 71 ms.
  • Workflow-level publish_head is 440 ms, including 429 retries from #4522's retry layer.
  • v3 over v2 near is +3.5 s: workflow-level commits +1.9 s, the steps +1.8 s, the pre-workflow catch-up +0.44 s.
  • Far adds: lock wait (0.9 s in health), about 150 ms of network per commit, and 1 s of catch-up before the workflow.

Work

  1. Commits per step and per workflow aren't where #4525 aimed (2 per step, 2 at workflow level). On GCS they're still 3–4 per step and 4 at workflow level, although #4525's local lane showed the lower counts. Find out why the batching doesn't take effect on the dogfood: chunk caps, the large-output path, the batch route not used, or counted differently. Then bring the counts down. This is the main lever for the far target, and the workflow-level commits are the biggest single piece.
  2. Head-publish contention: the workflow-level publish_head (440 ms) still includes 429 retries, so #4525's contention-triggered pacing doesn't seem to engage. Check it on the run 6 spans and fix it.
  3. The far member's lock wait per step (0.9 s) and its catch-up before the workflow (1 s).
  4. F41, teardown after large outputs:
    • swamp.cli.teardown took 7.7 s once after a near m_big run, 2.6–5.4 s in 4 of 5 far m_big runs, and 6.6 s after a 4 MiB output.
    • F39's fix (#4525) covers small commands only.
    • Span each teardown wait after a large output, and bound it.
  5. F42, fresh member clone regression: 62–64 s on run 6, against 37–44 s on run 4. Bisect against the releases since run 4 (#4512's recovery changes, #4539, #4542 and #4525 are candidates), fix it, and bring the clone within 2× v2.
  6. Cold read after 6 min idle (7.7–8.6 s): serve's token check was ruled out in run 4 (under 100 ms), so the cost is on the member: reconnection, TLS, catch-up after idle, or the read model's open. Find it and cut it.

Measurement note

K2a's p95 (near 4.65 s, far 6.12 s) is set by lock wait: the measured behavior is also written by other members' load, so it waits its turn, which is by design. K2a's bar is measured on a behavior nobody else writes. Contended writes are reported separately and aren't part of this issue.

Tests

  • Work-count tests, failing on current main where the counts are wrong:
    • commits per step and per workflow for a two-step workflow, including a step with a large output;
    • requests before the workflow.
  • A teardown test after a large output: each wait bounded, using an injected stall.
  • A clone work-count test: what a fresh clone fetches and writes, at a fixed history.

Must not regress

  • #4525's request counts per step, #4539's flat per-commit cost, and #4542's large-output behaviour.
  • No acknowledged write lost; #4431's exit contract.

Done when

The lane shows the target commit counts. Then the next dogfood shows:

  • the two-step workflow far ≤ 1.5× v2;
  • a fresh clone within 2× v2;
  • a cold read within 1.5× warm;
  • no teardown over 3 s after a large output.
02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 6 MOREREVIEW+ 12 MOREPR_LINKED+ 2 MORESESSION_SUMMARIZED

Shipped

11/10/2026, 9:05:34 AM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack7211/10/2026, 3:33:32 AM

Sign in to post a ripple.