Relationships
#4574 Team mode speed: far-member workflows within 1.5x v2, fewer commits per step and workflow, and no slow fresh clone, cold read or teardown after large outputs (F41, F42 from #4374 run 6)
Opened by stack72 · 11/10/2026· Shipped 11/10/2026
From the team-mode dogfood on real GCS (swamp-club#4374), run 6 parts B and C, release 20261109.214205.0-sha.2c24fe25 (with #4525 and #4539). This is the remaining speed work before team mode ships. It follows #4525, ideally with the same owner, since it built the commit batching. The full report is ~/Desktop/4374-run6-partBC-report.md.
Where we are
| Measure | Run 6 | Target | Verdict |
|---|---|---|---|
| Two-step workflow, near | median 9.1 s, 1.33× v2 (v2 6.86 s) | ≤ 1.5× v2 | pass |
| Two-step workflow, far | median 12.6 s, 1.84× v2 | ≤ 1.5× v2 (≤ 10.3 s) | fail |
Fresh member clone and data list |
62–64 s, 94 MB (run 4: 37–44 s; v2: 3.8–6.2 s) | within 2× v2 | fail, and a regression since run 4 |
| Cold read after 6 min idle | 7.7–8.6 s (warm 3.0–6.1 s) | within 1.5× warm | fail |
| Teardown after large outputs | 2.6–7.7 s (small commands: 64–80 ms) | no run over 3 s | fail |
The two-step workflow, per step (means of 10, ms)
| Near | Far | v2 | |
|---|---|---|---|
| Wall | 10,646 | 12,600 | 7,101 |
| Before the workflow (catch-up) | 895 (catch-up 434) | 1,408 (catch-up 1,022) | 457 |
Step health: commits / commit time / lock wait / requests |
3 / 1,681 / 274 / 4 | 3 / 1,880 / 901 / 6 | 0 (75 GCS calls) |
Step report: commits / commit time / requests |
3 / 2,220 / 6 | 4 / 3,486 / 9 | 0 (67 GCS calls) |
| Workflow-level, outside the steps | 4 commits, 2,118 | 4 commits, 2,315 | 222 |
- Per commit (
health, near): Litestream sync 213 ms,publish_head209 ms, after-commit 155 ms, lease check 71 ms. - Workflow-level
publish_headis 440 ms, including 429 retries from #4522's retry layer. - v3 over v2 near is +3.5 s: workflow-level commits +1.9 s, the steps +1.8 s, the pre-workflow catch-up +0.44 s.
- Far adds: lock wait (0.9 s in
health), about 150 ms of network per commit, and 1 s of catch-up before the workflow.
Work
- Commits per step and per workflow aren't where #4525 aimed (2 per step, 2 at workflow level). On GCS they're still 3–4 per step and 4 at workflow level, although #4525's local lane showed the lower counts. Find out why the batching doesn't take effect on the dogfood: chunk caps, the large-output path, the batch route not used, or counted differently. Then bring the counts down. This is the main lever for the far target, and the workflow-level commits are the biggest single piece.
- Head-publish contention: the workflow-level
publish_head(440 ms) still includes 429 retries, so #4525's contention-triggered pacing doesn't seem to engage. Check it on the run 6 spans and fix it. - The far member's lock wait per step (0.9 s) and its catch-up before the workflow (1 s).
- F41, teardown after large outputs:
swamp.cli.teardowntook 7.7 s once after a nearm_bigrun, 2.6–5.4 s in 4 of 5 farm_bigruns, and 6.6 s after a 4 MiB output.- F39's fix (#4525) covers small commands only.
- Span each teardown wait after a large output, and bound it.
- F42, fresh member clone regression: 62–64 s on run 6, against 37–44 s on run 4. Bisect against the releases since run 4 (#4512's recovery changes, #4539, #4542 and #4525 are candidates), fix it, and bring the clone within 2× v2.
- Cold read after 6 min idle (7.7–8.6 s): serve's token check was ruled out in run 4 (under 100 ms), so the cost is on the member: reconnection, TLS, catch-up after idle, or the read model's open. Find it and cut it.
Measurement note
K2a's p95 (near 4.65 s, far 6.12 s) is set by lock wait: the measured behavior is also written by other members' load, so it waits its turn, which is by design. K2a's bar is measured on a behavior nobody else writes. Contended writes are reported separately and aren't part of this issue.
Tests
- Work-count tests, failing on current main where the counts are wrong:
- commits per step and per workflow for a two-step workflow, including a step with a large output;
- requests before the workflow.
- A teardown test after a large output: each wait bounded, using an injected stall.
- A clone work-count test: what a fresh clone fetches and writes, at a fixed history.
Must not regress
- #4525's request counts per step, #4539's flat per-commit cost, and #4542's large-output behaviour.
- No acknowledged write lost; #4431's exit contract.
Done when
The lane shows the target commit counts. Then the next dogfood shows:
- the two-step workflow far ≤ 1.5× v2;
- a fresh clone within 2× v2;
- a cold read within 1.5× warm;
- no teardown over 3 s after a large output.
Shipped
Click a lifecycle step above to view its details.
Sign in to post a ripple.