Relationships
#4525 Team mode: a member write takes 2 requests to serve instead of 8, workflows batch per step, and write cost stays flat over a day of history (performance from #4374)
Opened by stack72 · 11/8/2026· Shipped 11/9/2026
From the team-mode dogfood on real GCS (swamp-club#4374), runs 4 and 5. Team-mode writes work, but they're slow: near members wait about 1.5 s per one-step write and far members about 4.4 s, and a two-step workflow costs about 2× v2. Most of that is member-to-serve round trips, not serve's own work. This issue cuts them. It follows #4457 (12 → 8 requests per write) and works on the same commit path as #4522 (F38), so it ideally has the same owner, after #4522 lands or rebased onto it.
Where the time goes (run 4, one-step write, joined member and serve spans)
| Request | Per write | Member p50 near / far | Serve p50 |
|---|---|---|---|
GET partitions/{p}/commits |
3 | 173 / 1,145 ms | 13 ms |
GET changes |
2 | 47 / 421 ms | 10–12 ms |
POST partitions/{p}/commits (the commit) |
1 | 333 / 542 ms | 324 / 428 ms |
POST locks |
1 | 94 / 630 ms | 16–18 ms |
DELETE locks/{id} |
1 | 12 / 127 ms | 7–9 ms |
- K2a, one write: near p50/p95 1.14–1.25 / 1.52–1.75 s fresh, rising to about 3.6–4.3 s p95 by the end of a set; far p95 4.1–4.4 s fresh.
- K2b, a two-step workflow: 40–45 requests, 1.9–2.4× v2.
- Serve's commit (about 320 ms) is mostly
confirm_replicated: Litestream sync about 104 ms, head publish about 94 ms, lease check about 34 ms. That stays; #4522 changes the head publish. - For a far member, round trips are the whole cost: serve's own work is 7–18 ms per request.
- Caveat: run 4 ran with the span exporter bug (F36, fixed in #4511). Dogfood run 5's parts B and E measure again with OTel off. Use those numbers as the baseline when they're in.
Work, in order
- Fold the catch-up into the lock.
POST locksreturns the commits the member's copy lacks for the partitions it's about to write, so the separate partition-commit and change fetches before a write go away. Where the delta is too large for a reply, the client falls back to today's catch-up. - Commit and release in one request. A commit made under a command's own lease can release that lease in the same request, removing the
DELETE locks/{id}. Explicit leases, which outlive the command, are unchanged.- Target: one-step write from 8 requests to 2: lock with catch-up, then commit-and-release.
- The lock must stay before the method runs, since it guards the method; only what comes after merges.
- Workflows: batch per step. Each step's run-record update, step start and data writes go as one commit (or one request) where they're already ordered together, instead of several. Count requests per two-step workflow before and after.
- The growth over a set. K2a p95 roughly doubles over a few hours of load, with catch-up time growing (far p95 2.8 s → 11 s) and
method.executegrowing from about 0.14 s to about 1.9 s outside every child span (F34).- Run 5 part D says how much of that was F36.
- What remains: find it with spans and remove it, so a write's cost stays flat over a day of history (the #4387 pattern).
- CLI start-up: about 2.2–2.5 s before the first span on the dogfood VMs. Only if run 5 part B confirms it with OTel off: find it (module loading, config and auth reads) and cut it back to run 3's release or better.
Targets (dogfood, near and far members)
| Measure | Target | Run 4 |
|---|---|---|
| Requests per one-step write | 2 | 8 |
| K2a near p95, fresh | ≤ 1.0 s | 1.5–1.75 s |
| K2a far p95, fresh | ≤ 2.0 s | 4.1–4.4 s |
| K2a after 6 h of load | ≤ 1.3× fresh | about 2–2.5× |
| K2b | ≤ 1.5× v2 | 1.9–2.4× |
| CLI start-up | at run 3's level or better | about 2.2–2.5 s (to be confirmed) |
Tests
- Work-count tests, failing on current main:
- requests per one-step write is 2;
- requests per two-step workflow drops by the batched amount;
- neither count grows with history or with a second member writing.
- Large delta: a lock reply too large to carry the catch-up falls back cleanly.
- Release on commit: a command's lease is released by its commit, and an explicit lease isn't.
- A failed commit-and-release (refused, unreachable, may-have-landed) leaves the lease as today's paths do, with the same exit codes (#4431's contract).
- An older serve that doesn't understand the combined requests: the client falls back to today's sequence, gated on serve's feature header.
Must not regress
- No acknowledged write lost.
- Leases and fencing: a lapsed or taken-over lease still refuses the commit.
- #4431's exit contract and #4454's lease-holder reporting.
- Offline queuing under an explicit lease.
- Catch-ups still apply only new commits.
Done when
The work-count tests pass and fail on main. A local lane against an emulator with about 100 ms per storage call shows 2 requests per write. Dogfood confirms the targets on GCS, with OTel off.
Shipped
Click a lifecycle step above to view its details.
Sign in to post a ripple.