Relationships
#4631 Team mode: a single save is slow on GCS (1.7 s near, 5.3 s far p95), mostly waiting for a lock nobody holds; far workflows and the first command after idle too
Opened by stack72 · 11/11/2026· Shipped 11/12/2026
Found in team-mode dogfood run 7 (swamp-club#4374) on GCS, release 20261110.234711.0-sha.3432b140. Team mode is now reliable (run 7's gate passed: no failed writes, no lost data, failover pauses of 3–5 s). What's left is speed, and this is the part every user feels on every command: a single save is slow, more so from far away or after an idle pause. #4574 shipped against these targets and they still miss on the VMs.
Numbers (run 7, part B)
| Measure | Run 7 | Target |
|---|---|---|
K2a: one-step write to a behavior nobody else writes (k2a-solo), near |
p50 1.70 s, p95 3.47 s | ≤ 1 s |
| K2a far | p50 2.97 s, p95 5.28 s | ≤ 2 s p95 |
| Lock wait inside K2a, with no competing writer | p95 1.89 s near, 3.68 s far | close to one round trip |
Two-step workflow (nightly-report) far |
p50 10.12 s, 1.66× v2 (6.10 s) | ≤ 1.5× v2 (9.15 s) |
| Per step far (means) | health 2.23 s, report 2.80 s; 3 requests per step |
— |
| First command after 6 min idle (cold read) | 6.5–6.8 s (first 17.4 s); warm 1.4–3.5 s | ≤ 1.5× warm |
- K2a sends 2 requests per write, so the time is not request count. The lock wait with no other writer is the largest single cost, and it is likely what keeps the far workflow over 1.5× (run 6 also showed far lock wait of ~0.9 s per step).
- Near two-step workflow passes (7.46 s, 1.22× v2).
- Run 7 may have measured K2a with swamp-club telemetry on, which costs about 420 ms per command (F33, a separate issue). Separate the two in the breakdown.
Evidence
- Report: the #4374 session's run 7 parts B/C report (ask the #4374 session for
4374-run7-partBC-report.md). - Controller VM:
~/dogfood/results/results.jsonl(m-k2a-*,b-twostep,m-sync), and per-step spans and breakdowns in~/dogfood/results/bx/. Member and serve OTel spans from the run are kept by the #4374 session.
Work
- Break the time down first, from run 7's spans (member and serve), before changing code: for one K2a near and one far, and one cold read, where does the lock wait go? Candidates:
- serve waiting for the lease commit to be confirmed replicated to the bucket before answering the lock;
- the catch-up bundled with the lock (#4525) doing work even when nothing is new;
- head reads or head publishes on the lock path;
- connection set-up or TLS per command (and, for the cold read, after the idle connection was retired by #4559's deadlines);
- telemetry, if it was on.
- Remove the waits that aren't needed for correctness. An uncontended lock should cost about one round trip to serve. Fencing guarantees (#4559, F43's fence at the append) must hold.
- Cold read: the first command after idle should not pay much more than a warm one.
Tests
- In-process: an uncontended lock acquisition does a bounded amount of work (requests, bucket reads/writes, commits), asserted by counts, not wall-clock.
- The existing failover, fence and F43 tests keep passing.
Done when
Measured on the dogfood VMs (GCS), not only locally — #4574 and #4580 both looked fine locally and missed on the VMs. Before the PR, run K2a (30 near, 30 far), the two-step workflow far (10) and the cold read (5) on the VMs with a build of the branch, telemetry off:
- K2a near p50 ≤ 1 s, far p95 ≤ 2 s;
- far two-step workflow ≤ 1.5× v2;
- cold read ≤ 1.5× warm.
Must not regress
- No acknowledged write lost; failover pauses ≤ 15 s; zero diverged copies (#4559, #4581).
- Run 7 gate results (100% workflow success under team load, zero version clashes).
Shipped
Click a lifecycle step above to view its details.
Sign in to post a ripple.