Relationships
#4457 Team mode: a member write makes one catch-up, its spans join serve's trace, and CLI start-up, fresh clones and cold reads stop costing seconds (group 4 latency from #4374, follows #4401)
Opened by stack72 · 11/5/2026· Shipped 11/6/2026
Found by the team-mode dogfood on real GCS (swamp-club#4374), run 3, release 20261105.030003.0 (contains #4387 and #4401). This is group 4 of the dogfood fixes: member write latency, following #4401. One owner, one PR. It runs in parallel with #4431 (member to serve) and #4454 (team admin); see "Overlap" below.
Where a member write's time goes
The dogfood's per-write breakdown, from 30 near one-step writes after about 3 h of team load (K2a p50 2.28 s, p95 4.93 s; fresh p95 was about 1.7 s):
| Phase (member spans) | p50 | p95 |
|---|---|---|
CLI start-up, before swamp.cli |
~1.5–1.8 s | — |
swamp.cli.bootstrap |
76 ms | 99 ms |
Catch-ups from serve (swamp.datastore.v3.serve.catchup) |
1,153 ms | 3,054 ms |
readmodel.catchup |
417 ms | 971 ms |
serve.lock.acquire |
123 ms | 526 ms |
swamp.model.method.run |
2,306 ms | 2,891 ms |
of which serve.forward (the commit) |
659 ms | 820 ms |
of which the rest of method.execute, in no request span (for an echo) |
~1.5 s | — |
serve.lock.release |
123 ms | 313 ms |
swamp.cli.teardown |
345 ms | 686 ms |
| Requests to serve per write | 12 (5 GET partition commits, 4 GET changes, 1 commit, 1 lock, 1 unlock) | max 18 |
Serve, same window: POST …/commits p50 523 ms, p95 891 ms (confirm_replicated 250 ms of it). GCS put 90 ms, get 39 ms, list 33 ms.
Also measured:
- K2b: a two-step workflow costs 2.1–2.6× v2, about 8.4 s per step against 3.4 s.
- F23: a fresh member clone plus
data listtakes 40–67 s and stores 143–415 MB (v2: 4–6 s, 1.3 MB). It appears to download every stored output, including large ones, instead of fetching outputs when they're read. - Cold read: a member read after 6 minutes idle takes 6.8–8.6 s, against 2.0–2.9 s warm.
Work, in order
- F25, trace propagation. Of 2,131 serve spans, none shares a member's trace ID, so the breakdown above is matched by time window. Send
traceparenton every member-to-serve request, so serve's spans join the member's trace. Do this first: every later step is measured with it. - The unexplained ~1.5 s inside
method.execute. Find what it is (anechostep should cost milliseconds), give it a span, and remove or cut it. - One catch-up per command. Today a write makes nine catch-up requests, and their cost grows with load (p95 3.1 s, the main cause of K2a creeping from 1.7 s to 4.9 s). A command should catch up once, then only fetch what its own commit needs. Pin it with a test that counts requests to serve per write, and one showing the count doesn't grow with history or load.
- CLI start-up, 1.5–1.8 s on every command. Find where it goes (binary start, the auth gate, the update check, module loading) and cut it. It falls outside K2a, but members feel it on every command.
- F23 and cold reads. A fresh clone fetches outputs when they're read, not all at once. Find why a member read after 6 minutes idle costs 3–4× a warm one (connection set-up, a full catch-up, the token check) and cut what doesn't need to happen.
- Re-measure on a compiled binary, against an emulator with about 100 ms injected per storage call: requests per write, K2a and K2b, branch against released. Then a short session on real GCS with the dogfood harness.
Scope: cuts to the current write path only, each with a test that counts the work done. If the numbers show the 500 ms K2a bar can't be met on the synchronous path (the commit to serve alone is 0.66 s p50 / 0.82 s p95, including a 250 ms replication confirm that must stay), report the numbers and the options. Local-first writes would need their own design doc, not this issue.
Overlap with #4431 and #4454
- #4431 changes
serve_datastore_client.ts,serve_upstream_binding.ts,serve_lock.ts,v3_datastore.tsandv3_teardown.ts. Step 1 (trace headers on the serve client) and any teardown or lock changes go on top of #4431 once it merges: rebase before touching those files. - Steps 2–5 are mostly elsewhere (
serve_catch_up.ts, the read model, CLI start-up, the copy build) and can start now. - #4454 changes token, lock-release and run-attribution code; little overlap.
Must not regress
- No acknowledged write lost; failover write pause 4–7 s.
- Concurrent writers all visible at first read; catch-ups still apply only new commits (#4387's invariant).
- Revocation refused on the next request on both nodes.
Done when
- Each cut has a test that fails on current main.
- The emulator re-measure shows fewer requests per write, and K2a that stays flat as history and load grow.
- A short real-GCS session records K2a near and far, K2b, CLI start-up, clone size and cold reads. It also re-runs a few correctness checks, since this changes the write path: no lost writes across two failover rounds, concurrent writers (c1), and the outage exit (c09).
- With those numbers, the K2a bar for the announcement is set.
Shipped
Click a lifecycle step above to view its details.
Sign in to post a ripple.