Skip to main content
← Back to list
01Issue
BugShippedSwamp CLIPublic
Assigneesstack72

Relationships

#4457 Team mode: a member write makes one catch-up, its spans join serve's trace, and CLI start-up, fresh clones and cold reads stop costing seconds (group 4 latency from #4374, follows #4401)

Opened by stack72 · 11/5/2026· Shipped 11/6/2026

Found by the team-mode dogfood on real GCS (swamp-club#4374), run 3, release 20261105.030003.0 (contains #4387 and #4401). This is group 4 of the dogfood fixes: member write latency, following #4401. One owner, one PR. It runs in parallel with #4431 (member to serve) and #4454 (team admin); see "Overlap" below.

Where a member write's time goes

The dogfood's per-write breakdown, from 30 near one-step writes after about 3 h of team load (K2a p50 2.28 s, p95 4.93 s; fresh p95 was about 1.7 s):

Phase (member spans) p50 p95
CLI start-up, before swamp.cli ~1.5–1.8 s —
swamp.cli.bootstrap 76 ms 99 ms
Catch-ups from serve (swamp.datastore.v3.serve.catchup) 1,153 ms 3,054 ms
readmodel.catchup 417 ms 971 ms
serve.lock.acquire 123 ms 526 ms
swamp.model.method.run 2,306 ms 2,891 ms
of which serve.forward (the commit) 659 ms 820 ms
of which the rest of method.execute, in no request span (for an echo) ~1.5 s —
serve.lock.release 123 ms 313 ms
swamp.cli.teardown 345 ms 686 ms
Requests to serve per write 12 (5 GET partition commits, 4 GET changes, 1 commit, 1 lock, 1 unlock) max 18

Serve, same window: POST …/commits p50 523 ms, p95 891 ms (confirm_replicated 250 ms of it). GCS put 90 ms, get 39 ms, list 33 ms.

Also measured:

  • K2b: a two-step workflow costs 2.1–2.6× v2, about 8.4 s per step against 3.4 s.
  • F23: a fresh member clone plus data list takes 40–67 s and stores 143–415 MB (v2: 4–6 s, 1.3 MB). It appears to download every stored output, including large ones, instead of fetching outputs when they're read.
  • Cold read: a member read after 6 minutes idle takes 6.8–8.6 s, against 2.0–2.9 s warm.

Work, in order

  1. F25, trace propagation. Of 2,131 serve spans, none shares a member's trace ID, so the breakdown above is matched by time window. Send traceparent on every member-to-serve request, so serve's spans join the member's trace. Do this first: every later step is measured with it.
  2. The unexplained ~1.5 s inside method.execute. Find what it is (an echo step should cost milliseconds), give it a span, and remove or cut it.
  3. One catch-up per command. Today a write makes nine catch-up requests, and their cost grows with load (p95 3.1 s, the main cause of K2a creeping from 1.7 s to 4.9 s). A command should catch up once, then only fetch what its own commit needs. Pin it with a test that counts requests to serve per write, and one showing the count doesn't grow with history or load.
  4. CLI start-up, 1.5–1.8 s on every command. Find where it goes (binary start, the auth gate, the update check, module loading) and cut it. It falls outside K2a, but members feel it on every command.
  5. F23 and cold reads. A fresh clone fetches outputs when they're read, not all at once. Find why a member read after 6 minutes idle costs 3–4× a warm one (connection set-up, a full catch-up, the token check) and cut what doesn't need to happen.
  6. Re-measure on a compiled binary, against an emulator with about 100 ms injected per storage call: requests per write, K2a and K2b, branch against released. Then a short session on real GCS with the dogfood harness.

Scope: cuts to the current write path only, each with a test that counts the work done. If the numbers show the 500 ms K2a bar can't be met on the synchronous path (the commit to serve alone is 0.66 s p50 / 0.82 s p95, including a 250 ms replication confirm that must stay), report the numbers and the options. Local-first writes would need their own design doc, not this issue.

Overlap with #4431 and #4454

  • #4431 changes serve_datastore_client.ts, serve_upstream_binding.ts, serve_lock.ts, v3_datastore.ts and v3_teardown.ts. Step 1 (trace headers on the serve client) and any teardown or lock changes go on top of #4431 once it merges: rebase before touching those files.
  • Steps 2–5 are mostly elsewhere (serve_catch_up.ts, the read model, CLI start-up, the copy build) and can start now.
  • #4454 changes token, lock-release and run-attribution code; little overlap.

Must not regress

  • No acknowledged write lost; failover write pause 4–7 s.
  • Concurrent writers all visible at first read; catch-ups still apply only new commits (#4387's invariant).
  • Revocation refused on the next request on both nodes.

Done when

  • Each cut has a test that fails on current main.
  • The emulator re-measure shows fewer requests per write, and K2a that stays flat as history and load grow.
  • A short real-GCS session records K2a near and far, K2b, CLI start-up, clone size and cold reads. It also re-runs a few correctness checks, since this changes the write path: no lost writes across two failover rounds, concurrent writers (c1), and the outage exit (c09).
  • With those numbers, the K2a bar for the announcement is set.
02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 9 MOREREVIEW+ 21 MOREPR_LINKED+ 2 MORESESSION_SUMMARIZED

Shipped

11/6/2026, 9:40:17 AM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack7211/5/2026, 9:53:12 PM

Sign in to post a ripple.