Relationships
#4539 Team mode: a commit's cost on serve grows with history
Opened by stack72 · 11/9/2026· Shipped 11/9/2026
Found while measuring swamp-club#4525 (shipped in PR 3684). This is #4525's item 4, "the growth over a set", which the #4374 dogfood saw as K2a p95 roughly doubling over a few hours (F34).
The problem
A team member's commit costs more on serve the longer the datastore's history is. Main and the #4525 build are affected alike, so this is older than #4525. Request counts per command stay flat with history (a write is 2 requests to serve, a two-step workflow 10). The growth is in the work serve does per request.
Lane: scripts/measure_v3_team_round_trips.ts with fake-gcs behind 100 ms per storage call, real @swamp/gcs-datastore and Litestream, and 6 rounds. Two runs differ only in --history (50 or 2,000 seeded commits).
Wall p50, ms (near member unless noted):
| history 50 | history 2,000 | growth | |
|---|---|---|---|
| write, main | 1,429 | 1,911 | +482 |
| write, #4525 | 1,410 | 2,002 | +592 |
| two-step workflow, main | 6,020 | 6,640 | +10% |
| two-step workflow, #4525 | 3,955 | 4,513 | +14% |
Span medians over the near write samples, ms:
| span | main h50 | main h2000 | #4525 h50 | #4525 h2000 |
|---|---|---|---|---|
| member's commit request | 366 | 541 | 373 | 538 |
serve after_commit |
9 | 73 | 9 | 89 |
serve confirm_replicated |
351 | 365 | 349 | 357 |
member method.execute |
23 | 32 | 22 | 35 |
What to fix
- Serve's catch-up after each commit grows with history.
DatastoreApi.#afterCommit(src/serve/datastore_api.ts) runs serve's read-model catch-up after every forwarded commit. It goes from 9 ms to about 80 ms between 50 and 2,000 commits, though it applies one commit either way (1 commit, 14 files written).- Triage found the cause. The read model's
catalog_msgoes from 1 to 35 ms andstate_msfrom 1 to 30 ms. - Catalog phase:
projectCatalog(src/domain/datastore_v3/catalog_projection.ts) loads every live version of each data name a commit touches throughliveVersions(read_model_state.ts), schema-parsing each version's metadata. - State phase:
recordAppliedthen deletes the name's version rows and inserts every live version again. - So the cost is O(live versions) for a commit that adds one. The full set is needed only to find the greatest version left after a versioned delete (
reconcile).
- Triage found the cause. The read model's
- About 90 ms more of serve's commit request is in stages no span covers. Triage ruled out the catch-ups before authorization: they apply nothing and take 0 ms at both sizes. Serve's append takes 0 ms and the replication confirm stays flat. The rest is somewhere in
#commitOne's authorization, content check (checkDataPut), proof of possession and forwarded-lock work. Span those stages, find what grows, and remove it. - A work-count test, failing on main, on
integration/serve_datastore_api_harness.ts. Seed history withseedHistory(server, n)at n = 50 and n = 2,000. For one forwarded commit, count the read-model work one forwarded commit costs (version rows read and written by the catch-up, plus whatever point 2 finds). Assert that the counts are equal at both sizes. Assert on work, never on wall time.
Done when
- The work-count test passes, and fails on main.
- The lane shows write and workflow wall times flat between
--history 50and--history 2000, within noise.
Each lane run takes about 30 minutes. The maintainer holds both runs' samples and a span-median script from #4525's measurement.
Shipped
Click a lifecycle step above to view its details.
Sign in to post a ripple.