Skip to main content
← Back to list
01Issue
BugShippedSwamp CLIPublic
Assigneesstack72

Relationships

#4539 Team mode: a commit's cost on serve grows with history

Opened by stack72 · 11/9/2026· Shipped 11/9/2026

Found while measuring swamp-club#4525 (shipped in PR 3684). This is #4525's item 4, "the growth over a set", which the #4374 dogfood saw as K2a p95 roughly doubling over a few hours (F34).

The problem

A team member's commit costs more on serve the longer the datastore's history is. Main and the #4525 build are affected alike, so this is older than #4525. Request counts per command stay flat with history (a write is 2 requests to serve, a two-step workflow 10). The growth is in the work serve does per request.

Lane: scripts/measure_v3_team_round_trips.ts with fake-gcs behind 100 ms per storage call, real @swamp/gcs-datastore and Litestream, and 6 rounds. Two runs differ only in --history (50 or 2,000 seeded commits).

Wall p50, ms (near member unless noted):

history 50 history 2,000 growth
write, main 1,429 1,911 +482
write, #4525 1,410 2,002 +592
two-step workflow, main 6,020 6,640 +10%
two-step workflow, #4525 3,955 4,513 +14%

Span medians over the near write samples, ms:

span main h50 main h2000 #4525 h50 #4525 h2000
member's commit request 366 541 373 538
serve after_commit 9 73 9 89
serve confirm_replicated 351 365 349 357
member method.execute 23 32 22 35

What to fix

  1. Serve's catch-up after each commit grows with history. DatastoreApi.#afterCommit (src/serve/datastore_api.ts) runs serve's read-model catch-up after every forwarded commit. It goes from 9 ms to about 80 ms between 50 and 2,000 commits, though it applies one commit either way (1 commit, 14 files written).
    • Triage found the cause. The read model's catalog_ms goes from 1 to 35 ms and state_ms from 1 to 30 ms.
    • Catalog phase: projectCatalog (src/domain/datastore_v3/catalog_projection.ts) loads every live version of each data name a commit touches through liveVersions (read_model_state.ts), schema-parsing each version's metadata.
    • State phase: recordApplied then deletes the name's version rows and inserts every live version again.
    • So the cost is O(live versions) for a commit that adds one. The full set is needed only to find the greatest version left after a versioned delete (reconcile).
  2. About 90 ms more of serve's commit request is in stages no span covers. Triage ruled out the catch-ups before authorization: they apply nothing and take 0 ms at both sizes. Serve's append takes 0 ms and the replication confirm stays flat. The rest is somewhere in #commitOne's authorization, content check (checkDataPut), proof of possession and forwarded-lock work. Span those stages, find what grows, and remove it.
  3. A work-count test, failing on main, on integration/serve_datastore_api_harness.ts. Seed history with seedHistory(server, n) at n = 50 and n = 2,000. For one forwarded commit, count the read-model work one forwarded commit costs (version rows read and written by the catch-up, plus whatever point 2 finds). Assert that the counts are equal at both sizes. Assert on work, never on wall time.

Done when

  • The work-count test passes, and fails on main.
  • The lane shows write and workflow wall times flat between --history 50 and --history 2000, within noise.

Each lane run takes about 30 minutes. The maintainer holds both runs' samples and a span-median script from #4525's measurement.

02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 5 MOREREVIEW+ 10 MOREPR_LINKED+ 2 MORESESSION_SUMMARIZED

Shipped

11/9/2026, 5:59:10 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack7211/9/2026, 12:18:09 PM

Sign in to post a ripple.