Skip to main content
← Back to list
01Issue
BugShippedSwamp CLIPublic
Assigneesstack72

Relationships

#4522 Team mode: member writes fail with GCS 429 once a team commits faster than once a second, because serve rewrites the lineage head object on every commit (F38 from #4374)

Opened by stack72 · 11/8/2026· Shipped 11/8/2026

Found by the team-mode dogfood on real GCS (swamp-club#4374), run 5, release 20261108.111402.0-sha.37c27525. A ship blocker: member writes fail under normal team load.

What happens

On every commit it confirms, serve rewrites the same small object in the bucket: the lineage head (_control/lineage/<lineage>/head-<lineage>.json), via publishV3Head (src/infrastructure/datastore_v3/litestream/v3_copy.ts), called from the confirm path (span swamp.datastore.v3.lineage.publish_head, litestream_replicator.ts). GCS allows about one mutation per second per object. Once a team commits faster than that, GCS answers 429 and the write fails.

Run 5 evidence:

  • Part A: 2 of 20 runs failed, and the load loop saw 3 more failures on member-2 and member-far between 12:42 and 12:48 UTC on 2026-11-08. Three members plus one test arm were writing.
  • The member's error: "Serve committed this change at seq 3 but could not confirm it (… the replicated state could not be published: GCS putObject failed HTTP 429 rateLimitExceeded … The object …/_control/lineage/000000000033/head-000000000033.json exceeded the rate limit for object mutation operations …)". The run is recorded as failed, with a resume hint.
  • Serve logs nothing: 0 lines matching 429, rateLimitExceeded or "could not confirm" on either node. Only the member whose write fails knows.
  • Run 4 never saw a 429 at similar load. Faster commits in this release are a likely reason; not confirmed.
  • Member logs: member-2 ~/load/fail/20261108T124354-wf-nightly.json and …124623-wf-nightly.err; member-far ~/load/fail/20261108T124252-wf-nightly.err.

Fix

  1. Coalesce the head publish without adding latency:
    • publish at once when no head has gone out for that object in the last second;
    • otherwise, confirms wait for the next publish, which covers every commit up to its txid.
    • So the bucket sees at most about one head write per second however fast the team commits, and a quiet team pays nothing extra.
    • This extends the way concurrent writes already share a Litestream sync.
  2. Retry 429 and 503 on every bucket write serve makes, not only the head: backoff with jitter, within the confirm's existing deadline. A confirm that still fails at the deadline fails as today, with the same coded error.
  3. Serve logs every rate-limited or failed bucket write, with a code, the object and the attempt count.
  4. Check S3 for the same per-commit rewrite and give it the same treatment. S3's per-object limits differ, but the pattern is the same.
  5. The test emulator enforces a per-object mutation rate (about one a second, answering 429), so the next hot object fails in CI rather than on GCS.

Tests

  • Sustained writes: three members commit back to back, faster than one a second, for at least a minute, against the rate-enforcing emulator. Zero failed writes, and head publishes at most about one a second. Red on current main.
  • No added latency at low rates: a single commit after an idle second publishes its head at once, with no wait for a coalescing window.
  • A 429 then success on the head write: the confirm succeeds, and serve logs the retry.
  • A 429 that persists past the deadline: the same coded error as today, and serve logs it.
  • Takeover: the follower still sees the latest head (including after coalesced publishes), and failover behaves as in runs 4 and 5.

Must not regress

  • No acknowledged write lost. A write is acknowledged only once the published head covers it, exactly as now.
  • Failover write pause 4–7 s, and takeover reads the latest head.
  • F19's fix (sync again until replicated_txid reaches the first answer's txid) is unchanged.

Done when

The tests above pass and fail on current main where marked. Dogfood run 5's soak, or a short sustained-write check on the fixed release, shows zero 429-caused failures.

Note

Further work on the same path is expected: fewer member-to-serve round trips per write (lock, commit and unlock as one request; one catch-up request) to bring K2a and K2b down. It will be filed once run 5's latency numbers are in. Ideally the same owner takes it, since it changes the same commit handler.

02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 9 MOREREVIEW+ 14 MOREPR_LINKED+ 2 MORESESSION_SUMMARIZED

Shipped

11/8/2026, 9:14:55 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack7211/8/2026, 1:50:03 PM

Sign in to post a ripple.