Skip to main content
← Back to list
01Issue
BugShippedSwamp CLIPublic
Assigneesstack72

Relationships

#4898 Team mode: a far member's large-output save takes about 2.5x v2, all in the single-connection upload to serve

Opened by stack72 · 11/19/2026· Shipped 11/19/2026

Problem

A team member far from serve (europe-west1 against us-central1 serves) saving large outputs takes about 2.5× as long as v2 does on the same member.

Evidence (M, dogfood, 2026-11-16)

Workflow big-outputs (17 MiB + 64 MiB outputs), member-far, 3 reps per arm, same day:

Arm Wall Inside
v2 (uploads straight to GCS) 25.9 / 32.1 / 45.6 s —
v3 team mode 84.3 / 88.4 / 112.4 s serve.forward 53.6–58.1 s; store_content ~3.5 s; chunks_durable ~1.0 s

Inside serve.forward for the 64 MiB commit (rep 1, client 44.7 s):

  • member → serve transfer, about 37–40 s: serve's swamp.serve.datastore.commit.ingest ran 39.9 s, reading the body off the wire at about 1.7 MiB/s on one connection;
  • serve's own work, about 4.5 s (after_commit 3.7 s);
  • serve → GCS, about 3 s, overlapped with the transfer.

The exit after large outputs is already fast (≤ 0.2 s); this issue is only the upload.

Candidate causes, decided by measurement

  • (A) One TCP stream is window-limited across the Atlantic: several connections would multiply throughput. A client-side change using existing routes (parallel chunk PUTs on separate HTTP clients, then the commit).
  • (B) The link itself is the limit: only members uploading chunks straight to the bucket could approach v2. That's architectural (credentials, authorization, GC of orphaned chunks) and needs design review.
  • (C) swamp's own upload is slow: for example how the body is streamed, or Deno client or TLS settings. Then a plain transfer on the same path is clearly faster than 1.7 MiB/s.

Scope (timeboxed, about one day)

  1. First, the probe (already written: farupload.sh on dogfood-ctl, about 15 min of VM time): 64 MiB from member-far to serve as 4 × 16 MiB one after another on one connection, 4 at once on separate connections, and 4 multiplexed on one HTTP/2 connection, 3 reps. Compare with swamp's 1.7 MiB/s. The result picks A, B or C.
  2. If A or C: build that fix with tests, one full verify, then one VM run of big-outputs from member-far, v2 against the build, 3 reps each.
  3. If B: stop and report; that's a design discussion.
  4. At most one fix round, only for a miss against the done criteria. Review suggestions are listed in the PR, not folded.

Done when

  • big-outputs from member-far is ≤ 1.5× v2 on the same day (median of 3).
  • No regression near (member-1) on the same workflow.
  • quick.sh safety unchanged: 0 lost, 0 diverged, 0 failed recoveries.
02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 5 MOREREVIEW+ 17 MOREPR_LINKED+ 2 MORESESSION_SUMMARIZED

Shipped

11/19/2026, 10:15:51 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack7211/19/2026, 4:52:27 PM

Sign in to post a ripple.