← Back to list
01Issue
BugShippedSwamp CLIPublic
Assigneesstack72
Relationships
#4898 Team mode: a far member's large-output save takes about 2.5x v2, all in the single-connection upload to serve
Opened by stack72 · 11/19/2026· Shipped 11/19/2026
Problem
A team member far from serve (europe-west1 against us-central1 serves) saving large outputs takes about 2.5× as long as v2 does on the same member.
Evidence (M, dogfood, 2026-11-16)
Workflow big-outputs (17 MiB + 64 MiB outputs), member-far, 3 reps per arm, same day:
| Arm | Wall | Inside |
|---|---|---|
| v2 (uploads straight to GCS) | 25.9 / 32.1 / 45.6 s | — |
| v3 team mode | 84.3 / 88.4 / 112.4 s | serve.forward 53.6–58.1 s; store_content ~3.5 s; chunks_durable ~1.0 s |
Inside serve.forward for the 64 MiB commit (rep 1, client 44.7 s):
- member → serve transfer, about 37–40 s: serve's
swamp.serve.datastore.commit.ingestran 39.9 s, reading the body off the wire at about 1.7 MiB/s on one connection; - serve's own work, about 4.5 s (after_commit 3.7 s);
- serve → GCS, about 3 s, overlapped with the transfer.
The exit after large outputs is already fast (≤ 0.2 s); this issue is only the upload.
Candidate causes, decided by measurement
- (A) One TCP stream is window-limited across the Atlantic: several connections would multiply throughput. A client-side change using existing routes (parallel chunk PUTs on separate HTTP clients, then the commit).
- (B) The link itself is the limit: only members uploading chunks straight to the bucket could approach v2. That's architectural (credentials, authorization, GC of orphaned chunks) and needs design review.
- (C) swamp's own upload is slow: for example how the body is streamed, or Deno client or TLS settings. Then a plain transfer on the same path is clearly faster than 1.7 MiB/s.
Scope (timeboxed, about one day)
- First, the probe (already written: farupload.sh on dogfood-ctl, about 15 min of VM time): 64 MiB from member-far to serve as 4 × 16 MiB one after another on one connection, 4 at once on separate connections, and 4 multiplexed on one HTTP/2 connection, 3 reps. Compare with swamp's 1.7 MiB/s. The result picks A, B or C.
- If A or C: build that fix with tests, one full verify, then one VM run of big-outputs from member-far, v2 against the build, 3 reps each.
- If B: stop and report; that's a design discussion.
- At most one fix round, only for a miss against the done criteria. Review suggestions are listed in the PR, not folded.
Done when
- big-outputs from member-far is ≤ 1.5× v2 on the same day (median of 3).
- No regression near (member-1) on the same workflow.
- quick.sh safety unchanged: 0 lost, 0 diverged, 0 failed recoveries.
02Bog Flow
Shipped
Click a lifecycle step above to view its details.
03Sludge Pulse
Sign in to post a ripple.