Relationships
#4542 Team mode: a large output (64 MiB) takes about 8 minutes to store and usually loses its lease mid-upload, losing the output (F35 from #4374)
Opened by stack72 · 11/9/2026· Shipped 11/9/2026
Found by the team-mode dogfood on real GCS (swamp-club#4374), runs 4 and 5. A ship blocker: in normal use, a step that produces a large output usually loses it, and even when it succeeds it takes minutes.
What happens
The dogfood's m_big check is one workflow writing a 17 MiB and then a 64 MiB output from a team member (writing through serve, no storage credentials). Another member reads both back.
Run 5, release 20261108.111402.0-sha.37c27525 (F36, the exporter fd exhaustion, fixed by #4511):
- 3 of 4 valid runs lost the lease mid-upload, and the 64 MiB output was lost.
- The one run that passed took 473 s for 64 MiB, about 140 KB/s.
- 2 further runs were void because of a harness bug, not swamp.
Run 4, release 20261106.094050.0-sha.fcdd7e60: 2 of 8 runs lost the output, both in set 2; sets 1 and 3 passed in 65–77 s. Timeline of one failure (member-1, serve-b leading):
| UTC 2026-11-06 | Where | Event |
|---|---|---|
| 14:16:45 | serve-b | member-1 opens a WebSocket; step 1 (17 MiB) commits |
| 14:17:14 | serve-b | member-1 opens a new WebSocket for the 64 MiB step |
| 14:17:18 | serve-a | WebSocket error: "Failed to send frame" (member-1's earlier connection) |
| 14:17:45 | serve-b | WebSocket error: "No response from ping frame." |
| 14:17:58 | serve-b | Lease on "data/command/shell/…/.lock" held for "user:alice" expired without a renewal; releasing it |
| ~14🔞06 | member-1 | exit 1, datastore_v3_serve_unreachable "(the lease on … lapsed unrenewed), so the datastore was not changed …" |
- For 30–45 s the member sent no lease renewals and answered no pings, while serve was up and leading throughout.
- It wasn't F36: run 4 was assumed to be the exporter's descriptor exhaustion, but run 5 has that fixed, and the failure rate got worse.
- The exit code is by design: exit 1, because step 1 had landed (#4431's rule). But the error names serve as unreachable when the real cause is the member's own lapsed lease.
Evidence:
- run 4: member-1
~/biglogs/20261106T141806-set2-start.errand…162808-set2-end.err, plus the serve-a and serve-b journals at those times; - run 5: the dogfood's results and end report (
~/dogfood/results/on the controller).
Questions to answer first
Where do 473 s go for 64 MiB? Break one upload down with spans: chunking and hashing on the member, chunk-presence checks, chunk PUTs to serve (one at a time or in parallel), serve's GCS writes per chunk, the commit, the confirm. About 140 KB/s points at per-chunk round trips done one after another, or a synchronous step per chunk.
Why do renewals and pings stop during the upload? Candidates:
- the member's event loop blocked by synchronous hashing, compression or serialisation of large buffers;
- renewals queued behind the upload on the same connection or worker;
- serve too busy with the upload to answer.
Since #4431 and #4525, renewals have their own kept client with a deadline under the renewal interval, so a starved event loop is the likeliest. Measure event-loop delay during an upload.
At what output size does it start? Try 4, 16, 32 and 64 MiB, near and far, so we know who it affects.
Check on the latest release first. #4525 (PR #3684) changed lock clients and chunk checks per step, and run 5 predates it.
Fix direction (once the cause is known)
- Upload throughput: chunks go to serve in parallel, bounded (or batched), with presence checks batched, so 64 MiB near takes seconds, not minutes.
- Liveness during big outputs: no synchronous work long enough to delay a renewal or a pong. Hash and chunk in slices that yield, or off the main thread. Renewals keep their own client.
- The error: a lease lost because the member couldn't renew reports
datastore_v3_lease_lapsed(with the size and phase it was in), notserve_unreachable. The exit rule stays as it is.
Tests
- A slow-bucket lane or integration test: a 64 MiB output with 100 ms per storage call. No lease lapse, and the upload finishes within a fixed bound in requests (not wall time). Red on current main if the cause reproduces locally.
- An event-loop delay check during a large output: below the renewal interval's margin.
- A forced lapse during an upload reports
datastore_v3_lease_lapsed.
Must not regress
- No acknowledged write lost. An output that can't be confirmed isn't reported as stored.
- #4431's exit contract.
- #4525's kept lock client, one per process, separate from uploads.
Done when
- Locally: a 64 MiB output, near and far latency, succeeds 10 of 10 with no lapse, in seconds-scale time.
- On the dogfood:
m_big10 of 10, with both outputs read back by another member, and the 64 MiB step's time recorded.
Shipped
Click a lifecycle step above to view its details.
Sign in to post a ripple.