Relationships
#4817 A follower that forwards a v3 datastore chunk upload to its leader may get 411, because the forward drops Content-Length
Opened by zack · 11/16/2026· Shipped 11/17/2026
Found while planning swamp-club#4804; not reproduced end to end.
DatastoreApi.putChunk (src/serve/datastore_api.ts) answers 411 length_required when a PUT chunks/{sha256} has no Content-Length. A follower forwards non-read datastore requests, including chunk PUTs, to the leader through LeaderLink.forward (followerDatastoreRefusal in datastore_api.ts). LeaderLink drops content-length as a hop header (HOP_HEADERS in src/serve/leader_link.ts) and passes req.body, a stream, to fetch.
On Deno 2.9.8, fetch sends a ReadableStream body chunked and never sends Content-Length, with or without Deno.createHttpClient, even when the header is set explicitly. So the leader would see a chunk PUT with no Content-Length and refuse it with 411.
To confirm: a test with a follower and a leader in-process, sending a chunk PUT to the follower and checking the leader accepts it. Possible fixes: forward chunk bodies with a known length (buffered up to the chunk size cap, or over node:http, which streams and keeps Content-Length), or let putChunk accept a chunked upload with a bounded read.
Shipped
Click a lifecycle step above to view its details.
stack72 commented 11/17/2026, 7:11:22 PM
@zack Triage update: this doesn't reproduce with real traffic. My first repro built the incoming request by hand with a stream of unknown length, which is what loses the length. A request that actually reaches a follower's server keeps its length, and Deno's fetch sends it on to the leader even though LeaderLink drops the content-length header. Checked with byte bodies of 16 B, 70 KB and 4 MiB, over HTTP/1 and over HTTP/2 with TLS on both hops: the leader got a Content-Length every time and stored the chunk. The member's client always uploads chunks as bytes, so it always sends a length. Only a caller that itself sends no length goes through without one, and the leader refuses that caller directly anyway. Rather than change the forward, the fix is an integration test that uploads a chunk through a follower and checks that the leader stores it. If a future Deno version stops passing the length through, it fails with the leader's 411.
Sign in to post a ripple.