Relationships
#3399 serve: the per-IP WebSocket burst limit can't be configured, so workers behind one NAT IP throttle each other
Opened by swamp_lord · 10/13/2026
Problem
swamp serve rate-limits WebSocket connections per source IP ("WebSocket IP burst rate-limited from "). There is no flag or serve.yaml setting for it: no limit, no window, no exempt CIDRs, and authenticated workers aren't treated differently. --trust-proxy only changes which IP gets counted.
Many workers can share one source IP. Cloud NAT, a corporate egress, or a Kubernetes node all do this. When they do, serve treats them as one client and throttles them all together, even though every connection presents a valid worker token.
What we saw (ops.swamp-club.com)
- Serve 20261013.004531.0-sha.1373120f; workers 20260928.201643.0-sha.6a1df41a.
- One UAT run used 34 GCE worker VMs. All of them reached serve through the public load balancer and left through one Cloud NAT IP.
- Between 19:27 and 20:56 UTC, serve logged 11,977
WebSocket IP burst rate-limitedlines, all from that IP. The rate climbed to about 2,300 per 10 minutes as the run went on. - In the same window serve logged
Authenticated token "uat-worker"3,982 times, about 117 per worker. The workers were connecting and reconnecting, and the limiter kept refusing them. - Effect on one shard:
- its dispatch reached the worker about 7 minutes after the step started (enrolled 19:30:28, dispatch received 19:37:17);
- the worker finished at 19:42:48, but serve recorded the result at 19:54:50, 12 minutes later.
- Shards then failed with
The operation was aborted due to timeout. A suite that takes 10–15 minutes on GitHub runners was still unfinished after 90 minutes. - Nothing in the run's output said it was being rate-limited. The only sign was serve's own journal.
Our workaround: workers now connect to serve inside the VPC, so each has its own source IP. That isn't possible on every network.
Proposed
Any of these would help, and they can be combined:
- Make the burst limit configurable in
serve.yamland with flags: the limit, the window, and a list of trusted or exempt CIDRs. - Key the limit on the credential, not the IP, once a connection authenticates. A valid worker token shouldn't share a budget with every other client on the same IP. #1615 did this for the auth-failure limiter.
- Tell the client it's being rate-limited. Close the socket with a clear code or
Retry-After. The worker should log it and back off, instead of reconnecting in a tight loop and making the burst worse. - Report it to operators. Expose a rate-limited count per IP in metrics or
swamp worker list, so it shows up without reading the journal.
Related: #1615 (the auth limiter keyed by IP blocked users sharing an IP; fixed for auth failures, not for this connection burst limit).
Open
No activity in this phase yet.
bixu commented 10/14/2026, 7:11:59 AM
🤖
A second case, from HiveMQ: interactive CLI clients, not workers.
- Serve 20261013.175608 runs behind Traefik on k3s, with
trust-proxyon.- Laptops reach it through a Tailscale subnet router. kube-proxy SNAT then rewrites the source to the node IP.
- So every operator, agent and dashboard client shares one address, in
X-Forwarded-Fortoo.
- Each
swampCLI call opens its own WebSocket. One dashboard refresh ran about 15 calls at once.- With a few operators and agents online, the shared budget of 50 per 60 s runs out.
- Serve rejected about 1,100 connections from one 10.42.x.0 address in 2 hours.
- The client sees only
Rate-limited by server — try again later.- A human gate decision that lands during a burst gets rejected with the rest.
Every one of these connections carries a valid token. Option 2, keying the limit on the credential after auth, would fix our case. Option 1 alone would help as a stopgap.
Sign in to post a ripple.