Skip to main content
← Back to list
01Issue
BugShippedSwamp CLIPublicTeam
Assigneesstack72

Relationships

#1843 worker connect fails against any h2-speaking TLS server (client offers h2 ALPN but WS layer is HTTP/1.1 only)

Opened by bixu · 8/26/2026· Shipped 8/26/2026

Symptom

swamp worker connect wss://<serve>:<port> fails to enroll against any TLS-terminated endpoint that advertises h2 in ALPN. Repro on both:

  • wss://[HOST-2] (Traefik-fronted, k3s)
  • wss://swamp.swamp.svc.[HOST-1]:9090 (dialing swamp serve directly, no ingress in the path)

Worker log:

Connecting to wss://swamp.swamp.svc.[HOST-1]:9090 (attempt 1)...
Disconnected: socket closed before enrollment: NetworkError: failed to
connect to WebSocket: HTTP/2 not supported by this client
Reconnecting in 1s
...
Error: connection failed 3 consecutive times before enrollment — giving up
(last error: socket closed before enrollment: NetworkError: failed to
connect to WebSocket: HTTP/2 not supported by this client)

Both endpoints negotiate h2 via ALPN. The WS client's own error message calls it out: HTTP/2 is not supported by this client. But the client offers ALPN protocols that include h2, so any TLS server that speaks h2 picks it and the handshake ends there — WS upgrade never runs.

Impact

Any deployment that terminates TLS in front of swamp serve with a stack that offers h2 (Traefik, Envoy, NGINX ingress, many CDNs) cannot run workers. And swamp serve itself offers h2 via ALPN in its own TLS listener, so bypassing the ingress and dialing the Service DNS directly doesn't help. There is no route today from a swamp worker connect to a real deployment.

Expected

One of, ranked by lowest-effort-that-unblocks-me first:

  1. Client-side fix: swamp worker connect (and every other WS client in the swamp CLI) should not offer h2 in its ALPN list until the WS layer supports HTTP/2 Extended CONNECT (RFC 8441). Offering http/1.1 only forces every h2-capable server to fall back cleanly.
  2. Client escape hatch: --http1 / --alpn http/1.1 on swamp worker connect and swamp workflow run --server (for people running against a deployment that must speak h2 for other reasons).
  3. Extended CONNECT support in the WS client (RFC 8441). Real long-term fix. Larger change.
  4. Server-side switch on swamp serve: --no-http2 / --alpn-protocols http/1.1. Useful for deployments that need to run workers today, orthogonal to the client fix.

Item (1) alone would unblock every current caller. (2) is a nice belt-and-braces. (4) is the platform escape hatch when the client is not under my control.

Environment

  • swamp version: 20260826.004253.0-sha.b64b034d
  • Both worker and serve run the same swamp image (swampclub/swamp:<same tag>).
  • k3s v1.31, Traefik v3 (default k3s bundle).
  • Serve config: --config /config/serve.yaml, TLS via cert-manager-issued Let's Encrypt cert.

Notes

Filed on behalf of /, which needs remote workers to un-suspend its swamp serve deployment (dashboard reports SUSPENDED / Workers: 0 without them). Currently blocked; StatefulSet for worker pool is deployed but crashlooping on this error.

02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 4 MOREREVIEW+ 7 MOREPR_MERGED+ 2 MORESESSION_SUMMARIZED

Shipped

8/26/2026, 6:13:45 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
bixu moved this from the private collective hivemq to public9/8/2026, 3:53:23 PM
stack72 assigned stack728/26/2026, 4:13:24 PM
Editable. Press Enter to edit.

bixu commented 8/26/2026, 4:13:12 PM

🤖

Reproduced against a non-worker CLI entry point, so the bug is in the shared WebSocket client, not swamp worker connect code:

$ swamp workflow history search --server wss://<host> --log-level=info --json
{
  "error": "Could not connect to wss://<host>/: NetworkError: failed to connect to WebSocket: HTTP/2 not supported by this client"
}

Same error text, same failure mode. Any --server invocation against an h2-speaking TLS server is broken — mint via --server, workflow run --server, workflow history search --server, etc. Widens impact past "workers can't enroll" to "remote CLI usage against a real deployment is unusable".

Also confirmed no config workaround: no --http1 / --insecure / --alpn on any client entry point, no HTTP/2 disable knob on swamp serve, no SWAMP_* env vars for TLS/ALPN. Verified against swamp help serve/worker connect/workflow at 20260826.004253.0-sha.b64b034d.

bixu commented 8/26/2026, 4:40:20 PM

🤖

Narrowed further with a same-pod, same-image, same-network comparison — retract the earlier "any --server invocation" framing:

$ kubectl exec deploy/swamp -- swamp workflow history search --server wss://<host> --json
{"error":"Authentication failed — run: swamp auth server-login --server wss://<host>/"}

$ kubectl exec deploy/swamp -- swamp worker connect wss://<host> --token fake.token
... HTTP/2 not supported by this client

--server path reaches auth (TLS + WS upgrade succeed). worker connect fails at WS upgrade with the h2 error. Same binary, same TLS chain, same server endpoint — the only difference is the client code path.

Reads like the Deno.createHttpClient({ http2: false }) fix in #2209 didn't reach worker connect's WS client. That path presumably still uses a default h2-enabled HTTP client, negotiates h2 via ALPN with the server, and the WS upgrade fails because tokio-tungstenite/deno-ws can't do Extended CONNECT.

Suggested fix: same http2: false treatment on the enrollment-connect client. Then --ca-cert / SWAMP_CA_CERT on worker connect for the eventual self-signed-cert case (currently only on the --server commands).

stack72 commented 8/26/2026, 6:14:26 PM

Thanks @bixu for reporting this! The fix has been merged and a release is on its way. We appreciate your contribution to swamp.

bixu commented 8/27/2026, 8:17:58 AM

Symptom

swamp worker connect wss://<host> still fails to enroll on the release that ships #2268 (20260826.175814.0-sha.0a537cad). The mask is gone, but the underlying WebSocket upgrade now fails against any TLS-terminated endpoint that offers h2 in ALPN:

Connecting to wss://<host> (attempt 1)...
Disconnected: socket closed before enrollment: NetworkError:
  failed to connect to WebSocket:
  stream error received: unspecific protocol error detected

Same failure against both Traefik-fronted and direct in-cluster paths — anything that negotiates h2 breaks the WS upgrade because the swamp WS client does not implement RFC 8441 Extended CONNECT.

What changed

#2268 dropped { http2: false } from wsHttpClient:

-const wsHttpClient = Deno.createHttpClient({ http2: false });
+const wsHttpClient = Deno.createHttpClient({});

The mask that flag caused was misleading, but the flag itself was doing real work — forcing HTTP/1.1 ALPN so the WS upgrade never sees h2. Removing it re-exposes the underlying h2-vs-WS problem that Deno's WS layer can't handle, just with a different error string.

Suggested direction

Keep http2: false on wsHttpClient (client-side ALPN of http/1.1 only) AND keep the new diagnoseTlsMessage() string-matching diagnostic. Those are orthogonal fixes:

  • http2: false avoids the h2 upgrade failure on Traefik/nginx/Caddy front-ends.
  • diagnoseTlsMessage() surfaces the real TLS cause when it fires.

Removing the flag traded one broken deployment scenario (self-signed w/ CA:TRUE, expired) for a much larger one (any h2-capable ingress).

Environment

  • Image: swampclub/swamp:20260826.175814.0-sha.0a537cad
  • Serve behind Traefik v3 (k3s default), LE cert, websecure entrypoint (h2 on)
  • Also reproduces dialing the in-cluster Service directly, so this isn't Traefik-specific

Reopening / filing against #1843 since it's the same root behavior even though the error string is different.

bixu commented 8/27/2026, 9:49:46 AM

🤖

More diagnosis: Deno.createHttpClient({ http2: false }) does not affect ALPN for new WebSocket(url, { client }). Verified with a minimal repro on Deno 2.9.5 against a Traefik-fronted wss endpoint:

--- no client (default) ---
  error: HTTP/2 not supported by this client   (Deno's mask)
--- client with http2:false ---
  error: stream error received: unspecific protocol error detected
--- client with default ---
  error: stream error received: unspecific protocol error detected

client with http2:false still triggers the real h2 error, meaning the WS layer negotiated h2 in ALPN regardless of the client's http2: false setting. That flag only controls fetch()'s HTTP version, not WebSocket ALPN. The deno-issues/16923 comment in remote_run.ts referenced fetch behavior, not WebSocket, and was misapplied when the flag was added to wsHttpClient.

Consequence: neither the pre-#2268 code (mask + http2: false) nor the post-#2268 code (no flag, no mask) nor my fork (flag restored) is a working WS client against any Traefik/nginx/Caddy that offers h2 in ALPN — the whole cluster of workarounds is cosmetic.

The real fix is one of:

  1. Deno adds a per-WebSocket ALPN opt-out (upstream Deno bug, best long-term)
  2. Swamp reimplements the WS handshake on Deno.connectTls({ alpnProtocols: ["http/1.1"] }) + hand-rolled 101-Upgrade
  3. Deployment-side: run swamp behind an h1.1-only proxy sidecar

Attaching my repro as guidance for whoever picks this up. Happy to send a PR against (2) if a maintainer wants to sketch the shape.

bixu commented 8/27/2026, 10:00:15 AM

🤖

Workaround identified for anyone hitting this in the meantime: put a Caddy sidecar (or any h1.1-only proxy) between the worker and the wss:// serve endpoint. Caddy terminates TLS from serve/ingress and re-exposes plain ws:// on 127.0.0.1; the worker dials that plain ws:// and never touches TLS ALPN, so the Deno WebSocket bug never fires. Caddyfile is about five lines:

:8080 {
  reverse_proxy https://<serve-host> {
    transport http { tls_insecure_skip_verify; versions h1 }
  }
}

Not a fix, just a way to unblock deployments while the real work happens on the client side. Filing a Deno-side issue against denoland/deno too so their WS ALPN handling is on the tracker.

Sign in to post a ripple.