Relationships
#2347 serve dashboard: 2.6 MB uncompressed per page (no permessage-deflate, unpaginated workflow/run search); 15 s WS close stall; serve.config rejected
Opened by randybias · 9/22/2026· Shipped 9/22/2026
Observed (serve 20260922.011324.0, dashboard, 320 workflows, ~200 runs)
The dashboard feels slow for users who are not on the LAN. Server-side time is fine. Payload size is not. Each call was timed over one WebSocket with the dashboard's own request types; bytes are the frame size:
| request | server time | response |
|---|---|---|
workflow.search {} |
257 ms | 825,627 B |
workflow.run.search {} |
278 ms | 1,753,368 B |
| model.search | 27 ms | 22,235 B |
| data.search | 6 ms | 29,031 B |
| workflow.approvals | 243 ms | 109 B |
- The upgrade response negotiates no
permessage-deflate, even when the client offers it. About 2.6 MB of raw JSON crosses the wire per page load, and again on each navigation that re-queries. workflow.searchreturns full definitions for every workflow, but a list view needs name, description and tags.workflow.run.searchhas no default limit.
Separately, the dashboard bundle sends serve.config and cluster.instances, and this server rejects both: Invalid request: type: Invalid discriminator value. Expected 'server.version' | ....
And about 1 in 10 WebSocket connections through ingress-nginx is held for 15.05 s after the response is delivered. The upstream connection shows 2 extra bytes (827148 vs 827146), which suggests the close handshake waits out a 15 s timeout. The CLI sees this as a 15.7 s swamp workflow list --server.
Ask
- Support
permessage-deflateon the serve WebSocket. - Summary projections plus pagination/limits for
workflow.searchandworkflow.run.searchas used by the dashboard. - Check the
serve.config/cluster.instancesschema mismatch between the dashboard and the server. - Check the 15 s close stall.
Shipped
Click a lifecycle step above to view its details.
stack72 commented 9/22/2026, 9:35:25 PM
@randybias thanks for the detailed report and measurements — they made this easy to pin down. The fix shipped in swamp 20260922.213037.0. What changed:
- System view:
serve.configandcluster.instances(anddoctor.datastores) were missing from the server's request schema, so they were rejected before reaching their handlers. They now work (admin-only over WebSocket, matching the REST routes). - Payload size: the dashboard now asks the server to gzip large responses, and the Executions and Workflows views page on the server instead of fetching everything. On a test repo with 320 workflows, the full workflow list dropped from 133 KB to 10 KB on the wire, and run lists from ~89 KB to ~13 KB.
workflow.searchandworkflow.run.searchacceptoffset/limitand return atotal;workflow.run.searchdefaults to 500 results.
Could you re-run your probe against the new release and let us know how the dashboard feels from off the LAN? If anything is still slow, your per-request numbers will tell us where to look next.
randybias commented 9/23/2026, 7:37:35 AM
Re-measured on 20260923.001003.0, with the same probe as before. It connects the way the new dashboard does (/?compress=gzip) and sends the dashboard's own payloads. The repo has 320 workflows and ~200 runs.
| request | before (011324.0), bytes on the wire | after |
|---|---|---|
workflow.search {limit:25, offset:0} (Workflows page) |
825,627 (unpaged) | 24,389 (59,165 JSON) |
workflow.run.search {limit:200} (Workflows page, run stats) |
— | 495,821 (1,538,857 JSON) |
workflow.run.search {limit:500} (overview) |
1,753,368 | 578,337 (1,795,473 JSON) |
serve.config, cluster.instances |
rejected by schema | now reach the handler: Access denied ... 'admin' for a non-admin, as intended |
The Workflows view is fixed, a 34× reduction. The run lists are the remaining weight. gzip gives about 3×, but our run records average about 7.7 KB of JSON each (1,538,857 / 200). So the overview's limit:500 and the Workflows page's limit:200 still move 0.5–0.6 MB per load, and the Workflows page only needs per-workflow counts from them. Two suggestions:
- A summary projection for
workflow.run.search: id, workflowName, status, startedAt/completedAt, stepProgress. Or a server-side aggregate for the per-workflow stats, so that view does not fetch runs at all. - Default the overview to the most recent ~50 runs.
Server time is fine throughout: 250–370 ms per call. The 15 s WebSocket close stall I reported is a separate item; I have not re-tested it yet.
randybias commented 9/23/2026, 7:40:08 AM
Follow-up on the 15 s WebSocket close stall: it is still present on 20260923.001003.0, and more frequent. 20 consecutive swamp workflow list --server wss://... --json calls from the same host, in ms: 1021 15742 1035 993 1040 948 991 15719 1105 15705 15705 15738 983 979 985 15768 1116 15713 15719 1118. That is 8/20 at ~15.7 s, against ~1 in 10 on 20260922.011324.0. The rest take ~1.0 s. The fixed ~15 s says a timeout is being waited out rather than work being done. The earlier evidence still applies: the ingress-nginx access log holds the upstream WebSocket for 15.05 s on the slow ones, with 2 extra bytes sent (827148 vs 827146), after the response was already delivered. That points at the close handshake. Path: ingress-nginx 1.x, with TLS terminated at the ingress and again in the pod (backend-protocol HTTPS).
Sign in to post a ripple.