Users running swamp serve on Kubernetes report that swamp serve reload --server and swamp model edit --server timeout with no server-side logs produced. Other --server commands (e.g. model get, model list) work fine.

Environment

Kubernetes with nginx ingress controller
OAuth auth mode, user is an admin
Server started with --hot-reload
Repo volume (/repo) and home directory (/home/[REDACTED]/.swamp) on different volume mounts (separate PVCs)
Ingress proxy timeouts are set to 3600s — not the cause

Investigation Summary

Code analysis

The handleServeReload handler (admin_handlers.ts:1716) and handleModelEdit handler (model_handlers.ts:1369) both have comprehensive try/catch blocks that always send a response frame. Every branch — auth failure, hot-reload disabled, success, error — calls send() or sendError(). The RPC path for serve reload never uses SIGHUP; it calls performServeReload() directly in-process via WebSocket RPC.

What "no logs" tells us

handleServeReload logs "Extension reload requested by ..." immediately after the auth check passes (line 1742). No logs means either:

The request never reaches the handler
The auth check at authorizeOrReject fails — but this always calls sendError() which should produce a client-side error, not a timeout

Ruled out

Ingress proxy timeout — user's nginx ingress has proxy-read-timeout: 3600
Missing --hot-reload — confirmed enabled
Auth/token issues — other --server commands work with the same credentials
Schema validation — serve.reload and model.edit are both in the ServerRequestSchema discriminated union
Worker protocol intercepting messages — RpcChannel.handleRaw returns false for non-rpc.* message types
Local reproduction — operations complete in <1ms locally, even through an nginx reverse proxy with 1s timeout, so could not trigger the timeout

Remaining hypotheses

PVC I/O stalls — if the repo PVC is NFS/EFS-backed, write operations or SQLite file locking (used by serve reload for the extension catalog at .swamp/_extension_catalog.db) could hang indefinitely. model edit writes YAML to the PVC; serve reload opens a SQLite DB on the PVC.
OAuth-specific WebSocket auth latency — authenticateServerToken runs the server-token model's redeem method on every WebSocket connection, which reads data from the PVC. If PVC I/O is slow, auth could take long enough for something else to timeout.
sendError with id: "unknown" — if validateServerRequest fails (line 1297-1299 of connection.ts), sendError is called with id: "unknown" rather than the request's actual ID. The client filters by message.id !== requestId, so this error frame would be silently ignored and the client would timeout. This is a confirmed bug in the error-reporting path, though it shouldn't trigger for well-formed serve.reload / model.edit messages.

Diagnostic steps for the user

Port-forward test — bypass ingress entirely:

kubectl port-forward <pod> 4111:4111
swamp model edit <name> --server ws://127.0.0.1:4111 < definition.yaml

In-pod test — bypass WebSocket/RPC entirely:

kubectl exec -it <pod> -- swamp model edit <name> < definition.yaml

PVC I/O test — check for stale/slow mounts:

kubectl exec -it <pod> -- time dd if=/dev/zero of=/repo/testfile bs=1k count=1
kubectl exec -it <pod> -- rm /repo/testfile

Debug logging — start serve with --log-level debug and retry the failing command; check for WebSocket auth entries and message dispatch logs.

PR #2091 fixes a related EXDEV (cross-device rename) crash in the same Kubernetes environment during migrateGrantDefinitions at serve startup

02Bog Flow

Open

8/7/2026, 2:07:54 AM

No activity in this phase yet.

03Sludge Pulse

git: add pull, fetch --tags, and cherry-pick methods

Skill docs use invalid two-argument 'vault get' form

access token mint prints an invalid retrieval command (vault get, not vault read-secret)

serve reload and model edit timeout over --server in Kubernetes

Show user:<name> instead of user:<id> in grant list output

swamp auth whoami should report paid tier and trial status

Exclude swamp internal model types from model list output

Use whoami entitlement to explain private-extension push refusals

Document --restricted-commands flag for swamp serve

vault put: pass tags through to extension provider CreateSecret/setSecret calls

Docs: update manual for 23 new --server commands

Update VaultProvider interface in @systeminit/swamp-testing to support optional tags in put()

Docs: document --grants-file flag for swamp serve

Docs: document --server support for serve reload

Docs: add --tag flag to vault put reference

Docs: update manual for --detach-runs deprecation and HA auto-detection

Document the Codex plugin in swamp docs

Add --server support to CLI commands missing it

Document the swamp Claude Code plugin in the swamp documentation

Make --server a global CLI option

Add --server support to model cancel (server-side run cancellation)

Load grants from external source on swamp serve startup

swamp vault put should support tags for secret governance

Improve hot reload of the deamon, include extension source/bundle changes

Authenticated users must re-login after swamp serve restart

Add --server support to serve reload

Harden queue claim semantics: atomic claims + status-guarded markDelivered/markFailed across app queues

Auto-detect HA mode and deprecate --detach-runs flag

version-drift check (from #236) is a raw string-inequality, false-positives on multi-model manifests and metadata-only bumps

Resend newsletter audience sync: contacts, segments, backfill, unsubscribe sync-back

Move per-login token secrets from vault to encrypted control-plane store

Docs: document macOS TLS trust limitation and SSL_CERT_FILE workaround

Validate swamp serve deployment stack on startup

Docs: serve config file (.swamp/serve.yaml) and --config flag

Serve config file — read swamp serve configuration from YAML

Deno runtime does not trust ISRG Root YR — extensions fail with valid Let's Encrypt certs

Docs: update hot-reload reference to include trust list refresh

Cross-instance client re-attach for multi-instance swamp serve

vercel codegen: inline rawSyncResult in non-unwrap sync path

vercel codegen: unused listAll import blocks extension publish for 4 models

vercel codegen: add round-trip idempotency test for the pipeline

Operatives button on Collectives page does nothing

Track active runs in the control-plane store for cross-instance visibility

validateGlobalArgInputRefs checks step-level inputs but step schema rejects 'inputs' key

Docs: document continuous reconciliation and multi-instance serve flags

Docs: update workflow guard skip output to include guardExpression and guardResult

Make it aware hot reload on the deamon does not take into account trust list changes, or do take into account the trust list for the hot reload.

workflow validate: new 'GlobalArgument input references' check resolves model-definition ${{ inputs.* }} against workflow inputs, false-failing steps that supply the input

Continuous reconciliation for multi-instance swamp serve

Split the score-read circuit breaker so leaderboard pressure can't blank profile scores

oauth_sign_in telemetry records provider as the literal ":id" for every event

Ghost profile bypasses the score-read seam and 500s on a slow ledger

Profile score section blocks the render on a ClickHouse read

Council: set the scoring balance — command tiers, farm-decay parameters, cross-category ratio

Scoring coverage gaps: instrument drop-off, score ripples/mentions, reward invite-converts

Push down the two cheap leaderboard scans: ghost-gate dictionary and eventCounts owner filter

Materialize an active-days-by-id table for the streak board

score-rollup-projector: O(all-time ledger) closure + per-pod fan-out saturates prod ClickHouse CPU

Startup cache hydration for swamp serve instance replacement

Output a log line for steps skipped due to guards

extension quality scorer misreads a template-literal import specifier in generated code as an unresolvable bare import, hard-erroring the whole extension as UNSCORABLE

Add @swamp/kubernetes/statefulset model type

Docs: access token commands reference says plaintext is 'shown once'

Document --restricted-model-types flag for swamp serve

swamp auth server-login ignores SWAMP_SERVE_URL

s3-datastore extension: pullChanged writes namespace-stripped paths to local cache

serve oauth: an unrecognized device-token error kills the whole serve process instead of the bootstrap

s3-datastore: bidirectional sync pull writes namespace-stripped copies to cache root, wedging push with no supported recovery

CLI reports 'HTTP/2 not supported by this client' when the server actually returned 401 — masks auth failures as a transport error

access token mint: allow reusing a revoked token name

extension quality --json exits 1 with 'rubric factors missing' even when a full score was computed (unearned factor, not uncomputable)

serve oauth: startDeviceGrant discards the OAuth error field, making device-code failures undiagnosable

access token rotate: add --vault flag to override inherited vault

Extensions without a repository can't receive issues — including the issue asking them to add one

serve oauth bootstrap: device flow rate-limited to a 45-second window by validateClient counter, shared globally across tenants

workflow edit: after saving an invalid workflow, re-running edit reports "Workflow not found"

Authentication via Google Broken

Plus-addressed emails stripped on invite acceptance, breaking collective join and quest reward

Profile edit broken on Safari

Configurable write paths for object-creating commands (repo-relative swamp root)