#4439 swamp serve: apply config changes without failed requests (full reload, vault-backed webhook secrets, harmless restarts)

Problem

On a long-running swamp serve, most configuration changes need a restart. A restart:

• interrupts in-flight runs, or waits out the drain and then kills them;
• answers 503 to webhooks and --server clients until serve is back. That's 30–45 s normally, and minutes on the first start of a new version.

So every config change is a possible outage. Today, with --hot-reload and swamp serve reload:

Where it bites

ops.swamp-club.com is moving its serve config into git, applied by CI on merge. Changes that happen routinely become restarts:

• adding or removing a user;
• adding a webhook;
• rotating a webhook secret.

The obvious workaround, bundling changes and restarting off-hours, is the downtime-prone process we want to stop.

Ask

Make serve config changes cause no failed requests, and make restarts rare and harmless. Two complementary parts:

1. Reload everything that can be reloaded. swamp serve reload re-reads the whole config file and applies it in place:
   • auth: admit new users; revoke removed users' sessions and tokens at once.
   • Webhook secrets: allow a vault reference (e.g. secret: '@vault=<vault>/<key>'), resolved at start and again on every reload, so adding or rotating one needs no restart. @env= and @file= re-read on reload too.
   • trusted-hosts, trust-proxy, dashboard, max-concurrent-scheduled-runs, audit sinks: apply in place.
   • TLS: re-read the certificate and key on reload, for rotation.
   • Report what was applied and what wasn't, and why. A setting that genuinely can't change in place (e.g. port) should be named, never silently ignored.
2. Restarts without failed requests, and interrupted runs resume. Some changes still need a new process (listen address, a new swamp version, a changed process environment). For those:
   • No failed requests. Hand the listening socket to the new process (systemd socket activation, SO_REUSEPORT, or an explicit handover), so connections queue briefly instead of being refused. On a team serve (v3: followers, lease takeover), restart one host at a time behind the load balancer.
   • Interrupted runs resume, not fail. A run interrupted by the restart is resumed automatically from its last completed step when its workflow allows it (autoResume), instead of being left interrupted for an operator.
   • Steps that can't safely repeat say so in the definition, and are reported, never silently re-run.
   • Running steps can't truly survive a process restart: a 50-minute sync is in the old process. So this part aims to make restarts harmless and rare, not invisible.

Done when

• Changing users, webhooks and their secrets, grants, TLS certificates, or any other serve.yaml setting that can be applied in place needs no restart, and no request fails.
• When a restart is needed, no request fails, and every interrupted run whose workflow allows it resumes by itself.
• On a team serve (two or more hosts), upgrading the swamp version causes no failed requests either.