Relationships
#4558 serve: a vault outage during swamp serve reload removes every working webhook route file
Opened by zack · 11/9/2026· Shipped 11/9/2026
Problem
Webhook route files (swamp-club#4547) are parsed one file at a time on every startup and swamp serve reload, so a bad file is refused alone. Parsing a route file includes resolving its @vault= secret. If the vault cannot be read for a moment during a reload (backend timeout, transient error), every route file fails to parse and is refused. The reload then calls WebhookService.updateEndpoints with only the operator routes, so every previously working file route is removed and its POSTs stop matching until the next successful reload. The reload still reports success; the refusals appear only as reload errors.
Operator routes behave differently: serve.yaml webhooks parse all-or-nothing, so a vault failure fails the webhook update and the live routes stay as they were.
Steps to reproduce
- Serve with one or more working route files in webhook-routes/ (and no serve.yaml webhooks, or flag mode).
- Make the vault named by their secrets unreadable (stop the backend, or make reads time out).
- Run swamp serve reload.
- Every file route is reported refused and removed; POSTs to those routes return 404 until the vault recovers and serve reloads again.
Expected
A failure that is about the environment rather than the file (vault unreachable, a transient scheme-resolution failure) keeps the route's previous endpoint, while a failure that is about the file itself (validation, a missing key in a reachable vault) still refuses it. At minimum the behaviour is documented.
Notes
- Code: buildWebhookRouteTable and parseRouteFiles in src/serve/webhook_reload.ts and src/serve/webhook_route_files.ts; parseWebhookConfigs in src/cli/commands/serve.ts.
- Keeping the previous endpoint needs WebhookService to expose the current endpoint for a route (with its resolved secret) to the table builder.
- Raised by the code and adversarial reviews during swamp-club#4547 verification.
Shipped
Click a lifecycle step above to view its details.
Sign in to post a ripple.