Relationships
#4680 Serve won't release a worker token's secret to a dispatched step (reserved_secret), and this isn't documented
Opened by ehazlett · 11/12/2026· Shipped 11/12/2026
Description
A worker token minted through serve into a user vault cannot be read by any
dispatched step, even in a run triggered by an admin. Serve refuses with
reserved_secret (HTTP 403). An ordinary secret in the same vault, read by a
step on the same worker in the same run, resolves fine.
The serve guide documents one reservation: reserved _ vaults are refused to
every run except admins. This is a different rule. The vault is a user vault,
the run is an admin's, and the refusal is keyed to the secret being a worker
token's. It isn't mentioned in swamp help or in the serve docs. The docs also
present worker token create --vault as the way to keep a token's secret
somewhere other than _token-secrets, which suggests workflows should be able
to read it.
Keeping enrollment credentials away from workflow steps is a reasonable rule. The problem is that it's undocumented, and that there is no supported way to do what this path looked designed for: delivering a successor token to a connected worker before its current token expires.
Steps to reproduce
Run
swamp servewith one worker connected asweb-01. Run the steps below as an admin.Mint a successor into a user vault through serve:
swamp worker token create web-01-r2 --duration 4h --vault fleet --server wss://…swamp worker token list --jsonreports it withvaultName: fleet, secretKey: worker-token-web-01-r2.Store an ordinary secret in the same vault:
swamp vault put fleet plain-secret.Run a workflow with two
command/shellsteps placed on that worker (target: web-01,allowFailure: true). Each passes a secret inenvand runstest -n "$V":control-plain-secret:V: ${{ vault.get(fleet, plain-secret) }}worker-token-secret:V: ${{ vault.get(fleet, worker-token-web-01-r2) }}
control-plain-secretsucceeds.worker-token-secretfails before it runs:Serve refused to load secret 'worker-token-web-01-r2' from vault 'fleet' for this dispatch (reserved_secret, HTTP 403)@swamp/workflow-summaryreports the failed step's detail as only "No data output."; the refusal appears in the run JSON's step error.
Expected
Either the step gets the secret, or the reservation is documented where
worker token create --vault is described, along with what --vault is for.
Why it matters
Worker tokens are a hard deadline: a host whose token expires leaves the fleet
and has to be bootstrapped again over SSH. Rotating ahead of expiry means
getting the new token onto the host over the connection it already has. A
dispatched step is the only way to put a file on a worker, and vault.get is
the only way a step gets a secret. With this refusal, no workflow can rotate a
worker's token. The only alternative is SSH-ing to every host for every
rotation.
Copying the secret to a vault key serve doesn't know about would get past the check, but that would defeat the protection, so we haven't done it.
Ask
A supported way to deliver a successor token to the worker it's meant for. For example:
- a
worker tokencommand that rotates a connected worker over its own connection (mint, deliver, cut over); or - a grant that lets a token's secret be dispatched only to the worker enrolled
under its predecessor (
web-01-r2only toweb-01).
At minimum, document the reservation and what --vault is for.
Related: a worker's name is its token's name, so every rotation renames the
worker (web-01 → web-01-r2). That's worth keeping in mind for whatever
rotation path you design.
Version
swamp 20261112.171308.0 (latest at time of filing). Also seen on 20261109.005252.0.
Shipped
Click a lifecycle step above to view its details.
zack commented 11/12/2026, 8:34:31 PM
Thanks for the thorough report and the clean repro, and for not copying the secret to get past the check.
What you hit is intended. Serve never releases its control-plane vault (_token-secrets) or its own credential keys (server-token-, worker-token-, oauth-*) to a dispatched step. The check is on the secret's key, so it holds in every vault and for every principal, admins included. A token minted with worker token create --vault fleet is still stored under worker-token-, which is why your control secret resolved and the token didn't. --vault controls where the plaintext is kept, for operators and provisioning outside workflows to read. It doesn't make the token readable by steps.
You were right that this wasn't written down anywhere. The fix makes the reserved_secret error say why it refused, adds the rule to the --vault help, and documents it in the serve guide and remote-execution docs.
The bigger problem you raised still stands: there's no supported way to rotate a connected worker's token without SSH. We're following up in swamp-club#4689 with a way to rotate worker tokens over the worker's own connection, keeping the credential out of workflow steps. Your note that every rotation renames the worker is captured there too.
ehazlett commented 11/12/2026, 8:48:51 PM
Thank you!
Sign in to post a ripple.