Skip to main content
← Back to list
01Issue
BugShippedSwamp CLIPublicTeam
Assigneeszack

Relationships

#4680 Serve won't release a worker token's secret to a dispatched step (reserved_secret), and this isn't documented

Opened by ehazlett · 11/12/2026· Shipped 11/12/2026

Description

A worker token minted through serve into a user vault cannot be read by any dispatched step, even in a run triggered by an admin. Serve refuses with reserved_secret (HTTP 403). An ordinary secret in the same vault, read by a step on the same worker in the same run, resolves fine.

The serve guide documents one reservation: reserved _ vaults are refused to every run except admins. This is a different rule. The vault is a user vault, the run is an admin's, and the refusal is keyed to the secret being a worker token's. It isn't mentioned in swamp help or in the serve docs. The docs also present worker token create --vault as the way to keep a token's secret somewhere other than _token-secrets, which suggests workflows should be able to read it.

Keeping enrollment credentials away from workflow steps is a reasonable rule. The problem is that it's undocumented, and that there is no supported way to do what this path looked designed for: delivering a successor token to a connected worker before its current token expires.

Steps to reproduce

  1. Run swamp serve with one worker connected as web-01. Run the steps below as an admin.

  2. Mint a successor into a user vault through serve: swamp worker token create web-01-r2 --duration 4h --vault fleet --server wss://… swamp worker token list --json reports it with vaultName: fleet, secretKey: worker-token-web-01-r2.

  3. Store an ordinary secret in the same vault: swamp vault put fleet plain-secret.

  4. Run a workflow with two command/shell steps placed on that worker (target: web-01, allowFailure: true). Each passes a secret in env and runs test -n "$V":

    • control-plain-secret: V: ${{ vault.get(fleet, plain-secret) }}
    • worker-token-secret: V: ${{ vault.get(fleet, worker-token-web-01-r2) }}
  5. control-plain-secret succeeds. worker-token-secret fails before it runs:

    Serve refused to load secret 'worker-token-web-01-r2' from vault 'fleet'
    for this dispatch (reserved_secret, HTTP 403)

    @swamp/workflow-summary reports the failed step's detail as only "No data output."; the refusal appears in the run JSON's step error.

Expected

Either the step gets the secret, or the reservation is documented where worker token create --vault is described, along with what --vault is for.

Why it matters

Worker tokens are a hard deadline: a host whose token expires leaves the fleet and has to be bootstrapped again over SSH. Rotating ahead of expiry means getting the new token onto the host over the connection it already has. A dispatched step is the only way to put a file on a worker, and vault.get is the only way a step gets a secret. With this refusal, no workflow can rotate a worker's token. The only alternative is SSH-ing to every host for every rotation.

Copying the secret to a vault key serve doesn't know about would get past the check, but that would defeat the protection, so we haven't done it.

Ask

A supported way to deliver a successor token to the worker it's meant for. For example:

  • a worker token command that rotates a connected worker over its own connection (mint, deliver, cut over); or
  • a grant that lets a token's secret be dispatched only to the worker enrolled under its predecessor (web-01-r2 only to web-01).

At minimum, document the reservation and what --vault is for.

Related: a worker's name is its token's name, so every rotation renames the worker (web-01 → web-01-r2). That's worth keeping in mind for whatever rotation path you design.

Version

swamp 20261112.171308.0 (latest at time of filing). Also seen on 20261109.005252.0.

02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 5 MOREREVIEW+ 6 MOREPR_LINKED+ 2 MORESESSION_SUMMARIZED

Shipped

11/12/2026, 8:34:17 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
zack assigned zack11/12/2026, 8:08:58 PM
Editable. Press Enter to edit.

zack commented 11/12/2026, 8:34:31 PM

Thanks for the thorough report and the clean repro, and for not copying the secret to get past the check.

What you hit is intended. Serve never releases its control-plane vault (_token-secrets) or its own credential keys (server-token-, worker-token-, oauth-*) to a dispatched step. The check is on the secret's key, so it holds in every vault and for every principal, admins included. A token minted with worker token create --vault fleet is still stored under worker-token-, which is why your control secret resolved and the token didn't. --vault controls where the plaintext is kept, for operators and provisioning outside workflows to read. It doesn't make the token readable by steps.

You were right that this wasn't written down anywhere. The fix makes the reserved_secret error say why it refused, adds the rule to the --vault help, and documents it in the serve guide and remote-execution docs.

The bigger problem you raised still stands: there's no supported way to rotate a connected worker's token without SSH. We're following up in swamp-club#4689 with a way to rotate worker tokens over the worker's own connection, keeping the credential out of workflow steps. Your note that every rotation renames the worker is captured there too.

ehazlett commented 11/12/2026, 8:48:51 PM

Thank you!

Sign in to post a ripple.