DEPLOY WITH HEADLESS OAUTH
This guide shows you how to deploy swamp serve with OAuth authentication in an
environment where no browser is available — Kubernetes pods, CI runners, or
systemd services.
Prerequisites
- A collective with owner or admin role
swamp auth logincompleted (to create the token)
1. Create a collective API token
Create a token with both oauth:manage and serve:* scopes:
swamp auth token create --collective myorg \
--scopes oauth:manage,serve:* \
--name headless-serveThe output shows the token's key and a signin token. Copy both immediately — they are shown only once, and a lost signin token can only be replaced by creating a new collective token.
oauth:manage allows the server to register itself as an OAuth client and to
look up the names in --admins and --allowed-users. serve:* allows instance
registration and heartbeats.
2. Set the token in your deployment environment
Store the key as SWAMP_API_KEY and the signin token as SWAMP_SIGNIN_TOKEN in
your deployment's secret management. Every Swamp process needs a credential to
start; the signin token lets swamp serve start while swamp-club.com is
unreachable — see
Swamp Account Requirement.
Kubernetes Secret:
kubectl create secret generic swamp-serve \
--from-literal=SWAMP_API_KEY=swamp_org_abc123... \
--from-literal=SWAMP_SIGNIN_TOKEN=eyJmcHIiOi...Reference both in your pod spec:
env:
- name: SWAMP_API_KEY
valueFrom:
secretKeyRef:
name: swamp-serve
key: SWAMP_API_KEY
- name: SWAMP_SIGNIN_TOKEN
valueFrom:
secretKeyRef:
name: swamp-serve
key: SWAMP_SIGNIN_TOKENsystemd:
# /etc/swamp-serve.env
SWAMP_API_KEY=swamp_org_abc123...
SWAMP_SIGNIN_TOKEN=eyJmcHIiOi...# In the [Service] section of the unit file
EnvironmentFile=/etc/swamp-serve.envShell (CI or scripts):
export SWAMP_API_KEY=swamp_org_abc123...
export SWAMP_SIGNIN_TOKEN=eyJmcHIiOi...Read the key from a mounted file
To keep the key out of the process environment, mount it as a file and pass the
path with --club-api-key-file. In Kubernetes, mount the secret as a volume:
containers:
- name: swamp-serve
args:
- serve
- --auth-mode=oauth
- --club-api-key-file=/run/secrets/swamp/SWAMP_API_KEY
- --admins=swampadmin
- --allowed-collectives=myorg
env:
- name: SWAMP_SIGNIN_TOKEN
valueFrom:
secretKeyRef:
name: swamp-serve
key: SWAMP_SIGNIN_TOKEN
volumeMounts:
- name: swamp-api-key
mountPath: /run/secrets/swamp
readOnly: true
volumes:
- name: swamp-api-key
secret:
secretName: swamp-serve
items:
- key: SWAMP_API_KEY
path: SWAMP_API_KEYSetting SWAMP_API_KEY_FILE to the path works too. --club-api-key-file takes
precedence over both environment variables; setting SWAMP_API_KEY and
SWAMP_API_KEY_FILE together is an error. Whitespace around the key in the file
is ignored.
For a daemon, pass the flag to swamp serve daemon enable. The service
definition stores only the file's path.
3. Start the server
swamp serve --auth-mode oauth \
--admins swampadmin \
--allowed-collectives myorgWith SWAMP_API_KEY set, the server registers its OAuth client and resolves
admin usernames without any browser interaction. No verification URL is shown,
no code needs to be entered.
Setting a stable client name
In container and Kubernetes environments, the default client name
(swamp-serve-{repoName}-{hostname}) includes an ephemeral hostname that
changes on every pod restart. Use --oauth-client-name or
SWAMP_OAUTH_CLIENT_NAME to set a stable, deployment-meaningful name:
swamp serve --auth-mode oauth \
--oauth-client-name prod-api-server \
--admins swampadmin \
--allowed-collectives myorgOr via environment variable in a pod spec:
env:
- name: SWAMP_OAUTH_CLIENT_NAME
value: prod-api-server4. Verify the bootstrap succeeded
Check the server logs for these messages in order:
SWAMP_API_KEY verified with oauth:manage scope — registering OAuth client (headless)
Registered OAuth client <clientId> via SWAMP_API_KEY (headless)
Using SWAMP_API_KEY to resolve admin/allowed-user usernames: swampadmin
Resolved admin swampadmin to user:<sub>Confirm the server is ready:
curl -s https://your-server:9090/ready | jq .{ "ready": true, "instanceId": "a1b2c3d4" }Subsequent starts
After the first boot, the OAuth client credentials and admin mappings are in the
datastore's control-plane store, which Swamp manages; see
where token secrets are stored.
Subsequent starts reuse them. SWAMP_API_KEY is still read, for instance
registration, but the OAuth registration step is skipped. The first boot logs:
No stored OAuth client credentials found — first-time setup requiredLater starts log:
Using stored OAuth client credentials from vaultA new or emptied datastore has no stored client, so its first start registers another one. To keep the same client, see Recreate an instance without registering a new client.
Recreate an instance without registering a new client
This runbook gives an instance on a new or emptied datastore the OAuth client it
already has. It requires Swamp v20261103.225037.0-sha.5911dc9c or later.
You do not need it when a Pod is replaced and its remote datastore survives: the new Pod finds the stored client.
Reusing the client does not restore the rest of the instance. On an empty datastore:
- Server tokens are gone, including those from
swamp auth server-login. Users log in again, and minted tokens are minted again. - Grants made with
swamp access grant createare gone. Grants in grant files come back if the files ship with the new instance. - Pinned names are gone, so each name in
--admins,--allowed-usersand grant files is looked up again and resolves to whoever holds it at that time.
1. Register the client and store it
Run this once per deployment, with a collective API token that holds
oauth:manage, from the collective the instance runs in. It needs jq and
kubectl, and NAMESPACE set to the instance's namespace:
CLIENT=$(swamp auth oauth-client register --name prod-api-server --json) \
&& printf '%s' "$CLIENT" \
| jq -r '"SWAMP_OAUTH_CLIENT_ID=\(.clientId)\nSWAMP_OAUTH_CLIENT_SECRET=\(.clientSecret)"' \
| kubectl create secret generic swamp-serve-oauth-client \
--namespace "$NAMESPACE" --from-env-file=/dev/stdinThe secret is shown once, so it is held in CLIENT until the Secret exists. If
register fails, nothing else runs, so fix the cause and run the command again.
If kubectl fails, the client is already registered: fix the cause and repeat
only the printf pipeline. When the Secret exists, run unset CLIENT.
Use one client per deployment. Replicas of a deployment share it. swamp-club keeps one instance record per client, so independent instances that share a client overwrite each other's record.
2. Give the client to the server
Mount the secret as a file and set the client id in the pod spec:
containers:
- name: swamp-serve
env:
- name: SWAMP_OAUTH_CLIENT_ID
valueFrom:
secretKeyRef:
name: swamp-serve-oauth-client
key: SWAMP_OAUTH_CLIENT_ID
- name: SWAMP_OAUTH_CLIENT_SECRET_FILE
value: /run/secrets/swamp-oauth/SWAMP_OAUTH_CLIENT_SECRET
volumeMounts:
- name: swamp-oauth-client
mountPath: /run/secrets/swamp-oauth
readOnly: true
volumes:
- name: swamp-oauth-client
secret:
secretName: swamp-serve-oauth-client
items:
- key: SWAMP_OAUTH_CLIENT_SECRET
path: SWAMP_OAUTH_CLIENT_SECRET--oauth-client-id and --oauth-client-secret-file work too. Setting
SWAMP_OAUTH_CLIENT_SECRET also works, but extension code runs in-process and
can read it. See
Serve Flags.
The key the instance runs with needs both oauth:manage and serve:*, because
a start on an empty datastore looks up its names again. That key can also list
and delete the collective's OAuth clients.
The bootstrap client
explains why.
3. Check the configuration
Before deploying, run this in an environment that has the same variables and mounts:
swamp serve check-config --auth-mode oauth \
--admins swampadmin \
--allowed-collectives myorgIt reports a secret file that cannot be read, a secret with no client id, and both secret environment variables set together. It passes a client id with no secret, so step 4 is what shows the server starts.
4. Start the server and confirm the client
The logs show the supplied client, and no Registered OAuth client line:
Using OAuth client "<client-id>" with the supplied client secret
OAuth client credentials resolved (clientId: "<client-id>")
Registered serve instance with swamp-club (clientId: "<client-id>")Log in to confirm the client id and admission work:
swamp auth server-login --server https://your-server:90905. Recreate the instance
Start the new instance on its new or emptied datastore with the same Secret, then repeat step 4. The logs name the same client id, and the collective's client list gains no new entry.
Supply the id and secret on every start: they are never stored. A later start with only the id refuses to start unless the store holds a usable secret.
Keep a seeded client in the user vault
On start, Swamp moves OAuth credentials it finds in the default user vault into
the control-plane store and deletes them from the vault. To leave
oauth-client-id and oauth-client-secret in the vault, set
--keep-oauth-client-in-vault. It is off by default, because anything that can
read that vault can read the client secret. Other migrated keys are still
deleted.
A start with a supplied client secret (--oauth-client-secret-file,
SWAMP_OAUTH_CLIENT_SECRET_FILE or SWAMP_OAUTH_CLIENT_SECRET) skips this move
entirely and never reads those keys from the user vault, so
--keep-oauth-client-in-vault has no effect on it. A seeded client in a vault
that several deployments read is shared by every deployment that adopts it; use
one client per deployment, as in
Recreate an instance without registering a new client.
Key rotation
SWAMP_API_KEY is read fresh from the environment on each boot. To rotate the
key:
- Create a new token with the same scopes
- Update the secret in your deployment environment
- Restart the server
The server picks up the new key on restart. A key read from
--club-api-key-file or SWAMP_API_KEY_FILE follows the same rule: replacing
the file's contents takes effect on the next restart. No OAuth re-registration
is needed — the client credentials remain valid.
To change the OAuth client's secret, see Rotate a client secret.
Clean up OAuth clients
A start with no client id and no stored client registers a new OAuth client, so test runs and decommissioned instances leave clients behind. A collective owner can list and delete them.
In the browser, open your collective's Serve page at /o/<slug>/serve. Its
tab appears once the collective has a registered instance. The OAuth clients
section lists each client with its status:
- Active — its instance sent a heartbeat in the last hour
- Stale — its instance's last heartbeat was 1 to 3 hours ago
- Inactive — its instance has been quiet for more than 3 hours
- No instance — no instance ever registered with it
Active and Stale clients are in use, and deleting one asks you to confirm that its instance will lose its registration.
Select Delete OAuth client and confirm.
With a collective API key that holds oauth:manage, list the clients:
curl -H "Authorization: Bearer $SWAMP_API_KEY" \
https://swamp-club.com/api/v1/collectives/<slug>/oauth-clientsAdd ?name=<client-name> to list only the clients with that exact name. To
delete one, send DELETE to .../oauth-clients/<client-id>. An in-use client
answers 409 client_in_use; add ?force=true to delete it anyway.
Deleting a client removes it, its grants, its instance registration and the
swamp-club sign-ins people made through its device login. The response counts
them in sessions_ended and tokens_revoked. People signed in to that instance
are signed out of it the next time it refreshes their collectives, within its
--group-refresh-interval (default 4h). New logins through it fail, and its
heartbeats and any attempt to register it again are refused. To bring the
instance back, register a new client and supply its id and secret on every
start, as in
Recreate an instance without registering a new client.
The supplied pair overrides any stored client. Delete an in-use client only when
you are retiring its instance. Any key with oauth:manage can delete any of the
collective's clients, including one another instance is using.
The list shows clients registered with a collective API key. A client created by the browser bootstrap belongs to the operative who signed in, so it does not appear here.
Rotate a client secret
Rotate a secret routinely, or when someone who could read it leaves. If you think the secret leaked, follow Respond to a leaked client secret instead.
1. Rotate and keep links
In the browser, open your collective's Serve page at /o/<slug>/serve,
select Rotate secret on the client, choose Keep links, and select
Rotate secret. Copy the new secret. It is shown once.
With a collective API key that holds oauth:manage, write the response to a
file only you can read, so the secret is not printed to the terminal:
(umask 077 && curl -sf -X POST -H "Authorization: Bearer $SWAMP_API_KEY" \
-H "Content-Type: application/json" \
-d '{"links": "keep"}' \
-o rotation.json \
https://swamp-club.com/api/v1/collectives/<slug>/oauth-clients/<client-id>/secret)The file holds the new secret in client_secret, and the time the old secret
stops working in old_secret_expires_at. Delete it once the secret is stored. A
request without links is refused with 400 choice_required, and one that is
not application/json with 415 unsupported_media_type.
The old secret keeps working until you remove it, or for 7 days at most. People
linked through the client stay linked. While an old secret still works, another
rotation that keeps links is refused with 409 overlap_in_progress.
2. Give the server the new secret
Replace the secret where the server reads it: the file at
--oauth-client-secret-file, or your Kubernetes Secret. For a client the server
registered itself, start it with --oauth-client-id and the new secret file,
and supply both on every start from then on.
3. Restart every instance
Restart every instance that uses the client. An instance keeps using the old secret until it restarts.
4. Remove the old secret
The client's row on the Serve page shows when each secret was last used. When the old secret's time stops changing, every instance has switched. An instance that makes no calls to swamp-club shows nothing, so check that each one has restarted.
Select Remove old secret and confirm, or send DELETE to
.../oauth-clients/<client-id>/secret/previous. Anything still using the old
secret stops working at once. If you do nothing, the old secret stops at the
time shown.
Respond to a leaked client secret
1. Revoke the server's API key
If the server's host may be exposed, revoke the collective API key the server
runs with before anything else, and create a replacement with the same scopes. A
key with oauth:manage can rotate the client's secret. Do not put the
replacement key or the new secret on that host until it is rebuilt or cleaned.
2. Rotate and end all links
On the Serve page, select Rotate secret on the client, choose End all
links, and confirm. Copy the new secret. With the API, send {"links": "end"}
to the endpoint in Rotate a client secret.
The old secret stops working at once, any old secret from an earlier rotation
stops too, and every link made through the client ends. Link requests not yet
finished are withdrawn, and swamp-club revokes the OAuth tokens it issued to the
client and ends every swamp-club sign-in made through the server's device login.
The response counts them in sessions_ended and tokens_revoked. People signed
in to the server are signed out of it the next time it refreshes their
collectives, within its --group-refresh-interval (default 4h), and log in
again once it has the new secret.
Sign-ins made before swamp-club recorded which client a device login came through are not ended. If someone signed in through this server before then, ask them to sign out of their other swamp-club sessions.
If the response is 503 links_not_all_ended, the secret was replaced but not
every link has ended. Rotate again with {"links": "end"}, and use the secret
from that response.
3. End the delegations on the server
Serve refuses these links on its own within about 30 seconds and ends them once it reads swamp-club again. To end them at once, revoke each integration on the server:
swamp access integration revoke <integration-id>To end one person's delegation instead:
swamp access delegation end <delegation-id>4. Revoke the server's tokens
If the server's host may be exposed, revoke the tokens the server issued. List
them with swamp access token list, and revoke each with
swamp access token revoke <name>. People then log in to the server again.
5. Give the server the new secret
Replace the secret where the server reads it, as in step 2 of Rotate a client secret, and restart every instance.
6. Check that nothing is still standing
For each integration, list its delegations:
swamp access delegation list --integration <integration-id>You are done when none is awaiting_consent, reporting, consented, issued
or suspended. End any that are with
swamp access delegation end <delegation-id>.
7. Ask people to link again
Each person linked through the client links again from the app they use.
Troubleshooting
Missing oauth:manage scope:
SWAMP_API_KEY is missing the "oauth:manage" scope required for headless OAuth
client registration. Generate a new collective API token that includes
"oauth:manage".Create a new token with both oauth:manage and serve:* scopes.
Admin username not found:
Failed to resolve admin 'swampadmin': <message>. Ensure the username exists on
https://swamp-club.com.Verify the username in --admins exists on swamp-club. The server resolves
usernames to principal IDs at startup.
Key validation failed:
SWAMP_API_KEY validation failed: <status> <statusText>Check that the token has not been revoked or expired. Verify with
swamp auth whoami.
Login to the server fails with 502 after the browser approval:
Failed to poll device token: 502 Bad GatewayThe server answers Upstream provider returned an unexpected error, and
swamp-club logs OAuth client refused. The server sent a client secret
swamp-club does not accept: a typo, a secret from before a rotation that ended
all links, or an old secret after its overlap ended or was removed. Give the
server the client's current secret, as in
step 2 of Rotate a client secret, and
restart it.
Instance registration or heartbeat refused:
Failed to register serve instance with swamp-club: "Instance registration failed: 422 Unprocessable Entity — OAuth client not found"
OAuth client not found
No registration found for this oauthClientIdThe server keeps running without a registered instance. The client was deleted, it belongs to a different collective than the key, or it was created by the browser bootstrap and the operative who created it has left the collective. Register a new client with a key from this collective and supply its id and secret on every start. See Recreate an instance without registering a new client.
Key from a different collective:
Failed to register serve instance with swamp-club: "Instance registration failed: 403 Forbidden — Collective token cannot register for: myorg"The server keeps running without a registered instance. Use a key from the
collective named in --allowed-collectives.
Username lookup refused:
Error: Failed to resolve admin 'swampadmin': Failed to resolve username 'swampadmin': 403 Forbidden. https://swamp-club.com refused to look up users with this credential; if it is a collective API token, it needs the oauth:manage scope.A name in --admins or --allowed-users has no stored record, which is every
name on an empty datastore. Give the key oauth:manage.
A client id with no secret:
Error: No client secret for OAuth client <client-id>. Supply it with --oauth-client-secret-file, SWAMP_OAUTH_CLIENT_SECRET_FILE or SWAMP_OAUTH_CLIENT_SECRET. serve does not register a new client when a client id is set.Error: The stored OAuth client is <stored-id>, not <client-id>, so its secret cannot be used. Supply the secret for <client-id> with --oauth-client-secret-file, SWAMP_OAUTH_CLIENT_SECRET_FILE or SWAMP_OAUTH_CLIENT_SECRET. serve does not register a new client when a client id is set.Supply the secret for that client id. Serve does not register a client when an id is set.
A secret with no client id:
Error: An OAuth client secret was supplied without a client id. Set --oauth-client-id or SWAMP_OAUTH_CLIENT_ID to the client the secret belongs to.Set the client id the secret belongs to.
Both secret environment variables set:
Error: SWAMP_OAUTH_CLIENT_SECRET and SWAMP_OAUTH_CLIENT_SECRET_FILE are mutually exclusive — unset one of themUnset one of them. --oauth-client-secret-file overrides both.
Key file not found:
Error: --club-api-key-file file not found: /run/secrets/swamp/SWAMP_API_KEYThe error names the source and the path it read. Check the volume mount and the
path passed to --club-api-key-file or set in SWAMP_API_KEY_FILE. An empty or
unreadable file fails the same way, with file is empty or file not readable.
Related
- Set Up OAuth Authentication — interactive OAuth setup for environments with a browser
- Run as a System Daemon —
run
swamp serveas a launchd or systemd service - Run a Multi-Instance Deployment
— deploy multiple
swamp serveinstances behind a load balancer - Serve Flags — full flag and environment variable reference
- The bootstrap client —
how headless bootstrap works and why it requires
oauth:manage