Skip to main content

DEPLOY WITH HEADLESS OAUTH

This guide shows you how to deploy swamp serve with OAuth authentication in an environment where no browser is available — Kubernetes pods, CI runners, or systemd services.

Prerequisites

  • A collective with owner or admin role
  • swamp auth login completed (to create the token)

1. Create a collective API token

Create a token with both oauth:manage and serve:* scopes:

swamp auth token create --collective myorg \
  --scopes oauth:manage,serve:* \
  --name headless-serve

The output shows the token's key and a signin token. Copy both immediately — they are shown only once, and a lost signin token can only be replaced by creating a new collective token.

oauth:manage allows the server to register itself as an OAuth client and to look up the names in --admins and --allowed-users. serve:* allows instance registration and heartbeats.

2. Set the token in your deployment environment

Store the key as SWAMP_API_KEY and the signin token as SWAMP_SIGNIN_TOKEN in your deployment's secret management. Every Swamp process needs a credential to start; the signin token lets swamp serve start while swamp-club.com is unreachable — see Swamp Account Requirement.

Kubernetes Secret:

kubectl create secret generic swamp-serve \
  --from-literal=SWAMP_API_KEY=swamp_org_abc123... \
  --from-literal=SWAMP_SIGNIN_TOKEN=eyJmcHIiOi...

Reference both in your pod spec:

env:
  - name: SWAMP_API_KEY
    valueFrom:
      secretKeyRef:
        name: swamp-serve
        key: SWAMP_API_KEY
  - name: SWAMP_SIGNIN_TOKEN
    valueFrom:
      secretKeyRef:
        name: swamp-serve
        key: SWAMP_SIGNIN_TOKEN

systemd:

# /etc/swamp-serve.env
SWAMP_API_KEY=swamp_org_abc123...
SWAMP_SIGNIN_TOKEN=eyJmcHIiOi...
# In the [Service] section of the unit file
EnvironmentFile=/etc/swamp-serve.env

Shell (CI or scripts):

export SWAMP_API_KEY=swamp_org_abc123...
export SWAMP_SIGNIN_TOKEN=eyJmcHIiOi...

Read the key from a mounted file

To keep the key out of the process environment, mount it as a file and pass the path with --club-api-key-file. In Kubernetes, mount the secret as a volume:

containers:
  - name: swamp-serve
    args:
      - serve
      - --auth-mode=oauth
      - --club-api-key-file=/run/secrets/swamp/SWAMP_API_KEY
      - --admins=swampadmin
      - --allowed-collectives=myorg
    env:
      - name: SWAMP_SIGNIN_TOKEN
        valueFrom:
          secretKeyRef:
            name: swamp-serve
            key: SWAMP_SIGNIN_TOKEN
    volumeMounts:
      - name: swamp-api-key
        mountPath: /run/secrets/swamp
        readOnly: true
volumes:
  - name: swamp-api-key
    secret:
      secretName: swamp-serve
      items:
        - key: SWAMP_API_KEY
          path: SWAMP_API_KEY

Setting SWAMP_API_KEY_FILE to the path works too. --club-api-key-file takes precedence over both environment variables; setting SWAMP_API_KEY and SWAMP_API_KEY_FILE together is an error. Whitespace around the key in the file is ignored.

For a daemon, pass the flag to swamp serve daemon enable. The service definition stores only the file's path.

3. Start the server

swamp serve --auth-mode oauth \
  --admins swampadmin \
  --allowed-collectives myorg

With SWAMP_API_KEY set, the server registers its OAuth client and resolves admin usernames without any browser interaction. No verification URL is shown, no code needs to be entered.

Setting a stable client name

In container and Kubernetes environments, the default client name (swamp-serve-{repoName}-{hostname}) includes an ephemeral hostname that changes on every pod restart. Use --oauth-client-name or SWAMP_OAUTH_CLIENT_NAME to set a stable, deployment-meaningful name:

swamp serve --auth-mode oauth \
  --oauth-client-name prod-api-server \
  --admins swampadmin \
  --allowed-collectives myorg

Or via environment variable in a pod spec:

env:
  - name: SWAMP_OAUTH_CLIENT_NAME
    value: prod-api-server

4. Verify the bootstrap succeeded

Check the server logs for these messages in order:

SWAMP_API_KEY verified with oauth:manage scope — registering OAuth client (headless)
Registered OAuth client <clientId> via SWAMP_API_KEY (headless)
Using SWAMP_API_KEY to resolve admin/allowed-user usernames: swampadmin
Resolved admin swampadmin to user:<sub>

Confirm the server is ready:

curl -s https://your-server:9090/ready | jq .
{ "ready": true, "instanceId": "a1b2c3d4" }

Subsequent starts

After the first boot, the OAuth client credentials and admin mappings are in the datastore's control-plane store, which Swamp manages; see where token secrets are stored. Subsequent starts reuse them. SWAMP_API_KEY is still read, for instance registration, but the OAuth registration step is skipped. The first boot logs:

No stored OAuth client credentials found — first-time setup required

Later starts log:

Using stored OAuth client credentials from vault

A new or emptied datastore has no stored client, so its first start registers another one. To keep the same client, see Recreate an instance without registering a new client.

Recreate an instance without registering a new client

This runbook gives an instance on a new or emptied datastore the OAuth client it already has. It requires Swamp v20261103.225037.0-sha.5911dc9c or later.

You do not need it when a Pod is replaced and its remote datastore survives: the new Pod finds the stored client.

Reusing the client does not restore the rest of the instance. On an empty datastore:

  • Server tokens are gone, including those from swamp auth server-login. Users log in again, and minted tokens are minted again.
  • Grants made with swamp access grant create are gone. Grants in grant files come back if the files ship with the new instance.
  • Pinned names are gone, so each name in --admins, --allowed-users and grant files is looked up again and resolves to whoever holds it at that time.

1. Register the client and store it

Run this once per deployment, with a collective API token that holds oauth:manage, from the collective the instance runs in. It needs jq and kubectl, and NAMESPACE set to the instance's namespace:

CLIENT=$(swamp auth oauth-client register --name prod-api-server --json) \
  && printf '%s' "$CLIENT" \
  | jq -r '"SWAMP_OAUTH_CLIENT_ID=\(.clientId)\nSWAMP_OAUTH_CLIENT_SECRET=\(.clientSecret)"' \
  | kubectl create secret generic swamp-serve-oauth-client \
      --namespace "$NAMESPACE" --from-env-file=/dev/stdin

The secret is shown once, so it is held in CLIENT until the Secret exists. If register fails, nothing else runs, so fix the cause and run the command again. If kubectl fails, the client is already registered: fix the cause and repeat only the printf pipeline. When the Secret exists, run unset CLIENT.

Use one client per deployment. Replicas of a deployment share it. swamp-club keeps one instance record per client, so independent instances that share a client overwrite each other's record.

2. Give the client to the server

Mount the secret as a file and set the client id in the pod spec:

containers:
  - name: swamp-serve
    env:
      - name: SWAMP_OAUTH_CLIENT_ID
        valueFrom:
          secretKeyRef:
            name: swamp-serve-oauth-client
            key: SWAMP_OAUTH_CLIENT_ID
      - name: SWAMP_OAUTH_CLIENT_SECRET_FILE
        value: /run/secrets/swamp-oauth/SWAMP_OAUTH_CLIENT_SECRET
    volumeMounts:
      - name: swamp-oauth-client
        mountPath: /run/secrets/swamp-oauth
        readOnly: true
volumes:
  - name: swamp-oauth-client
    secret:
      secretName: swamp-serve-oauth-client
      items:
        - key: SWAMP_OAUTH_CLIENT_SECRET
          path: SWAMP_OAUTH_CLIENT_SECRET

--oauth-client-id and --oauth-client-secret-file work too. Setting SWAMP_OAUTH_CLIENT_SECRET also works, but extension code runs in-process and can read it. See Serve Flags.

The key the instance runs with needs both oauth:manage and serve:*, because a start on an empty datastore looks up its names again. That key can also list and delete the collective's OAuth clients. The bootstrap client explains why.

3. Check the configuration

Before deploying, run this in an environment that has the same variables and mounts:

swamp serve check-config --auth-mode oauth \
  --admins swampadmin \
  --allowed-collectives myorg

It reports a secret file that cannot be read, a secret with no client id, and both secret environment variables set together. It passes a client id with no secret, so step 4 is what shows the server starts.

4. Start the server and confirm the client

The logs show the supplied client, and no Registered OAuth client line:

Using OAuth client "<client-id>" with the supplied client secret
OAuth client credentials resolved (clientId: "<client-id>")
Registered serve instance with swamp-club (clientId: "<client-id>")

Log in to confirm the client id and admission work:

swamp auth server-login --server https://your-server:9090

5. Recreate the instance

Start the new instance on its new or emptied datastore with the same Secret, then repeat step 4. The logs name the same client id, and the collective's client list gains no new entry.

Supply the id and secret on every start: they are never stored. A later start with only the id refuses to start unless the store holds a usable secret.

Keep a seeded client in the user vault

On start, Swamp moves OAuth credentials it finds in the default user vault into the control-plane store and deletes them from the vault. To leave oauth-client-id and oauth-client-secret in the vault, set --keep-oauth-client-in-vault. It is off by default, because anything that can read that vault can read the client secret. Other migrated keys are still deleted.

A start with a supplied client secret (--oauth-client-secret-file, SWAMP_OAUTH_CLIENT_SECRET_FILE or SWAMP_OAUTH_CLIENT_SECRET) skips this move entirely and never reads those keys from the user vault, so --keep-oauth-client-in-vault has no effect on it. A seeded client in a vault that several deployments read is shared by every deployment that adopts it; use one client per deployment, as in Recreate an instance without registering a new client.

Key rotation

SWAMP_API_KEY is read fresh from the environment on each boot. To rotate the key:

  1. Create a new token with the same scopes
  2. Update the secret in your deployment environment
  3. Restart the server

The server picks up the new key on restart. A key read from --club-api-key-file or SWAMP_API_KEY_FILE follows the same rule: replacing the file's contents takes effect on the next restart. No OAuth re-registration is needed — the client credentials remain valid.

To change the OAuth client's secret, see Rotate a client secret.

Clean up OAuth clients

A start with no client id and no stored client registers a new OAuth client, so test runs and decommissioned instances leave clients behind. A collective owner can list and delete them.

In the browser, open your collective's Serve page at /o/<slug>/serve. Its tab appears once the collective has a registered instance. The OAuth clients section lists each client with its status:

  • Active — its instance sent a heartbeat in the last hour
  • Stale — its instance's last heartbeat was 1 to 3 hours ago
  • Inactive — its instance has been quiet for more than 3 hours
  • No instance — no instance ever registered with it

Active and Stale clients are in use, and deleting one asks you to confirm that its instance will lose its registration.

Select Delete OAuth client and confirm.

With a collective API key that holds oauth:manage, list the clients:

curl -H "Authorization: Bearer $SWAMP_API_KEY" \
  https://swamp-club.com/api/v1/collectives/<slug>/oauth-clients

Add ?name=<client-name> to list only the clients with that exact name. To delete one, send DELETE to .../oauth-clients/<client-id>. An in-use client answers 409 client_in_use; add ?force=true to delete it anyway.

Deleting a client removes it, its grants, its instance registration and the swamp-club sign-ins people made through its device login. The response counts them in sessions_ended and tokens_revoked. People signed in to that instance are signed out of it the next time it refreshes their collectives, within its --group-refresh-interval (default 4h). New logins through it fail, and its heartbeats and any attempt to register it again are refused. To bring the instance back, register a new client and supply its id and secret on every start, as in Recreate an instance without registering a new client. The supplied pair overrides any stored client. Delete an in-use client only when you are retiring its instance. Any key with oauth:manage can delete any of the collective's clients, including one another instance is using.

The list shows clients registered with a collective API key. A client created by the browser bootstrap belongs to the operative who signed in, so it does not appear here.

Rotate a client secret

Rotate a secret routinely, or when someone who could read it leaves. If you think the secret leaked, follow Respond to a leaked client secret instead.

In the browser, open your collective's Serve page at /o/<slug>/serve, select Rotate secret on the client, choose Keep links, and select Rotate secret. Copy the new secret. It is shown once.

With a collective API key that holds oauth:manage, write the response to a file only you can read, so the secret is not printed to the terminal:

(umask 077 && curl -sf -X POST -H "Authorization: Bearer $SWAMP_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"links": "keep"}' \
  -o rotation.json \
  https://swamp-club.com/api/v1/collectives/<slug>/oauth-clients/<client-id>/secret)

The file holds the new secret in client_secret, and the time the old secret stops working in old_secret_expires_at. Delete it once the secret is stored. A request without links is refused with 400 choice_required, and one that is not application/json with 415 unsupported_media_type.

The old secret keeps working until you remove it, or for 7 days at most. People linked through the client stay linked. While an old secret still works, another rotation that keeps links is refused with 409 overlap_in_progress.

2. Give the server the new secret

Replace the secret where the server reads it: the file at --oauth-client-secret-file, or your Kubernetes Secret. For a client the server registered itself, start it with --oauth-client-id and the new secret file, and supply both on every start from then on.

3. Restart every instance

Restart every instance that uses the client. An instance keeps using the old secret until it restarts.

4. Remove the old secret

The client's row on the Serve page shows when each secret was last used. When the old secret's time stops changing, every instance has switched. An instance that makes no calls to swamp-club shows nothing, so check that each one has restarted.

Select Remove old secret and confirm, or send DELETE to .../oauth-clients/<client-id>/secret/previous. Anything still using the old secret stops working at once. If you do nothing, the old secret stops at the time shown.

Respond to a leaked client secret

1. Revoke the server's API key

If the server's host may be exposed, revoke the collective API key the server runs with before anything else, and create a replacement with the same scopes. A key with oauth:manage can rotate the client's secret. Do not put the replacement key or the new secret on that host until it is rebuilt or cleaned.

On the Serve page, select Rotate secret on the client, choose End all links, and confirm. Copy the new secret. With the API, send {"links": "end"} to the endpoint in Rotate a client secret.

The old secret stops working at once, any old secret from an earlier rotation stops too, and every link made through the client ends. Link requests not yet finished are withdrawn, and swamp-club revokes the OAuth tokens it issued to the client and ends every swamp-club sign-in made through the server's device login. The response counts them in sessions_ended and tokens_revoked. People signed in to the server are signed out of it the next time it refreshes their collectives, within its --group-refresh-interval (default 4h), and log in again once it has the new secret.

Sign-ins made before swamp-club recorded which client a device login came through are not ended. If someone signed in through this server before then, ask them to sign out of their other swamp-club sessions.

If the response is 503 links_not_all_ended, the secret was replaced but not every link has ended. Rotate again with {"links": "end"}, and use the secret from that response.

3. End the delegations on the server

Serve refuses these links on its own within about 30 seconds and ends them once it reads swamp-club again. To end them at once, revoke each integration on the server:

swamp access integration revoke <integration-id>

To end one person's delegation instead:

swamp access delegation end <delegation-id>

4. Revoke the server's tokens

If the server's host may be exposed, revoke the tokens the server issued. List them with swamp access token list, and revoke each with swamp access token revoke <name>. People then log in to the server again.

5. Give the server the new secret

Replace the secret where the server reads it, as in step 2 of Rotate a client secret, and restart every instance.

6. Check that nothing is still standing

For each integration, list its delegations:

swamp access delegation list --integration <integration-id>

You are done when none is awaiting_consent, reporting, consented, issued or suspended. End any that are with swamp access delegation end <delegation-id>.

Each person linked through the client links again from the app they use.

Troubleshooting

Missing oauth:manage scope:

SWAMP_API_KEY is missing the "oauth:manage" scope required for headless OAuth
client registration. Generate a new collective API token that includes
"oauth:manage".

Create a new token with both oauth:manage and serve:* scopes.

Admin username not found:

Failed to resolve admin 'swampadmin': <message>. Ensure the username exists on
https://swamp-club.com.

Verify the username in --admins exists on swamp-club. The server resolves usernames to principal IDs at startup.

Key validation failed:

SWAMP_API_KEY validation failed: <status> <statusText>

Check that the token has not been revoked or expired. Verify with swamp auth whoami.

Login to the server fails with 502 after the browser approval:

Failed to poll device token: 502 Bad Gateway

The server answers Upstream provider returned an unexpected error, and swamp-club logs OAuth client refused. The server sent a client secret swamp-club does not accept: a typo, a secret from before a rotation that ended all links, or an old secret after its overlap ended or was removed. Give the server the client's current secret, as in step 2 of Rotate a client secret, and restart it.

Instance registration or heartbeat refused:

Failed to register serve instance with swamp-club: "Instance registration failed: 422 Unprocessable Entity — OAuth client not found"
OAuth client not found
No registration found for this oauthClientId

The server keeps running without a registered instance. The client was deleted, it belongs to a different collective than the key, or it was created by the browser bootstrap and the operative who created it has left the collective. Register a new client with a key from this collective and supply its id and secret on every start. See Recreate an instance without registering a new client.

Key from a different collective:

Failed to register serve instance with swamp-club: "Instance registration failed: 403 Forbidden — Collective token cannot register for: myorg"

The server keeps running without a registered instance. Use a key from the collective named in --allowed-collectives.

Username lookup refused:

Error: Failed to resolve admin 'swampadmin': Failed to resolve username 'swampadmin': 403 Forbidden. https://swamp-club.com refused to look up users with this credential; if it is a collective API token, it needs the oauth:manage scope.

A name in --admins or --allowed-users has no stored record, which is every name on an empty datastore. Give the key oauth:manage.

A client id with no secret:

Error: No client secret for OAuth client <client-id>. Supply it with --oauth-client-secret-file, SWAMP_OAUTH_CLIENT_SECRET_FILE or SWAMP_OAUTH_CLIENT_SECRET. serve does not register a new client when a client id is set.
Error: The stored OAuth client is <stored-id>, not <client-id>, so its secret cannot be used. Supply the secret for <client-id> with --oauth-client-secret-file, SWAMP_OAUTH_CLIENT_SECRET_FILE or SWAMP_OAUTH_CLIENT_SECRET. serve does not register a new client when a client id is set.

Supply the secret for that client id. Serve does not register a client when an id is set.

A secret with no client id:

Error: An OAuth client secret was supplied without a client id. Set --oauth-client-id or SWAMP_OAUTH_CLIENT_ID to the client the secret belongs to.

Set the client id the secret belongs to.

Both secret environment variables set:

Error: SWAMP_OAUTH_CLIENT_SECRET and SWAMP_OAUTH_CLIENT_SECRET_FILE are mutually exclusive — unset one of them

Unset one of them. --oauth-client-secret-file overrides both.

Key file not found:

Error: --club-api-key-file file not found: /run/secrets/swamp/SWAMP_API_KEY

The error names the source and the path it read. Check the volume mount and the path passed to --club-api-key-file or set in SWAMP_API_KEY_FILE. An empty or unreadable file fails the same way, with file is empty or file not readable.