Skip to main content
← Back to list
01Issue
BugShippedSwamp CLI
Assigneeskeeb

Relationships

#1476 serve oauth bootstrap: device flow rate-limited to a 45-second window by validateClient counter, shared globally across tenants

Opened by keeb · 7/30/2026· Shipped 7/30/2026

Summary

The swamp serve --auth-mode oauth first-boot OAuth client bootstrap cannot complete unless a human approves the device flow within ~45 seconds. After that the flow hard-fails and swamp serve exits FTL. Once it has failed, subsequent attempts fail immediately at the device authorization request for the rest of the hour.

This has been broken since the feature landed (swamp-club ccd594ee, 2026-07-07, Lab #1015). It is not a regression — the CLI side of the request has been byte-identical since 0aea55f1f (2026-07-08, Lab #1016). It surfaced now because a customer deployment is the first to actually exercise the bootstrap.

Root cause

swamp-club lib/auth.ts:880 implements a 10-request-per-hour rate limit inside the deviceAuthorization plugin's validateClient hook:

async validateClient(clientId) {
  const now = Date.now();
  const entry = deviceCodeRateLimit.get(clientId);
  if (entry && now < entry.resetAt && entry.count >= 10) {
    return false;
  }
  ...

But validateClient is not called once per device flow. better-auth invokes it on both endpoints:

  • POST /api/auth/device/code — device-authorization/routes.mjs:88
  • POST /api/auth/device/token — routes.mjs:181, i.e. every poll

With interval: "5s" (lib/auth.ts:879), a single bootstrap spends its entire hourly budget in about 45 seconds:

call what count
1 /device/code — verification URL + user code logged 1
2-10 nine /device/token polls at 5s intervals 2-10
11 tenth poll, about 45 seconds in rejected

On rejection the token endpoint returns 400 invalid_grant "Invalid client ID" (routes.mjs:181-184). invalid_grant is not in KNOWN_POLL_ERRORS (swamp/src/serve/oauth_client.ts:69), so pollForToken throws a plain Error that escapes the retry loop at src/cli/commands/serve.ts:1196. There is no try/catch around resolveOAuthClientCredentials, so the process exits.

Subsequent restarts then fail one step earlier — at /device/code, with 400 invalid_client — because the hour-long window is already exhausted. That is the error the customer reports:

[INF] serve·oauth-registration: No stored OAuth client credentials found — first-time setup required
[FTL] error: Error: Device authorization request failed: 400 Bad Request

It self-heals roughly an hour after the window's first call, then re-breaks 45 seconds later, which reads as "intermittent" but is deterministic.

Aggravating factor: the bucket is global across tenants

deviceCodeRateLimit is keyed on client_id alone (lib/auth.ts:45-48), and every swamp serve bootstrap sends the same hardcoded swamp-serve-bootstrap (swamp/src/serve/oauth_registration.ts:35). The bootstrap client is exempted from the registration lookup at lib/auth.ts:900 but only after the counter increments, so it is not exempt from the limit.

Net effect: one global 10/hour bucket shared by every customer bootstrapping swamp serve, per web replica. Unrelated tenants can lock each other out, and because the map is in-process memory the same request can succeed on one replica and 400 on another.

The limit is also redundant

/api/auth/device/code and /api/auth/device/token are already rate limited per-IP by two layers above this hook, with no /api/auth carve-out:

  • GLOBAL_RATE_LIMIT_RULES — lib/app/repos.ts:1581 — 1000/min per IP on the /api/ prefix, pre-auth (routes/_middleware.ts:471)
  • API_PRINCIPAL_RATE_LIMIT_RULES — lib/app/repos.ts:1613 — 120/min per IP for anonymous /api/ requests (routes/_middleware.ts:209)

Rule matching is a plain path.startsWith() (lib/infrastructure/rate-limiter.ts:81-103). better-auth's own default limiter applies on top.

Steps to reproduce

  1. swamp serve --auth-mode oauth --allowed-collectives <slug> --admins <username> against a repo with a vault and no stored OAuth client credentials.
  2. Observe First-time OAuth setup — visit ... and verify code: ....
  3. Do not approve in the browser. Wait about 45 seconds.
  4. swamp serve exits: Token request failed: 400 invalid_grant.
  5. Restart. It now fails at Device authorization request failed: 400 Bad Request and keeps failing for the remainder of the hour.

Approving in under ~45 seconds is the only way through, which is not achievable in a Kubernetes deployment where the URL and code are only visible by tailing pod logs.

Proposed fix

Remove the rate limiting from validateClient and let it be a pure client-validity check. Delete lib/auth.ts:881-898 and the now-unused deviceCodeRateLimit map at lib/auth.ts:45-48:

async validateClient(clientId) {
  if (clientId === BOOTSTRAP_CLIENT_ID) return true;
  if (!mongoClient) return false;
  try {
    const app = await mongoClient.db().collection("oauthApplication")
      .findOne({ clientId, disabled: { $ne: true } });
    return app !== null;
  } catch {
    return false;
  }
}

Rate limiting a protocol-mandated poll interval is the core mistake — the client polls at the rate the server told it to in the interval field, so charging those polls against a quota is guaranteed to reject conforming clients. If a device-specific limit is still wanted, it belongs at the middleware layer keyed by IP, where the request is actually available and the two endpoints can be distinguished.

Why tests did not catch it

lib/auth_test_utils.ts:169-179 substitutes a rate-limit-free validateClient, so tests/auth/device_authorization_test.ts has never exercised the production implementation. Removing the counter makes the two converge. Worth adding a regression test that drives a full device flow through more than ten polls.

  1. The 400 body is discarded, making this undiagnosable from logs. startDeviceGrant (swamp/src/serve/oauth_client.ts:87-91) interpolates only resp.status and resp.statusText, dropping the OAuth error field. Its two siblings both surface it — pollForToken (oauth_client.ts:127-137) and the /register call (serve.ts:1220-1227). Diagnosing this required reading server source, because 400 Bad Request alone cannot distinguish invalid_client from VALIDATION_ERROR.
  2. invalid_grant is not a known poll error. Adding it to KNOWN_POLL_ERRORS (oauth_client.ts:69) would not fix the root cause, but the current behavior turns a mid-flow server rejection into an unhandled throw that kills the server rather than a clear error.
  3. A Mongo throw reports invalid_client. catch { return false } at lib/auth.ts:907-909 makes a database blip indistinguishable from a bad client id.
02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 5 MOREREVIEW+ 3 MOREPR_MERGED+ 1 MORENOTIFICATION_SKIPPED

Shipped

7/30/2026, 3:03:54 AM

Click a lifecycle step above to view its details.

03Sludge Pulse
keeb assigned keeb7/30/2026, 2:19:28 AM
Editable. Press Enter to edit.

keeb commented 7/30/2026, 2:47:44 AM

Triage + fix in progress. Two findings worth recording beyond the original report.

The blast radius is wider than bootstrap. swamp auth server-login also runs a device grant, and src/serve/device_auth_handler.ts:145,189 passes authConfig.oauthClientId — the deployment's single registered client id. So every end-user login for a deployment shared ONE 10/hour bucket per web replica, not just the bootstrap. Two unhurried user logins could lock out every other operative on that deployment for the rest of the hour.

The broken counter was the only bound on unauthenticated deviceCode writes. POST /api/auth/device/code needs no auth and writes a document per call, but better-auth only deletes a row when it is polled after expiry, denied, or approved — an abandoned grant is never reaped, and the collection carried no TTL index. Removing the counter without a replacement would have left the 120/min per-IP anonymous rule as the only ceiling. The fix therefore pairs the deletion with a TTL index on deviceCode.expiresAt (migration 019), which is the correct mechanism anyway.

Reproduced exactly as described (poll 10 → invalid_grant, then /device/code → invalid_client), and confirmed fixed: a conforming client polling at the advertised 5s interval now stays authorization_pending for all 15 polls across 75s, and the approve → token path still returns a Bearer [REDACTED-SECRET-1]

Two CLI-side follow-ups from the report have been filed separately, since they live in the swamp repo rather than swamp-club: #1477 (startDeviceGrant discards the OAuth error field, which is why the customer log said only 400 Bad Request) and #1478 (an unrecognized device-token error escapes the retry loop and kills the whole swamp serve process).

Sign in to post a ripple.