Lab #2481: serve HA: a server token minted on one replica is rejected by the other replicas until they restart (auto-definitions/ is only pulled at boot)

Summary

On a multi-replica swamp serve with a remote datastore, a server token minted on one replica cannot authenticate on the others until they restart. The token definition is written to auto-definitions/, and running replicas never pull that directory after boot.

This came out of #2466 (fixed in swamp-club/swamp#2606): access token mint --server now works and is live at once on the replica that handled it. Other replicas still reject the token.

Why

Auth reads the token definition from the local repo on every connection: createServerTokenAuthDeps then definitionRepo.findByName(SERVER_TOKEN_MODEL_TYPE, name) in src/serve/token_auth.ts. A missing definition fails with "Server token ... does not exist — mint it first".
A mint writes the definition to auto-definitions/ and pushes it (per-path markDirty, since #2127/#2275). The token data goes to data/, and the secret goes to the control-plane store.
Other replicas pull auto-definitions/ only at boot: the full boot pull, plus the managedConfig pull in src/cli/commands/serve.ts around line 1910. After boot, the pollers pull only data/ (RuntimeDataPoller, every 30s), config/ and extension dirs (ConfigPoller), and the grant/group data dirs (AccessDataPoller). None of them pull auto-definitions/.

So a peer replica gets the token data within about 30s and can read the secret from the shared control-plane store, but it never sees the definition, and auth fails until that replica restarts.

Likely also affected (from reading the code, not reproduced)

Anything else authenticated through an auto-definition created at runtime on another replica:

server tokens minted by the OAuth device login (device_auth_handler.ts), when the next connection lands on a different replica;
worker enrollment tokens.

Steps to reproduce

Run two swamp serve replicas against the same S3/GCS datastore, with --auth-mode token.
As an admin, run swamp access token mint probe --principal user:probe --server <replica A>.
Reveal the credential and connect to replica B with it (for example swamp access can-i --server <replica B> --token <credential>).

Expected

Replica B accepts the token within its normal poll interval, with no restart.

Actual (expected from the code path above)

Replica B fails authentication with "does not exist — mint it first" until it restarts. Replica A accepts the token.

Possible directions

Have a poller also pull auto-definitions/, or at least the server-token and enrollment-token type directories.
Or, on a definition miss during auth, do a targeted pull of that token definition before failing.

Session-start 'swamp model search --json' in the AGENTS.md template is unfulfillable - wire it to a hook or drop it

Flaky property test: analyzeExpression dot/bracket agreement fails when the generated model name is a CEL keyword

Accept usernames for user subjects in grant files

Docs: enable-managed-config should say to re-run config migrate when its push fails

Docs: grant file reference shows a stale format and omits resources and subjects

Flaky property test: analyzeExpression hyphenated dot access fails when modelName generates a CEL keyword ("in")

extension quality and push crash with IsADirectory when additionalFiles lists a directory

extension push: a required global argument renders with a stray "" ("text": "string""")

Docs: extension publishing reference is stale on bare specifiers, quality output and fmt config (swamp-club#3022)

swamp-extensions: replace bare import specifiers with npm:/jsr: specifiers in first-party extensions

extension push: 'Requested visibility' line prints to stdout without the logger prefix, twice per run

A nested structural swamp that outlives the run that started it keeps skipping that run's lock while the holder writes

Show signal waits to remote clients and the dashboard

Settle expired signal waits from swamp serve

Resume signalled workflow runs automatically under swamp serve

extension push/quality: structured acceptances for agents, a clearer title than For next time, and an optional reason

extension push: the undecided private-entitlement verdict still says the registry may start a trial at publish

seed:collective cannot produce a superseded personal trial or a returned trial, so the UAT cannot cover them

Replayable state fragments: Git-reviewable desired state for resources changed by model methods

extension push: registry metadata and content hash label files relative to the repo dir, so a push with --extensions-dir records ../ paths

registry: store and show declared acceptances sent in contentMetadata.acceptances at confirm (swamp-club#3021)

extension push --dry-run: authentication verdict prints 'Signed in as .' when the credential is an API key with no username

serve --remote-only: step output from workers is never persisted to the run log (history logs always empty)

serve HA: worker enrollment on a replica that lacks the enrollment token's definition may create a second definition (unverified)

serve HA: pull a token's definition on an auth miss so a peer accepts a new token at once

extension push: version-drift check reports 'no previously published version found' when the registry call failed

Search and list output follows directory iteration order in the definition, workflow, vault and output repositories

extension push: accept attestation-backed adversarial-review evidence (lookup by extension and content hash), generated provenance, and carry the evidence to the registry at confirm (design: Lab #3023)

Workflow load errors print the raw Zod issue dump instead of a readable message

Decide a stricter naming rule for workflow step and job names

serve: run scheduled workflows concurrently (per-workflow serialization), and report queue delay

Show expired approval gates as expired, with a Cancel action in the dashboard

digitalocean codegen leaves an orphaned security_secret.ts model file after its endpoint left the spec

extension promote accepts a yanked version and reports it promoted

Docs: doctor install reports a stale or failing autoupdate scheduler

extension push output: double-quoted adjacent interpolations, repo-relative paths, and --json emits three JSON documents

auto-resolve: the Installing line prints the extension's entire multi-line description

s3-datastore/serve: readResource without a version returns an older version; the latest marker is rewritten backwards

Publish agent-facing projections of the manual: /llms.txt, .md pages, and llms-full.txt

Let agent-runner drive more agent CLIs (Kilo Code and others)

s3-datastore: every fast-path miss re-downloads every _index shard, so a busy serve pulls the whole index every poll (~230 GB/day S3 egress)

Feedback: Swamp as an approval-gated control plane for a small fleet

extension safety analyzer's Deno.Command( warning is a plain substring match

Tell timeouts apart from cancels in method-run records, and review the hidden 30 s fallback timer for step-called model methods

Run from a git worktree: definitions from the worktree, state from the shared repo

S3/GCS datastore: a push that fails after uploading drops its recorded deletes, so the retry brings deleted data back

Tests: bring the in-memory remote's default semantics up to @swamp/s3-datastore and @swamp/gcs-datastore 2026.10.01.1

Tell the pushing client when its extension contentMetadata fails validation

Extension datastore: query still returns an item after its delete is pulled

Cancel, reject and supersede of a parent run should settle its suspended nested child runs

Relationships

#2481 serve HA: a server token minted on one replica is rejected by the other replicas until they restart (auto-definitions/ is only pulled at boot)