Relationships
#3074 serve HA: worker enrollment on a replica that lacks the enrollment token's definition may create a second definition (unverified)
Opened by hammz · 10/6/2026
Summary
Found by code reading while fixing #2481 (swamp-club/swamp#2861). This is not reproduced. It needs someone to confirm or rule it out.
A worker enrollment token created through one swamp serve replica has its definition written to auto-definitions/ on that replica. If a worker then enrolls against a different replica before that replica has pulled the definition, the redeem may auto-create a second definition with the same name and a new id, instead of failing.
Why I think so
WorkerGatewayredeems through its default model-method runner (#defaultRunModelMethodinsrc/serve/worker_gateway.ts), which callsmodelMethodRunwith both a type argument (ENROLLMENT_TOKEN_MODEL_TYPE.normalized) and a definition name. That is direct type execution, which creates the definition when none exists under that name.- Token records, by contrast, are found by scanning data for the type (
#scanTokenRecords), which does not need the definition. - So on a peer without the definition, redeem could run against a freshly created definition whose id has no token data. If that new definition is then pushed, it would land at the same path as the original (
auto-definitions/swamp/enrollment-token/<name>.yaml) with a different id, and the original token data would no longer be reachable by name.
Each step after the first bullet is inference. I have not confirmed that direct type execution auto-creates in this path, what redeem does against an empty model, or whether the new definition is pushed.
Current exposure
Since #2861 a running replica pulls auto-definitions/ on every poll, so the window is one poll interval (30s by default, about four intervals under steady run load) rather than until restart. Before that fix it would have applied until the peer restarted.
What to check
- Two replicas on a shared remote datastore. Create an enrollment token through replica A (
swamp worker token create <name> --duration 1h --server <A>). - Before replica B's next poll (or with a build that predates #2861), connect a worker to replica B with that token.
- Look at: B's server log for the redeem result;
auto-definitions/swamp/enrollment-token/<name>.yamlon A, on B and in the datastore, comparing theidfield; andswamp worker token list --server <A>afterwards.
What I tried
I attempted step 2 against the unfixed binary. swamp worker connect ws://<replica> --token <token> was rejected with HTTP 401, and the server logged "WebSocket auth rejected: no token provided", on both replicas, including the one that created the token. So my worker connection setup was wrong and the attempt says nothing about redeem. The definition's id was unchanged on A and in the datastore afterwards, which is expected given redeem never ran.
Expected
A redeem on a replica that lacks the token's definition fails cleanly (or pulls the definition first), and never creates or pushes a second definition for the same token name.
Open
No activity in this phase yet.
Sign in to post a ripple.