Skip to main content
← Back to list
01Issue
BugOpenSwamp CLIPublic
AssigneesNone

Relationships

#3074 serve HA: worker enrollment on a replica that lacks the enrollment token's definition may create a second definition (unverified)

Opened by hammz · 10/6/2026

Summary

Found by code reading while fixing #2481 (swamp-club/swamp#2861). This is not reproduced. It needs someone to confirm or rule it out.

A worker enrollment token created through one swamp serve replica has its definition written to auto-definitions/ on that replica. If a worker then enrolls against a different replica before that replica has pulled the definition, the redeem may auto-create a second definition with the same name and a new id, instead of failing.

Why I think so

  • WorkerGateway redeems through its default model-method runner (#defaultRunModelMethod in src/serve/worker_gateway.ts), which calls modelMethodRun with both a type argument (ENROLLMENT_TOKEN_MODEL_TYPE.normalized) and a definition name. That is direct type execution, which creates the definition when none exists under that name.
  • Token records, by contrast, are found by scanning data for the type (#scanTokenRecords), which does not need the definition.
  • So on a peer without the definition, redeem could run against a freshly created definition whose id has no token data. If that new definition is then pushed, it would land at the same path as the original (auto-definitions/swamp/enrollment-token/<name>.yaml) with a different id, and the original token data would no longer be reachable by name.

Each step after the first bullet is inference. I have not confirmed that direct type execution auto-creates in this path, what redeem does against an empty model, or whether the new definition is pushed.

Current exposure

Since #2861 a running replica pulls auto-definitions/ on every poll, so the window is one poll interval (30s by default, about four intervals under steady run load) rather than until restart. Before that fix it would have applied until the peer restarted.

What to check

  1. Two replicas on a shared remote datastore. Create an enrollment token through replica A (swamp worker token create <name> --duration 1h --server <A>).
  2. Before replica B's next poll (or with a build that predates #2861), connect a worker to replica B with that token.
  3. Look at: B's server log for the redeem result; auto-definitions/swamp/enrollment-token/<name>.yaml on A, on B and in the datastore, comparing the id field; and swamp worker token list --server <A> afterwards.

What I tried

I attempted step 2 against the unfixed binary. swamp worker connect ws://<replica> --token <token> was rejected with HTTP 401, and the server logged "WebSocket auth rejected: no token provided", on both replicas, including the one that created the token. So my worker connection setup was wrong and the attempt says nothing about redeem. The definition's id was unchanged on A and in the datastore afterwards, which is expected given redeem never ran.

Expected

A redeem on a replica that lacks the token's definition fails cleanly (or pulls the definition first), and never creates or pushes a second definition for the same token name.

02Bog Flow
◉OPEN○TRIAGED○IN PROGRESS○SHIPPED

Open

10/6/2026, 3:06:39 PM

No activity in this phase yet.

03Sludge Pulse

Sign in to post a ripple.