Skip to main content
← Back to list
01Issue
BugShippedSwamp CLIPublic
Assigneesstack72

Relationships

#1876 serve HA: grants do not replicate across cluster instances — policy is repo-local despite the shared datastore

Opened by stack72 · 8/27/2026· Shipped 8/27/2026

Summary

In an HA cluster (two swamp serve instances, token mode, shared S3 datastore/control plane), a grant created through instance A is completely invisible to instance B: it never appears in B's grant list, and an explicit swamp access reload --server B does not pick it up. Authorization policy is effectively per-repo even though runs, heartbeats, and token secrets share the control plane — authentication is cluster-shaped, authorization is not.

Steps to reproduce

Two serve instances (distinct repo dirs) on one shared @swamp/s3-datastore namespace (ministack), token mode, --admins user:stack72; tokens minted per instance pre-boot (per swamp#2197 CLI-minted tokens do not verify cross-instance):

swamp access grant create --subject user:member --allow run --on "workflow:*" --server <A> --token <admin-A>
swamp access reload --server <A> --token <admin-A>     # member run on A now allowed
swamp access reload --server <B> --token <admin-B>     # succeeds...
swamp workflow run single-step --server <B> --token <member-B>

Actual

  • Run on B: Access denied: user:member does not have 'run' on workflow:single-step — even after B's reload.
  • swamp access grant list against B (server or local repo view) shows only B's own config grant; A's member grant is absent entirely (probed: A's repo lists 2 grants, B's lists 1).

Expected

Either grants replicate through the shared datastore so any instance's reload sees them (matching how ordinary model data flows across HA instances), or the HA docs/UX say clearly that grants are per-instance and must be provisioned on every repo (grants files are the natural vehicle).

Likely mechanism

Grants are model data, but their materialized definitions live in the instance's local .swamp/auto-definitions/swamp/grant/ — the policy snapshot loader scans grants via the repo's definition/data view, so a grant materialized on A's repo never enters B's scan even though the datastore is shared. (Hypothesis from swamp-uat probing; the observable contract is the invisibility above.)

Impact

Operators running the documented HA topology cannot manage authorization centrally: a grant added "to the cluster" silently applies to whichever instance took the write, and nothing surfaces the divergence. Combined with the shared token plane, a member can be allowed or denied depending on which instance the load balancer picks. Pinned as characterization tests in swamp-uat tests/cli/serve/access/ha_test.ts (GRANTS-TEST-PLAN.md §9#3) — including the working workaround (create the grant on each instance's repo + reload that instance); the tests flip when replication lands.

Environment

swamp 20260826.235928.0-sha.da5373d1; 2-instance cluster via swamp-uat withServeCluster (ministack S3, fresh namespace, fast heartbeats); macOS.

02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 2 MOREREVIEW+ 7 MOREPR_MERGED+ 2 MORESESSION_SUMMARIZED

Shipped

8/27/2026, 9:37:46 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack728/27/2026, 8:12:54 PM
Editable. Press Enter to edit.

stack72 commented 8/27/2026, 10:04:06 PM

Fix verification against the shipped binary (20260827.213713.0-sha.d3c5e72d) — still failing

Re-ran the swamp-uat HA grants suite after #2305 shipped. Both mechanisms activate ("Access data poller started (interval: 30s)" logged on both instances; reload succeeds) — but B still never sees A's grant, because the push side never happens: the pull has nothing to pull.

Evidence (2-instance cluster, shared ministack namespace)

  1. After swamp access grant create ... --server <A>: bucket listing contains zero keys matching grant (37 objects total). The grant exists only in A's local .swamp/.
  2. Contrast: the pre-boot CLI mints of server tokens DID push — the bucket holds auto-definitions/swamp/server-token/*.yaml, data/swamp/server-token/<id>/token-main/..., definitions-evaluated/swamp/server-token/*.yaml. So the sync layer replicates access-family models fine when the write path pushes.
  3. Also probed: grant create via the local CLI against A's repo while serve runs — still zero grant keys in the bucket.
  4. Consequently reload --server B pulls nothing (B's .swamp/data/swamp/grant/ does not exist afterward), B's grant list still shows only its own config grant, and enforcement on B denies.

Conclusion

The #2305 premise — "grant data IS pushed to the shared remote datastore via the normal acquireModelLocks pull-lock-execute-push cycle" — does not hold for grants created while serve is running, via either the serve access handler or the local CLI. The missing half is the push on grant/group mutation; AccessDataPoller + reload-pull are downstream of that and work as designed once data exists remotely (cannot confirm end-to-end until the push lands).

The swamp-uat characterization tests remain pinned to non-replication (tests/cli/serve/access/ha_test.ts) and will flip once create-→push-→pull works end to end.

Environment: swamp 20260827.213713.0-sha.d3c5e72d, 2× serve token mode, @swamp/s3-datastore on ministack, fresh namespace per run.

stack72 commented 8/27/2026, 11:00:00 PM

Follow-up filed as #1882: verification against releases d3c5e72d and ab2e36e9 shows the #2305 pull mechanisms activate but grant/group mutations are never pushed to the shared datastore, so replication still does not happen end to end. #1882 carries the corrected context (push side missing) and the bucket-level evidence.

Sign in to post a ripple.