Skip to main content
← Back to list
01Issue
BugShippedSwamp CLIPublic
Assigneesstack72

Relationships

#1882 serve HA: grant/group mutations are never pushed to the shared datastore — replication pull (#2305) has nothing to pull

Opened by stack72 · 8/27/2026· Shipped 8/28/2026

Summary

Follow-up to swamp-club#1876 (marked shipped via swamp#2305). #2305 correctly added the pull half of HA grant replication — handleAccessReload pulls remote access data before rebuilding, and an AccessDataPoller (30s) runs on every instance. Verified on releases d3c5e72d and ab2e36e9: both mechanisms activate.

But grants still do not replicate, because the push half does not exist: no grant-creation path writes swamp/grant (or swamp/group) data to the shared remote datastore. The bucket never contains any grant keys, so the poller and the reload-pull are permanently starved. #2305's stated premise — "grant data IS pushed to the shared remote datastore via the normal acquireModelLocks pull-lock-execute-push cycle" — does not hold in practice; this issue carries the corrected context.

Steps to reproduce

Two swamp serve instances (distinct repo dirs), token mode, --admins user:stack72, shared @swamp/s3-datastore namespace (ministack), per-instance pre-boot tokens (swamp#2197):

swamp access grant create --subject user:member --allow run --on "workflow:*" --server <A> --token <admin-A>
# inspect the bucket           → ZERO objects matching "grant" (37 objects total)
swamp access reload --server <B> --token <admin-B>   # succeeds, pulls nothing
swamp workflow run single-step --server <B> --token <member-B>
# → Access denied: user:member does not have 'run' on workflow:single-step

Also reproduced with grant create via the local CLI on A's repo while serve runs — still zero grant keys in the bucket. Waiting 90s (three AccessDataPoller cycles) changes nothing.

Control experiment (proves the sync layer is fine)

Server tokens minted by the CLI pre-boot DO land in the bucket: auto-definitions/swamp/server-token/*.yaml, data/swamp/server-token/<id>/token-main/..., definitions-evaluated/swamp/server-token/*.yaml. The datastore sync handles access-family models correctly when the write path actually pushes — the defect is exclusively that grant/group mutations made while serve is running (serve access handlers and local CLI alike) never push.

Expected

A grant or group mutation on any instance is written through to the shared datastore (definitions + data, like the pre-boot server-token writes), so #2305's reload-pull and AccessDataPoller can replicate it to peers. After the create on A: bucket contains the grant under data/swamp/grant/...; reload --server B (or one poller cycle) makes B enforce it.

Impact

Same operator impact as #1876: HA authorization cannot be managed centrally — a grant "added to the cluster" applies only to the instance that took the write, silently. The swamp-uat suite pins this in tests/cli/serve/access/ha_test.ts (both characterizations re-verified green — i.e. still broken — on ab2e36e9); the pins flip the moment create→push→pull works end to end.

Environment

swamp 20260827.221315.0-sha.ab2e36e9 (also d3c5e72d); 2-instance cluster via swamp-uat withServeCluster (ministack S3, fresh namespace per run, fast heartbeats); macOS. Full bucket evidence also posted as a comment on swamp-club#1876 (2026-08-27).

02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 2 MOREREVIEW+ 7 MOREPR_MERGED+ 2 MORESESSION_SUMMARIZED

Shipped

8/28/2026, 12:00:02 AM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack728/27/2026, 11:01:32 PM

Sign in to post a ripple.