#146

feat: add support for Nix via a flake

s12d ago

← Back to list

01Issue

BugOpenSwamp Club

AssigneesNone

Relationships

⊘ blocks #929↔ sibling #925

#927 Activity projector runs on both telemetry replicas — same-day chunks can double-grant (grant_id collides but granted_at differs, ReplacingMergeTree never dedups)

Opened by keeb · 7/2/2026

Found during #925 planning (adversarial review of the projector stage).

Setup: prod telemetry-api runs 2 replicas, and SCORE_GRANTS_CUTOVER is set at the deployment level — so startActivityProjector (services/telemetry/main.ts:110) runs on BOTH replicas, each on its own unsynchronized 60s tick. Same for startIdentityMapSync (harmless there — ReplacingMergeTree, duplicate work only).

The race: activity-asof-projector.sql is one INSERT INTO swamp.score_grants ... SELECT whose granted CTE reads current units_before from the ledger. Two replicas whose ticks overlap can both read the same units_before for a grain and both insert a chunk with the SAME deterministic grant_id (keyed on units_before). For PAST days that's fine — granted_at = toDateTime64(day) is deterministic, so the ReplacingMergeTree key (distinct_id, granted_at, grant_id) collapses them. But TODAY's chunks carry granted_at = now64(3), which differs per replica → both rows survive FINAL → the amount is counted twice, permanently (next tick sees units_before doubled and the cnt > units_before guard just goes quiet — no clawback, the inflation sticks).

Why it hasn't bitten (visibly) yet: the race window is one overlapping tick at low event volume; today's figura parity check (1510=1510) predates much concurrency. It gets more likely as volume grows.

Fix directions (pick one):

Single-runner gate: Mongo lease/leader election so only one replica projects (the trace_control polling infra shows the desired-state pattern; a TTL lease doc is enough), or split the projector into its own single-replica deployment.
Make granted_at deterministic for today's chunks too (e.g. derive from (day, units_before)) so the RMT key collapses duplicates — but that changes the 'live stamp moment' semantics the council wanted.

Blocks the parity gate for the #925 Stage-5 projector work — should land before or with it. Related: epic #847.

02Bog Flow

Open

7/2/2026, 9:45:10 PM

No activity in this phase yet.

03Sludge Pulse

keeb linked related to #9257/2/2026, 9:57:21 PM

keeb linked blocked by #9297/2/2026, 9:57:23 PM

Scorecard: beta/rc-only extensions show "not scored" despite successful scoring (GET /score resolves only latestVersion)

`swamp extension search --channel beta|rc|stable` ignores the filter — all channels + the no-flag default return an identical result set

Beta-channel extensions are never scored — perpetual 'Scoring is taking longer than expected'

feat: allow models to call other models mid-execution via context.runModel()

swamp-club: enable OAuth provider for swamp serve collective-based admission

extension update/install reports new version while on-disk pulled files stay old

command/shell: step with timeout + a surviving child process reports exit -1 despite command exiting 0

swamp serve startup/shutdown is very slow on large repo and lacks diagnostics

Vault push log overstates what left the machine: Pushed N vault file(s) to datastore counts every dirty file, not just vault files

serve-auth: mode: oauth — device grant client + collective-based admission

serve-auth: enable OIDC Provider + Device Authorization plugins in swamp-club

Orphaned data (migrated-away/deleted model) can't be GC'd or deleted

Docs: document swamp data prune (orphaned-data reclamation)

UAT: CLI coverage for swamp data prune (orphaned-data reclamation)

Session credential revocation scans entire credential map on every dispatch completion

Docs: update worker-commands.md for --all flag and JSON output change

datastore migrate-index aborts on hardcoded 30s S3 requestTimeout for large shards

swamp worker list: hide disconnected workers by default, add --all flag

swamp serve should survive bad extension failures

Worker fleets phase 7: how-to guides and explanation docs

swamp serve: catch unhandled promise rejections from extensions

[@adam/cfgmgmt] SSH helper can crash swamp serve via unhandled finally() rejection

Docs: add --server-token flag to worker connect reference

Add --server-token to worker connect for authenticated orchestrators

Docs: document --trusted-hosts flag for swamp serve

Document SWAMP_SERVE_EXTRA_HEADERS for reverse proxy/tunnel setups

Add --trusted-hosts flag to swamp serve for Docker/Kubernetes worker connections

Allow swamp clients to send custom headers when connecting to swamp serve

Docs: update serve-flags.md default queue timeout from 60s to 10m

Worker fleets phase 6: raise default queue timeout to 10 minutes

Worker fleets phase 4b: capacity slots and concurrent dispatch

MethodContext docs/comments mention context.putSecret but runtime exposes vaultService.put

Docs: add --concurrency flag and SLOTS column to worker-commands reference

install.sh still points users to systeminit/swamp

Support user systemd units for 'swamp serve daemon enable'

swamp serve scheduled execution runs steps in an isolated data scope: readResource sees empty state, writeResource never reaches the model-scope chain — stateful scheduled workflows duplicate instead of deduplicating

Support installing swamp via mise GitHub backend

Worker fleets phase 4a: dispatch runner extraction — child process per dispatch

Docs: add swamp worker daemon enable/disable/status to worker-commands reference

Worker fleets phase 3 PR2: swamp worker daemon enable

Docs: add drain, lifecycle policies, and exit-code contract to references

Worker fleets phase 3 PR1: drain state machine, lifecycle policies, signal handling

Leaderboard search excludes ghosts — a visible ghost can't be found by its alias

issue-lifecycle: plan lifecycle entry silently dropped when summary exceeds 1000 chars

Docs: add worker verify, --verify-on-enroll, and env-var config to references

Labs - Filters do not apply to Blocked status and Billing Tier category

Worker fleets phase 2 PR2: env-var config, swamp worker verify, --verify-on-enroll

Docs: add --type flag to issue edit reference and how-to

Docs: update manual with whole-field expression syntax for workflow task inputs/globalArgs

Allow changing an existing issue to security/confidential after submission

Allow dynamic object expressions for workflow task inputs/globalArgs

Docs: update enrollment token references for fleet tokens (--max-enrollments)

Worker fleets phase 2 PR1: fleet tokens (--max-enrollments, bindings, worker naming)

$identify events are never consumed — anonymous web history never binds to the account

Worker fleets phase 1 PR3: boot reconciliation and integration test

Login success still shows login tip

Support confidential visibility for non-security issues

Fine-grained scopes for swamp.club access tokens

Add associative info to user profiles: collective membership + collaborative contributions

Remove comments from /feed and redesign the page

feed_post_score_events queue grows unboundedly (no TTL, no delete-on-delivery)

Make paid subscribers visually stand out (badge, profile popup summary)

Relax the profanity filter

Giphy integration via /giphy command

@-mention autocomplete for users when writing lab ripples

Worker fleets phase 1 PR2: step_queued events, pending-dispatch model, queue introspection

Docs: add swamp worker queue to worker-commands reference

/lab shipped list is sorted by issue number, not ship date

Docs: update manual for ephemeral data and step-output dependency deferral

Docs: update remote execution references for elastic queueing

Support dynamic host discovery from external sources

Worker fleets phase 1: elastic queueing

catalog backfill should support scoped population for targeted lookups

Docs: document global skill sync behavior for custom tools in manual

swamp update syncs global skills to default agent directories instead of configured custom Pi skillsDir

coerceToSuffix does not sanitize path-unsafe characters, causing forEach items and vary dimensions with / to produce invalid data instance names

Badge for activity on other people's issues

swamp serve: add @keychain=<service>/<account> indirection for secret args on macOS

swamp serve: eliminate wrapper scripts via @vault= secret refs and native startup hooks

Automate mechanical frontmatter fixes in the fixing stage