Lab #3208: Activity projector fails with ZooKeeper "Bad version" error on swamp.score

Description

The activity projector job on swamp-club fails intermittently with a ClickHouse Keeper / ZooKeeper error when writing to the replicated swamp.score_grants table:

Code: 244. UNEXPECTED_ZOOKEEPER_ERROR: Bad version for swamp.score_grants replica clickhouse-production-01

It happens during Keeper leader elections or periods of high Keeper latency. The cluster self-heals quickly on another node, but the projector run fails. Until the projector recovers or is restarted, activity feed data can be delayed or end up inconsistent.

Detection

Alert: [P3] "Activity projector erroring (new signal) on swamp-club" (category: application-reliability)
Signal: [REDACTED-SECRET-1]
First seen: 2026-10-07T10:13:34Z (1 occurrence at time of alert)

Steps to reproduce

The projector writes to swamp.score_grants on clickhouse-production-01.
A Keeper leader election happens or Keeper latency spikes during the write.
The insert fails with UNEXPECTED_ZOOKEEPER_ERROR: Bad version, and the projector run errors out.

Expected

The projector handles transient Keeper errors during writes to swamp.score_grants, so feed data is neither lost nor delayed.

Environment

swamp-club production, ClickHouse replica clickhouse-production-01
Related: the Keeper instability itself is still under investigation

Docs: choosing where a new workspace is created (--path, start page Location field, workspaces.dir)

Docs: serve-flags.md admin/allowed-user resolution is stale (pinned names, oauth:manage for lookups, plain usernames only)

Manual: working-with-data copy button copies a truncated data query

Dashboard: a live run page polls a full workflow.run.search every 2.5 s, and time-range presets refetch the run summary every minute

Extension push: a range dependency is sent to OSV as the range, not a version

Docs: manual quotes model wording that swamp-club#3437 changes to behavior

Activity scoring aliases only behavior run, not other swamp behavior subcommands

CI review gates read stale reviews when a passing review falls back to bold headings after Bash #-line denials

CI adversarial review gate keeps blocking on a stale CHANGES_REQUESTED review when the new review's heading is formatted differently

Datastore rework Phase 4: real-endpoint check of Phase 4A's replication figures on a hosted S3-compatible service (spike)

Configurable storage tiers: runs, outputs, data versions and audit move through an ordered list of user-defined tiers, and stay queryable across them

serve keeps a worker whose enroll finished after the worker timed out, then refuses the real worker as "already connected" and dispatches into the dead session

A manual worker verify racing an enrollment probe can leave a healthy worker unverified until it reconnects

literal() output in a step's inputs logs repeated Refusing to evaluate expression warnings

A global argument that evaluates to swamp-namespace expression text is silently dropped, and literal() cannot pass it through

serve: the per-IP WebSocket burst limit can't be configured, so workers behind one NAT IP throttle each other

Datastore rework Phase 4 G: S3 and GCS implement the writable datastore-wide control plane and keep _control/writer/ at the bucket root

Registry scorecard: dependency licenses follow the CLI's SPDX rules

extension push: read dependency licenses with the shared SPDX parser (OR, AND, WITH)

@swamp/aws/ec2 has not published since 2026.09.29.1: its 23.4MB archive's publish job is killed after ~14 minutes

bundle freshness ignores local modules outside the kind directory

extension pull: shared sources at the archive root are not installed

extension pull: safety analysis scans the bundle the extension runs

Teach defineBehavior in the extension skill now that swamp-testing 0.20261013.52 exports it

Registry UI and manual say Behaviors, keeping the stored models content type

verification: run the upgrade-path test in parallel, and skip the dynamic test for no-op bumps

S3/GCS datastores: write data version files create-only so a second writer cannot overwrite an existing version

codegen: AWS models emit CloudFormation patterns JavaScript cannot compile, so @swamp/aws/chime, glue and transfer fail to import

S3-backed serve: a poll that pulls one file invalidates the whole data catalog, so the next query re-reads every version

extension push: pinned deprecated and unpinned or range dependency specifiers become errors (after the two-week window)

Registry scorecard: audit the dependency version that ships, and JSR dependencies

extension push: audit the dependency versions the bundle actually ships (pinned facts, deno info resolution, JSR existence)

swamp-extensions: CI check that every manifest declares license: and ships a LICENSE file

extensions_search index is never created if the first attempt fails, until the process restarts

Rescore scores the newest version, but the scorecard reads the stable one

extension pull: answering y to a dependency's overwrite prompt silently skips that dependency

extension push --private: trust the registry's privateExtensions verdict instead of deciding from plan/trial

S3/GCS datastore sync: shared sync service lets one sync clear another's dirty marks and pull over unpushed files

swamp-uat: the SIGTERM drain test does not check that the run succeeded

stagecraft: a factory-wide read of work items waiting on a person (no studio, no per-item status loop)

stagecraft: rebind a work item to another tracker (mirrored built-in tickets to Jira)

s3-datastore: first push on a fresh cache prints 'Index merge failed' for a missing local .datastore-index.json

swamp-extensions: use explicit npm:/jsr: specifiers in root extensions/models

vercel/projects update: always resends the unchanged project name, which Vercel rejects with 409 "already exists"

stagecraft: studio state_test 'after its event stream reconnects' is timing-flaky under load

Show wait keys and pending signals to remote clients and the dashboard

Activity projector fails with ZooKeeper "Bad version" error on swamp.score_grants during keeper instability

@swamp/s3-datastore and @swamp/gcs-datastore: implement datastoreControlPlaneStore() for the format marker read

s3-datastore: model-scoped pullChanged lists and walks the whole namespace

Datastore sync: a no-op push after a model-scoped pull arms the remote commitSeq, hiding other hosts' commits until the next commit

extension safety analyzer does not warn on node:child_process or cluster subprocess spawning

extension push: a swamp-quality-ignore directive after a comment holding an apostrophe is silently ignored

Model types: decide whether names like @/@x/y or a+b/c should be legal (type grammar)

seed:collective cannot produce a superseded personal trial or a returned trial, so the UAT cannot cover them

serve HA: pull a token's definition on an auth miss so a peer accepts a new token at once

extension push: accept attestation-backed adversarial-review evidence (lookup by extension and content hash), generated provenance, and carry the evidence to the registry at confirm (design: Lab #3023)

Decide a stricter naming rule for workflow step and job names

digitalocean codegen leaves an orphaned security_secret.ts model file after its endpoint left the spec

s3-datastore/serve: readResource without a version returns an older version; the latest marker is rewritten backwards

Publish agent-facing projections of the manual: /llms.txt, .md pages, and llms-full.txt

Let agent-runner drive more agent CLIs (Kilo Code and others)

s3-datastore: every fast-path miss re-downloads every _index shard, so a busy serve pulls the whole index every poll (~230 GB/day S3 egress)

Feedback: Swamp as an approval-gated control plane for a small fleet

S3/GCS datastore: a push that fails after uploading drops its recorded deletes, so the retry brings deleted data back

Tracking: test baseline required before the datastore refactor (commit-log design)

serve: define and supply the collective and owner grant condition variables

No first-class workflow primitive for spawning a permission-scoped agent session

Decorative animations outside the Lab still repaint on the main thread

s3/gcs datastore: pullChanged overwrites dirty, unpushed cache files

Codegen/AWS: create-only required fields block list/get/delete in generated AWS models

End a collective's trial when it begins a paid subscription

Relationships

#3208 Activity projector fails with ZooKeeper "Bad version" error on swamp.score_grants during keeper instability

Description

Detection

Steps to reproduce

Expected

Environment

Open