feat(clickhouse): tracked archive→CH backfill tooling (backfill.sql)

swamp 20260609.232501.0-sha.873563f2 (newer build 20260616.195738.0-sha.2c8bba58 available but not yet installed — unable to confirm whether fixed there; reproducible on the installed version)
Datastore: @swamp/s3-datastore v2026.06.03.2 (S3-compatible / MinIO backend)

Steps to reproduce

Run swamp data gc --dry-run against a repo with accumulated data versions.

Expected

Per swamp help data gc, --dry-run should 'Show what would be deleted without deleting'. No mutation.

Actual

The command ran a real GC and reported:

INF data·gc GC complete: deleted 3430 expired items, 66382 excess versions reclaimed (137588260 bytes)
INF datastore·sync Pushing changes to "@swamp/s3-datastore"...

~70k versions / ~131 MB were reclaimed for real — the flag was silently ignored and a destructive operation proceeded.

Impact

--dry-run is the safe primitive operators reach for before a destructive GC. Silently executing the real operation under a dry-run flag is a footgun that can cause unrecoverable data loss. The flag should either gate all mutation or the command should error if the flag isn't honored.

Note

Also observed: data gc emitted dozens of identical WRN swamp·domain·data·lifecycle Ephemeral lifetime is not yet implemented lines (one per item) — likely should be deduped/summarized, but secondary to the dry-run bug.

02Bog Flow

Shipped

6/17/2026, 6:29:16 PM

Click a lifecycle step above to view its details.

03Sludge Pulse

stack72 assigned stack726/17/2026, 5:01:29 PM

mgreten commented 6/17/2026, 4:18:29 PM

Confirmed still present on 20260616.195738.0-sha.2c8bba58

Re-ran swamp data gc --dry-run after swamp update to the newer build. The flag is still ignored — it performed a real destructive GC:

INF data·gc GC complete: deleted 3433 expired items, 66394 excess versions reclaimed (137629018 bytes)
INF datastore·sync Pushing changes to "@swamp/s3-datastore"...

Additional finding: reclaimed versions are not actually deleted from the S3 backend

The counts are nearly identical across two separate gc runs (66382 then 66394 excess versions, ~131 MB each). The durable object count in the S3 datastore did not drop between runs (data/ stayed at ~217k objects, index stayed ~110 MB). This suggests GC reports reclaiming the local catalog/WAL but never persists the deletions to the S3 backend — so every run re-reports the same backlog and the bucket never shrinks.

Net: (1) --dry-run still mutates, and (2) the mutation it performs doesn't reach the backend, so it neither dry-runs nor durably collects. Both reproduce on the latest build.

mgreten commented 6/17/2026, 4:21:36 PM

Confirming for the record: swamp update + repo upgrade did not resolve this

To be explicit beyond the earlier ripple — after updating the CLI to 20260616.195738.0-sha.2c8bba58 and running swamp repo upgrade, swamp data gc --dry-run still performs a real destructive GC (flag ignored), and the reclaimed versions still don't durably leave the S3 backend (same ~66k versions / ~131 MB re-reported on each run; bucket object count unchanged). Both behaviors reproduce on the latest build. Filing this note so the issue isn't assumed fixed by the version bump.

stack72 commented 6/17/2026, 6:29:27 PM

Thanks @mgreten for reporting this! The fix has been merged and a release is on its way. We appreciate your contribution to swamp.

mgreten commented 6/17/2026, 6:49:27 PM

Thank you!!!!!!!

mgreten commented 6/17/2026, 11:03:57 PM

Verified fixed on 20260617.212026.0-sha.396e0952 — thank you

Confirmed the --dry-run half of this is resolved on the shipped build:

INF data·gc GC dry run: would delete 3619 expired items, would reclaim 71018 excess versions (144202579 bytes)

Two good signals: the output now reads "GC dry run: would delete … would reclaim" (was "GC complete: deleted …"), and the durable object count did not drop across the run (266501 → 266527; the small increase is unrelated concurrent writes, i.e. it went up, not down). So the flag is now honored and the dry run mutates nothing.

One open question: is the secondary finding covered?

This issue also captured a second behavior (earlier ripple): a real data gc reported reclaiming ~66k versions / ~131 MB but the versions never durably left the S3 backend — the bucket object count didn't drop and the same backlog was re-reported on each subsequent run. The dry run now correctly predicts reclaiming ~71k versions, but I haven't run a real (non-dry) gc on this build to confirm the deletions actually persist to the backend now.

Could you confirm whether the shipped status covers just the dry-run flag, or also the backend-durability behavior? If the latter is fixed too, a real gc should shrink the bucket object count — happy to verify and report back if useful.

feat(clickhouse): tracked archive→CH backfill tooling (backfill.sql)

feat(clickhouse): idempotent DDL migration path for running prod (#859 deliverable 2)

chore(clickhouse): retire S3-backed v1 + s3 objects after #859 cutover

Decouple prod ClickHouse from S3 (drop storage_policy=s3_main) + add a DDL migration path

Epic #847 · Unit 6: Document the Mongo-vs-ClickHouse storage-architecture split in scoring.md

Epic #847 · Unit 5: ClickHouse materialized-view projections + atomic leaderboard read-flip + delete Mongo OLAP

Epic #847 · Unit 4: Stream confirmed grants into ClickHouse score_grants (ReplacingMergeTree)

Epic #847 · Unit 3: Migrate the 5 recompute contributions to per-event grants; delete the recompute path

Epic #847 · Unit 2: score_grants append-only ledger write-model in Mongo (shadow, no read flip)

Epic #847 · Unit 1: Land the ClickHouse projection foundation (schema + init SQL + compose service)

Global skills should auto-sync when binary version advances

autoGc emits auto_gc_completed event on --json stdout, breaking single-parse consumers

Extension publish score is non-monotonic: yanking versions lowers a user's score

Live Swamp Club event console on /feed — scrolling stream of all non-sensitive events

Docs: add --ws-idle-timeout to serve flags reference

copy/rsync ignores transport extraOptions (and proxyCommand), unlike exec/script

Remove feed comments — consolidate discussion in Discord

Make serve WebSocket idle/keepalive timeout configurable (untunable default aborts runs when serve's loop briefly blocks)

Docs: update extension info reference with content metadata output

serve startup time regression: synchronous catalog init delays WebSocket listener by ~4.5 minutes

Expose run/job/step identifiers as SWAMP_* env vars + CEL values, and template placement selectors (extends #331's run.id)

fix: add .namespace.json to isInternalCacheFile() in datastore extensions

docs: document swamp serve daemon enable/disable/status subcommands

docs: document execution cancellation commands and cancelled status

docs: document autoGc config option for automatic garbage collection

Docs: document @env= and @file= webhook secret indirection in swamp-serve reference

fix: datastore sync --push deletes the namespace registration manifest (canonical namespace flow un-registers itself)

docs: bundled swamp agent skill lacks datastore-namespace guidance (giga-swamp)

data query --select crashes on BigInt: "Do not know how to serialize a BigInt" when CEL size() reaches the JSON renderer

Intra-namespace write concurrency: whole-index sync under the lock serializes fan-out workloads (split from shipped #666)

Telemetry recoverOrphaned startup race with multiple replicas (created_at-based)

Telemetry retry/failed path has the same non-atomic claim as #820

Batch / prefix delete for swamp data delete (single lock acquisition)

Surface extension type+method detail in CLI to eliminate expensive discovery loops

Skill guides lack progressive reveal boundaries — agents over-read by 4x

Opt-in automatic garbage collection for datastore data

UAT: swamp workflow evaluate/run with forEach dynamic workflowIdOrName targets

Telemetry watcher has no replica coordination: N replicas double-process the same batch (non-atomic find→updateMany claim)

Telemetry drain still capped ~80-100/s in prod: per-username full-history re-aggregate is O(users) sequential per batch (deferred #817 fix #4)

Telemetry ingest is consumer-bound: counter & stats dedup via O(N) sequential insertOne, throughput stuck ~20 events/s regardless of BATCH_SIZE

Resolve dynamic workflow task targets inside forEach

Leaderboard and profile streak not reporting

Same-namespace writers fully serialize on the per-namespace lock — could maintenance/append writes avoid holding it?

Could method-summary report artifacts get a default retention cap? They grow to dominate the datastore manifest

Docs: update vault inspect output in manual reference

Execution cancellation: abort stuck workflow runs and model method runs, bulk cleanup, and daemon-restart reaping

Docs: document .? optional select for null-safe CEL data access

Optional scheduled / automatic datastore GC (retention-policy-driven pruning)

Notify issue author/participants on ripples & status changes — with Discord bot DM as a delivery channel

Batch step 2 of enrichAuthorPlans (per-collective subscription reads)

Datastore should fail fast on unresolvable credentials instead of stalling on the AWS provider chain

Add SWAMP CLUB wordmark logo next to sc-mark.png in TRADEMARKS.md

Add SWAMP CLUB wordmark logo next to sc-mark.png in TRADEMARKS.md

pushChanged does not implement absence-on-disk deletion (markDirty contract rule #2)

pushChanged does not implement absence-on-disk deletion (markDirty contract rule #2)

Add `vault delete` support to @swamp/aws-sm extension

Add `vault delete` support to @swamp/azure-kv extension

Add `vault delete` support to @swamp/1password extension

Leaderboard window baseline: 90-day cutoff zeroes returning-dormant users (latent, 0 impact today)

swamp data gc prunes the catalog but never deletes objects from S3 datastores (markDirty hook not wired) — sync manifest never shrinks

SKILL.md Common Commands: model type search uses wrong command and syntax

SKILL.md Common Commands: model create uses wrong @<type> prefix

swamp issue bug times out posting to the Lab while swamp-club.com returns HTTP 200

telemetry stats fatally fails to load an installed datastore extension (auto-resolve path); all other commands load it fine

tf plan: FETCH_BUNDLE PAGE_FETCH_ERR / NO_STATES on cleanup-only plan (no resource changes)

Add deleteResource to MethodContext and document dataRepository.delete in skills

Homebrew formula

Yank semantics inconsistent: all-versions-yanked acts as a free hidden/private extension; extension-level yank hard-blocks re-push

Extension search returns edit-distance noise for short queries ("asdl" → "AWS DEADLINE")

workflow resume holds the global lock across the resumed step, deadlocking any datastore op the step performs

Trajectory chart: current-day x-axis label is clipped at the right edge

Telemetry not synced to swamp-club: local queue accumulating ~3 days despite valid auth

extension pull serves a stale version that disagrees with search (honors a legacy per-extension serverUrl)

serve --webhook usage string makes <header> look optional for generic scheme

serve: webhook scheme not surfaced in startup event, health endpoint, or log line

Slack webhook pre-body gate only checks signature header, not timestamp

Dead code: verifySignature in webhook.ts superseded by verifier abstraction

extension source: install skills from source-path extensions

Data-driven webhook signature verifiers (avoid a code change + release per provider)

swamp.club extension view: multi-line code fence in manifest description renders each line as a separate inline code span

Single global datastore lock serializes unrelated writes across all repos/namespaces