feat(clickhouse): tracked archive→CH backfill tooling (backfill.sql)

The cron scheduler reads from .swamp/workflows-evaluated/ but does not detect when the source workflow YAML changes. If a workflow is updated on disk (e.g. modelIdOrName replaced with modelType/modelName, inputs added), the cron keeps running the stale cached copy until swamp workflow evaluate is run manually.

Expected: the scheduler re-evaluates the workflow definition when it detects the source file has changed (mtime or content hash), or at least on each cron tick since the cost is low.

Observed: source workflow changed from modelIdOrName to modelType/modelName pattern. Cron ran the old cached definition for 60+ minutes, failing every run with Unknown model type, until swamp workflow evaluate was run manually.

02Bog Flow

Shipped

6/24/2026, 6:35:20 PM

Click a lifecycle step above to view its details.

03Sludge Pulse

stack72 assigned stack726/19/2026, 5:46:51 PM

stack72 marked as blocked6/19/2026, 11:04:11 PM

stack72 commented 6/19/2026, 7:19:15 PM

Triaging this issue — attempted to reproduce the staleness behavior but couldn't trigger it across several scenarios (built-in types, local extension types, switching from modelIdOrName to modelType/modelName). In every case, the cron re-read the workflow from disk on the next tick and used the updated definition.

Could you share a few more details about the setup where you observed the 60-minute staleness?

Which model type was the workflow referencing? (e.g., @namespace/type, a local extension, a pulled extension?)
Was swamp workflow evaluate run before starting serve? The issue mentions .swamp/workflows-evaluated/ — wondering if the evaluated cache played a role.
Were there CEL expressions in the workflow that needed evaluation, or were all fields literal values?
Was the workflow in workflows/ (primary repo directory) or in extensions/workflows/?

One thing I did find during reproduction: the serve log reports schedule_completed even when the underlying workflow run actually failed (status: failed in the run YAML). If the runs were silently failing with "Unknown model type" but the log said "completed", that could look like the old definition was still running. Could that match what you observed?

bixu commented 6/22/2026, 2:41:26 PM

If the runs were silently failing with "Unknown model type" but the log said "completed", that could look like the old definition was still running. Could that match what you observed? << could be, but unfortunately the tick-based (cron style) invocation was just too weird for my use case and I switched to TailScale Funnel and webhooks, so I don't have any good way to fill in more detail. TBH I don't have any strong use-cases right now for cron-style workflows (would be more likely to orchestrate that with GH Actions as a wrapper.

stack72 commented 6/24/2026, 6:33:07 PM

We weren't able to reproduce the original staleness behavior across several scenarios (built-in types, local extension types, switching from modelIdOrName to modelType/modelName). In every case, the cron re-read the workflow from disk on the next tick and used the updated definition.

However, during investigation we found a real bug: swamp serve logs schedule_completed even when the underlying workflow run actually fails. The root cause is that when workflowRun() yields an error event (e.g. "Unknown model type") without a completed event, the scheduled execution service incorrectly falls through to the success path.

We've repurposed this issue to fix that bug — PR https://github.com/swamp-club/swamp/pull/1673 inverts the completion check to require explicit succeeded status, so all other outcomes (explicit failure, error events, missing terminal events) now correctly emit schedule_failed with the actual error message.

stack72 commented 6/24/2026, 6:35:30 PM

Thanks @bixu for reporting this! The fix has been merged and a release is on its way. We appreciate your contribution to swamp.

feat(clickhouse): tracked archive→CH backfill tooling (backfill.sql)

feat(clickhouse): idempotent DDL migration path for running prod (#859 deliverable 2)

chore(clickhouse): retire S3-backed v1 + s3 objects after #859 cutover

Decouple prod ClickHouse from S3 (drop storage_policy=s3_main) + add a DDL migration path

Epic #847 · Unit 6: Document the Mongo-vs-ClickHouse storage-architecture split in scoring.md

Epic #847 · Unit 5: ClickHouse materialized-view projections + atomic leaderboard read-flip + delete Mongo OLAP

Epic #847 · Unit 4: Stream confirmed grants into ClickHouse score_grants (ReplacingMergeTree)

Epic #847 · Unit 3: Migrate the 5 recompute contributions to per-event grants; delete the recompute path

Epic #847 · Unit 2: score_grants append-only ledger write-model in Mongo (shadow, no read flip)

Epic #847 · Unit 1: Land the ClickHouse projection foundation (schema + init SQL + compose service)

Global skills should auto-sync when binary version advances

autoGc emits auto_gc_completed event on --json stdout, breaking single-parse consumers

Extension publish score is non-monotonic: yanking versions lowers a user's score

Live Swamp Club event console on /feed — scrolling stream of all non-sensitive events

Docs: add --ws-idle-timeout to serve flags reference

copy/rsync ignores transport extraOptions (and proxyCommand), unlike exec/script

Remove feed comments — consolidate discussion in Discord

Make serve WebSocket idle/keepalive timeout configurable (untunable default aborts runs when serve's loop briefly blocks)

Docs: update extension info reference with content metadata output

serve startup time regression: synchronous catalog init delays WebSocket listener by ~4.5 minutes

Expose run/job/step identifiers as SWAMP_* env vars + CEL values, and template placement selectors (extends #331's run.id)

fix: add .namespace.json to isInternalCacheFile() in datastore extensions

docs: document swamp serve daemon enable/disable/status subcommands

docs: document execution cancellation commands and cancelled status

docs: document autoGc config option for automatic garbage collection

Docs: document @env= and @file= webhook secret indirection in swamp-serve reference

fix: datastore sync --push deletes the namespace registration manifest (canonical namespace flow un-registers itself)

docs: bundled swamp agent skill lacks datastore-namespace guidance (giga-swamp)

data query --select crashes on BigInt: "Do not know how to serialize a BigInt" when CEL size() reaches the JSON renderer

Intra-namespace write concurrency: whole-index sync under the lock serializes fan-out workloads (split from shipped #666)

Telemetry recoverOrphaned startup race with multiple replicas (created_at-based)

Telemetry retry/failed path has the same non-atomic claim as #820

Batch / prefix delete for swamp data delete (single lock acquisition)

Surface extension type+method detail in CLI to eliminate expensive discovery loops

Skill guides lack progressive reveal boundaries — agents over-read by 4x

Opt-in automatic garbage collection for datastore data

UAT: swamp workflow evaluate/run with forEach dynamic workflowIdOrName targets

Telemetry watcher has no replica coordination: N replicas double-process the same batch (non-atomic find→updateMany claim)

Telemetry drain still capped ~80-100/s in prod: per-username full-history re-aggregate is O(users) sequential per batch (deferred #817 fix #4)

Telemetry ingest is consumer-bound: counter & stats dedup via O(N) sequential insertOne, throughput stuck ~20 events/s regardless of BATCH_SIZE

Resolve dynamic workflow task targets inside forEach

Leaderboard and profile streak not reporting

Same-namespace writers fully serialize on the per-namespace lock — could maintenance/append writes avoid holding it?

Could method-summary report artifacts get a default retention cap? They grow to dominate the datastore manifest

Docs: update vault inspect output in manual reference

Execution cancellation: abort stuck workflow runs and model method runs, bulk cleanup, and daemon-restart reaping

Docs: document .? optional select for null-safe CEL data access

Optional scheduled / automatic datastore GC (retention-policy-driven pruning)

Notify issue author/participants on ripples & status changes — with Discord bot DM as a delivery channel

Batch step 2 of enrichAuthorPlans (per-collective subscription reads)

Datastore should fail fast on unresolvable credentials instead of stalling on the AWS provider chain

Add SWAMP CLUB wordmark logo next to sc-mark.png in TRADEMARKS.md

Add SWAMP CLUB wordmark logo next to sc-mark.png in TRADEMARKS.md

pushChanged does not implement absence-on-disk deletion (markDirty contract rule #2)

pushChanged does not implement absence-on-disk deletion (markDirty contract rule #2)

Add `vault delete` support to @swamp/aws-sm extension

Add `vault delete` support to @swamp/azure-kv extension

Add `vault delete` support to @swamp/1password extension

Leaderboard window baseline: 90-day cutoff zeroes returning-dormant users (latent, 0 impact today)

swamp data gc prunes the catalog but never deletes objects from S3 datastores (markDirty hook not wired) — sync manifest never shrinks

SKILL.md Common Commands: model type search uses wrong command and syntax

SKILL.md Common Commands: model create uses wrong @<type> prefix

swamp issue bug times out posting to the Lab while swamp-club.com returns HTTP 200

telemetry stats fatally fails to load an installed datastore extension (auto-resolve path); all other commands load it fine

tf plan: FETCH_BUNDLE PAGE_FETCH_ERR / NO_STATES on cleanup-only plan (no resource changes)

Add deleteResource to MethodContext and document dataRepository.delete in skills

Homebrew formula

Yank semantics inconsistent: all-versions-yanked acts as a free hidden/private extension; extension-level yank hard-blocks re-push

Extension search returns edit-distance noise for short queries ("asdl" → "AWS DEADLINE")

workflow resume holds the global lock across the resumed step, deadlocking any datastore op the step performs

Trajectory chart: current-day x-axis label is clipped at the right edge

Telemetry not synced to swamp-club: local queue accumulating ~3 days despite valid auth

extension pull serves a stale version that disagrees with search (honors a legacy per-extension serverUrl)

serve --webhook usage string makes <header> look optional for generic scheme

serve: webhook scheme not surfaced in startup event, health endpoint, or log line

Slack webhook pre-body gate only checks signature header, not timestamp

Dead code: verifySignature in webhook.ts superseded by verifier abstraction

extension source: install skills from source-path extensions

Data-driven webhook signature verifiers (avoid a code change + release per provider)

swamp.club extension view: multi-line code fence in manifest description renders each line as a separate inline code span

Single global datastore lock serializes unrelated writes across all repos/namespaces