Skip to main content
← Back to list
01Issue
FeatureOpenPlatformPublic
AssigneesNone

Relationships

#4353 Package everything on ops.swamp-club.com; stop direct server edits

Opened by swamp_lord · 11/3/2026

Problem

ops.swamp-club.com's model and workflow definitions are created, edited and deleted directly on the server (swamp model|workflow create|edit with SWAMP_SERVE_URL), and only then copied into platform-swamp by hand. Git is a record that can fall behind, and nothing stops the copies drifting. Everything on ops should arrive from a reviewed, released place.

Inventory (2026-11-03)

Workflows: 161 on ops. 127 came from extensions; 34 were created directly on the server.

  • 31 have a copy in ops/workflows/:
    • @swamp/platform-ops/: gcp-instance-resize, install-bitbison-cluster, org-access-baseline, release-swamp-club-app, release-workload
    • axiom-weekly-report
    • do-spaces-archive-discover, do-spaces-archive-sync-all, do-spaces-archive-sync-bucket
    • dossier-app-publish, dossier-publish
    • fleet-healthcheck, fleet-otel-smoke
    • github-event-router, github-noop, lab-issue-lifecycle, nightly-extension-generation
    • ops-incident-investigate-axiom, ops-incident-refresh-discord
    • ops-release-authorize-action, ops-release-complete-action, ops-release-investigate
    • ops-signal-ingest, ops-signal-reconcile
    • platform-vault-anthropic-smoke, prod-dns-artifacts-edge-gcp-canary, regenerate-models
    • runtime-vaults-audit, runtime-vaults-ensure
    • swamp-release-archive, tailscale-host-ssh
  • 3 have no copy in the repo: deno-emudhra-smoke, uat-club-same-vm-comparison, do-spaces-archive-clickhouse-backups (in PR #293).

Models: 124 on ops, all definitions created on the server. Extensions ship types, not instances.

  • 81 have a copy in the repo.
  • 43 don't:
    • 34 uat-runner-* (cli 1–7, club, datastore filesystem/gcs/s3 1–4, quality, serve 1–7, vault-integration, worker 1–5)
    • project-network-swamp-club-{corp,prod,sandbox}
    • runtime-vault-shared, serve-runtime-maintenance, swamp-release-artifacts, worker-probe
    • do-spaces-inventory (in PR #292), gcs-clickhouse-backups (in PR #293)
  • Not counted: models that workflows create as they run (gcp-sa-*, tailscale-host-*, gcp-iam-change-*) don't appear in model search; they come from their workflows.

Grants: only platform, ops-discord-bot and prod-dns are delivered from ops/grants/ through the VM metadata (infra/ops-vm/render_instance.ts, startup.sh). ops-self-gc, runtime-upgrade and swamp-club-release aren't.

Plan

  1. Lock down direct edits (first, so it gets no worse). Serve grants deny write on workflow:* and model:* to people and agents. run and read are unchanged. Only the release pipeline's identity can still write. Schedules and webhooks are unaffected. Check before relying on it:

    • that a deny wins over a serve admin;
    • that workflow runs which create their own models aren't counted as write.
  2. Workflows into packages. Move each of the 34 into the package it belongs to:

    • ops workflows → @swamp/platform-ops;
    • archive workflows → @swamp/do-spaces-archive;
    • runtime vaults → a package of their own.

    Each change includes scoped names and updated callers (schedules, webhooks, workflows calling workflows), then deletes the server-created copy. Start with the archive and its new ClickHouse backups cron.

  3. Models from git, applied by the pipeline. Apply ops/models/ from main in the release pipeline, with a drift check reporting anything on ops that doesn't match git. Adopt or delete the 43 unrecorded models. If an extension manifest can carry model instances, use that instead.

  4. Grants from git. Deliver every ops/grants/ file, not just three, and drop the ones that exist only on the server.

Done when

  • Nothing on ops is created or changed outside the release pipeline.
  • Every workflow and model on ops has a home in git.
  • The drift check comes back clean.
02Bog Flow
◉OPEN○TRIAGED○IN PROGRESS○SHIPPED

Open

11/3/2026, 4:30:54 PM

No activity in this phase yet.

03Sludge Pulse

Sign in to post a ripple.