Relationships
#4353 Package everything on ops.swamp-club.com; stop direct server edits
Opened by swamp_lord · 11/3/2026
Problem
ops.swamp-club.com's model and workflow definitions are created, edited and deleted directly on the server (swamp model|workflow create|edit with SWAMP_SERVE_URL), and only then copied into platform-swamp by hand. Git is a record that can fall behind, and nothing stops the copies drifting. Everything on ops should arrive from a reviewed, released place.
Inventory (2026-11-03)
Workflows: 161 on ops. 127 came from extensions; 34 were created directly on the server.
- 31 have a copy in
ops/workflows/:@swamp/platform-ops/: gcp-instance-resize, install-bitbison-cluster, org-access-baseline, release-swamp-club-app, release-workload- axiom-weekly-report
- do-spaces-archive-discover, do-spaces-archive-sync-all, do-spaces-archive-sync-bucket
- dossier-app-publish, dossier-publish
- fleet-healthcheck, fleet-otel-smoke
- github-event-router, github-noop, lab-issue-lifecycle, nightly-extension-generation
- ops-incident-investigate-axiom, ops-incident-refresh-discord
- ops-release-authorize-action, ops-release-complete-action, ops-release-investigate
- ops-signal-ingest, ops-signal-reconcile
- platform-vault-anthropic-smoke, prod-dns-artifacts-edge-gcp-canary, regenerate-models
- runtime-vaults-audit, runtime-vaults-ensure
- swamp-release-archive, tailscale-host-ssh
- 3 have no copy in the repo: deno-emudhra-smoke, uat-club-same-vm-comparison, do-spaces-archive-clickhouse-backups (in PR #293).
Models: 124 on ops, all definitions created on the server. Extensions ship types, not instances.
- 81 have a copy in the repo.
- 43 don't:
- 34
uat-runner-*(cli 1–7, club, datastore filesystem/gcs/s3 1–4, quality, serve 1–7, vault-integration, worker 1–5) - project-network-swamp-club-{corp,prod,sandbox}
- runtime-vault-shared, serve-runtime-maintenance, swamp-release-artifacts, worker-probe
- do-spaces-inventory (in PR #292), gcs-clickhouse-backups (in PR #293)
- 34
- Not counted: models that workflows create as they run (
gcp-sa-*,tailscale-host-*,gcp-iam-change-*) don't appear in model search; they come from their workflows.
Grants: only platform, ops-discord-bot and prod-dns are delivered from ops/grants/ through the VM metadata (infra/ops-vm/render_instance.ts, startup.sh). ops-self-gc, runtime-upgrade and swamp-club-release aren't.
Plan
Lock down direct edits (first, so it gets no worse). Serve grants deny
writeonworkflow:*andmodel:*to people and agents.runandreadare unchanged. Only the release pipeline's identity can still write. Schedules and webhooks are unaffected. Check before relying on it:- that a deny wins over a serve admin;
- that workflow runs which create their own models aren't counted as
write.
Workflows into packages. Move each of the 34 into the package it belongs to:
- ops workflows →
@swamp/platform-ops; - archive workflows →
@swamp/do-spaces-archive; - runtime vaults → a package of their own.
Each change includes scoped names and updated callers (schedules, webhooks, workflows calling workflows), then deletes the server-created copy. Start with the archive and its new ClickHouse backups cron.
- ops workflows →
Models from git, applied by the pipeline. Apply
ops/models/frommainin the release pipeline, with a drift check reporting anything on ops that doesn't match git. Adopt or delete the 43 unrecorded models. If an extension manifest can carry model instances, use that instead.Grants from git. Deliver every
ops/grants/file, not just three, and drop the ones that exist only on the server.
Done when
- Nothing on ops is created or changed outside the release pipeline.
- Every workflow and model on ops has a home in git.
- The drift check comes back clean.
Open
No activity in this phase yet.
Sign in to post a ripple.