Skip to main content
← Back to list
01Issue
FeatureOpenPlatformPublic
AssigneesNone

Relationships

#4476 Image factory: golden GCE and Docker base images built on ops

Opened by swamp_lord · 11/6/2026

Problem

We build no images of our own.

GCE. Every VM boots a public image family and sets itself up at boot:

  • Ubuntu 24.04: Grafana, build workers, UAT VMs, the ClickHouse servers, keepers and readers.
  • Debian 12: the ops VM and the Tailscale routers.
  • Windows 2022 Core: the release lane's smoke VM.

The setup is duplicated and drifts:

  • Tailscale is installed three ways.
  • node_exporter runs at two versions (ClickHouse runs its own 1.8.2 on :9100).
  • BitBison, op, Docker and gcloud are each installed two to four times.

Hardening is uneven:

  • no Shielded VM or block-project-ssh-keys on build workers or UAT VMs;
  • unattended-upgrades only on Grafana;
  • only the build workers pin an exact image.

Docker.

  • fleets/general and services/ops-discord-bot use FROM swampclub/swamp:<hard-coded tag>.
  • swamp.git's Dockerfile, which produces swampclub/swamp for amd64 and arm64, uses FROM denoland/deno:2.9.8.
  • In the swamp-club repo, the app, telemetry and discord-bot images use FROM denoland/deno:2.9.8, Dockerfile.test-install uses ubuntu:24.04, and deploy/swamp uses systeminit/swamp:latest.
  • Docker Hub pulls are anonymous.
  • Nothing is scanned, given an SBOM or signed.

Goal

An image factory, run from ops.swamp-club.com, that produces one golden GCE image and one golden Docker base. These are the lowest layer everything is built on. Apart from GCP's default GKE node images, they are the only images allowed, and that's enforced.

Decisions

GCE golden image

  • One image with the whole stack installed, on Ubuntu 24.04: Tailscale (signed apt repo), node_exporter + Ops Agent, BitBison sensor, op, gcloud, Docker CE/buildx, Deno, AWS CLI, Playwright deps, mdadm, ClickHouse and Grafana prerequisites, unattended-upgrades, and the hardening baseline.
  • Each node turns on only its role at boot, from instance metadata (swamp-role); the rest stays installed but inactive.
  • Left to boot: Tailscale enrolment, secrets, data disks and the swamp binary.

Windows golden image

  • Windows Server 2022 Core, for the release lane's smoke VM, under the same enforcement.

Docker golden base

  • The lowest layer, used everywhere: Ubuntu 24.04 + pinned Deno + tini, certs, curl, git and the swamp user. Built for amd64 and arm64.
  • swamp.git builds swampclub/swamp FROM golden.
  • The swamp-club repo's images (including deploy/swamp and Dockerfile.test-install) build FROM golden.
  • platform-swamp images stay FROM swampclub/swamp, pinned by digest, so golden is underneath them too.
  • Dev builds use golden as well, so its repository is public.

Project: swamp-club-archive, the org's reference-data project.

  • It needs billing, a budget, and a scoped exception to iam.allowedPolicyMemberDomains for the public Docker repo (like swamp-club-alerts has).
  • The existing do-spaces-archive buckets aren't touched.

Cadence

  • Weekly: a security rebuild of both images, published as candidates. Nothing rolls out automatically; promotion and rollout are a manual monthly task.
  • On merge to the image definitions: build a candidate.
  • Every swamp stable release: the 12-hourly runtime upgrade already rebuilds the platform images FROM the new swampclub/swamp, which sits on top of the current golden.

Scanning: report only, posted to Discord. A scan never blocks publishing.

Enforcement

  • GCE: the org policy compute.trustedImageProjects allows only swamp-club-archive and GKE's node-image projects. It's switched on last, once every VM has moved.
  • Docker: a drift check on FROM lines.

Plan

  1. Foundations. All done through ops workflows (@swamp/gcp-organization).
    • In swamp-club-archive: billing and a budget, a public Docker Artifact Registry repo golden, and the GCE image families swamp-golden and swamp-golden-windows.
    • Give consumer projects' service accounts compute.imageUser.
  2. Factory package @swamp/image-factory. It extends @swamp/gcp/compute (images, imagefamilyviews) and @swamp/container-image.
    • GCE builds:
      1. Boot a builder VM from ubuntu-2404-lts-amd64.
      2. Run one consolidated provisioning script, folding in bitbison_host.ts, node_metrics_host.ts, tailscale_host.ts and the Docker/gcloud/op installers.
      3. Smoke-test it, create the image in the family, and deprecate older images, keeping the last N.
    • Docker builds: multi-arch buildx on the builder=docker workers, pushed by digest.
    • Scanning: Artifact Analysis plus a package CVE scan, reported to Discord.
    • Workflows:
      • build-candidate: weekly, and on merge.
      • promote: manual and monthly; moves the stable tag and the family to a candidate.
      • drift: reports anything not on golden.
  3. Role-based boot. A swamp-role metadata contract, plus a boot unit that enables what each role needs: ops, grafana, build-worker, uat, clickhouse-server, clickhouse-keeper, clickhouse-reader, tailscale-router, windows-smoke.
  4. Docker consumers.
    • swamp.git Dockerfile FROM golden (multi-arch).
    • swamp-club repo Dockerfiles FROM golden.
    • platform-swamp images FROM swampclub/swamp@digest; the runtime upgrade pins that digest.
  5. GCE consumers.
    • First the build workers and UAT VMs (which also gain Shielded VM and block-project-ssh-keys), Grafana, and the Tailscale routers.
    • Then ClickHouse, by rolling node replacement with the existing replace-node.
    • Then the ops VM, with a boot-disk rebuild in a planned window.
    • The Windows smoke VM moves to the Windows golden image.
  6. Enforce.
    • Turn on compute.trustedImageProjects once the drift check is clean.
    • Add a Docker FROM drift check to CI in platform-swamp, swamp.git and swamp-club.
    • Write a skill documenting the monthly promotion and rollout.

Done when

  • Every GCE VM boots swamp-golden or swamp-golden-windows.
  • Every Dockerfile in platform-swamp, swamp.git and swamp-club builds FROM golden, directly or through swampclub/swamp.
  • trustedImageProjects is enforced.
  • Candidates build weekly with a scan report, and promotion is a documented monthly step.

Open questions

  • arm64 builds: QEMU on the amd64 workers, or arm64 workers?
  • Mirror golden to Docker Hub as well?
  • How many old images to keep?
02Bog Flow
◉OPEN○TRIAGED○IN PROGRESS○SHIPPED

Open

11/6/2026, 2:49:15 PM

No activity in this phase yet.

03Sludge Pulse
Editable. Press Enter to edit.

swamp_lord commented 11/6/2026, 3:03:14 PM

Decisions since filing:

  • No Docker Hub mirror. The golden images live only in swamp-club-archive (this answers that open question).
  • Every builder and worker fleet runs in swamp-club-ops. swamp-club-archive is purely an artifact/data archive, with no VMs. GCE images are built from a short-lived builder VM in ops, and the image is created in the archive project from that VM's disk (GCE allows this across projects). The archive only needs the Compute API enabled to store images.
  • arm64 Docker builds start with QEMU on the existing builder=docker workers. An arm64 fleet, in ops, only if that's too slow.

Phase 1 progress:

  • Every archive bucket (11) already has public-access prevention enforced and none is public, so enforcing storage.publicAccessPrevention on the project comes before the domain-restriction exception.
  • The budget will be $500/mo.
  • Blocked on #316, a fix to @swamp/gcp-organization/enable-apis (it failed on projects ops hadn't read before), and on enabling the Org Policy API in swamp-club-ops.

swamp_lord commented 11/6/2026, 3:16:45 PM

Access decided: the Docker golden base is public (allUsers reader on [HOST-1]/swamp-club-archive/golden). It contains only dependencies (Ubuntu, Deno, common tools), so swamp.git, its contributors and CI can use FROM golden without credentials. The GCE golden images stay internal: compute.imageUser for our own projects only. Phase 1 so far: archive APIs enabled, $500/mo budget, storage.publicAccessPrevention enforced on the project (all 11 buckets were already non-public), and the member-domain exception applied for the Docker repo. Repository model in #317; a fix for the org-policy workflow's read-back of project policies in #318.

swamp_lord commented 11/6/2026, 4:24:36 PM

Phase 1 (foundations) done.

  • swamp-club-archive:
    • Compute, Artifact Registry, Container Analysis and Container Scanning enabled;
    • $500/mo budget (50/90/100% to the ops signals topic);
    • storage.publicAccessPrevention enforced (buckets stay private);
    • member domains allowed, for the public Docker repo.
  • Docker repo: [HOST-1]/swamp-club-archive/golden, mutable tags; allUsers reader; swamp-build-worker@swamp-club-ops writer. Anonymous pull checked: token issued, tags/list 200.
  • Fixes on the way:
    • #316: enable-apis on projects ops hadn't read;
    • #318: project org policies read back by project number;
    • #319: allUsers allowed only as reader on Artifact Registry repos.
  • Org service account now has Organization Policy Administrator (org level), so policy changes, including phase 6's trustedImageProjects, run through ops.

Next: phase 2, starting with the golden Docker base.

swamp_lord commented 11/6/2026, 5:37:28 PM

Phase 2: the golden Docker base is built on ops.

  • #320: images/golden/Dockerfile, a drop-in for denoland/deno:2.9.8. Ubuntu 24.04 pinned by digest; Deno 2.9.8 and tini pinned by sha256 per architecture; git, curl and security updates; uid 1000 left free. Also the @swamp/platform-ops/golden-image model (build_candidate, promote) and two workflows: golden-image-candidate (weekly, Mondays 06:00 UTC) and golden-image-promote (manual).
  • #321: the golden-base model, and a GitHub Action that builds a candidate when images/golden/** changes.
  • #322: fix: the data name latest is reserved.
  • First recorded candidate: [HOST-1]/swamp-club-archive/golden/base:candidate-20261106-6b12f1209e88 (sha256:477d1f4adb81ba3d8b2cb8cdd3b398d20e542ec6196624e6c7583c46336cb8c5), linux/amd64 and linux/arm64, built on a builder=docker worker in about 3.5 minutes (arm64 via QEMU). Pulled anonymously and smoke-tested on both platforms.

Next: promote to stable, then the Docker consumers (phase 4) and the GCE golden image.

swamp_lord commented 11/6/2026, 5:56:43 PM

Golden Docker base promoted. base:stable = sha256:477d1f4adb81ba3d8b2cb8cdd3b398d20e542ec6196624e6c7583c46336cb8c5, via @swamp/platform-ops/golden-image-promote on ops; the tag was verified anonymously.

Phase 4 (Docker consumers) started:

  • swamp-club/swamp#3651: Dockerfile FROM golden by digest, with ARG GOLDEN_DENO checked at build and kept equal to .tool-versions by the toolchain pin test. Built locally as release.yml does: swamp runs under tini as uid 1000, and a mismatched GOLDEN_DENO fails the build.
  • swamp-club/swamp-club#1494: app, discord-bot, telemetry and Dockerfile.test-install FROM golden. All built locally for amd64; install.sh installs swamp on golden.
  • Waiting on #3651's release:
    • deploy/swamp (systeminit/swamp:latest) moves to a digest-pinned swampclub/swamp;
    • platform-swamp fleets/general and services/ops-discord-bot pin the golden-based swampclub/swamp by digest. This also fixes their docs, which claim FROM follows SWAMP_VERSION; it's hard-coded today.

swamp_lord commented 11/6/2026, 6:34:34 PM

GCE golden image plus VM patching: plan agreed (2026-11-06).

  • Image: one Ubuntu 24.04 golden image with the whole stack installed but disabled. A swamp-role.service enables each node's units from swamp-role metadata at boot. Built in swamp-club-ops and stored in swamp-club-archive (family swamp-golden-candidate → promote → swamp-golden). UAT runners and build workers boot straight from it with every dependency baked in, so their startup does per-run setup only.
  • Patching: a reusable reimage path. It snapshots every disk first, swaps the boot disk in place (same instance, IPs, service account and data disks), gates on role health, and rolls back automatically if the gate fails. Each VM group has its own @swamp/platform-ops/patch-* workflow with its own logic: uat, build-workers, tailscale-routers, grafana, clickhouse (replace-node, strict health and caught-up replication after each node), forgejo, labs-blog, engineering-sandbox, windows, ops-serve. A patch-all orchestrator is the monthly routine, documented in a new vm-patching skill.
  • Testing: unit tests, plus a golden-reimage-drill (reimage and forced rollback on throwaway VMs) that must pass before every promotion.
  • Scope: all 29 long-lived VMs, including forgejo, labs-blog-ghost, Paul's engineering-sandbox VMs (created 2026-11-04) and paul-testing-windows (Windows golden image). The first window is sequential with checks at each stage; ClickHouse is the test subject, and ops.swamp-club.com's VM goes last. compute.trustedImageProjects is enforced per project afterwards.
  • PR order: the image; the reimage building blocks and drill; the per-group workflows; the skill and patch-all; then the window.

swamp_lord commented 11/7/2026, 12:09:26 AM

First golden GCE candidate built and smoke-tested on ops.

  • Image: swamp-golden-20261106-234534 (swamp-club-archive, family swamp-golden-candidate, READY), from ubuntu-2404-noble-amd64-v20261105. 10 GB disk, 4.9 GB used. UEFI/Secure Boot/SEV-capable.
  • Smoke test passed for every role: generic, build-worker, uat-worker, tailscale-router, ops.
  • SBOM: stored as golden-gce data named after the image. It's the full CycloneDX 1.7 document from Syft: 4,477 components (about 3,500 Go modules from inside binaries, 683 debs), 643 KB encoded on serial port 4.
  • BitBison: the job tools are baked in. The sensor installer has no install-only mode, so the host sensor stays a boot-time install.
  • Fixes to get here (#325–#329):
    • accounts holding swamp's ids moved aside (sgx 994→988);
    • snaps removed;
    • kernel modules excluded from the SBOM (6,858 components);
    • a short-line serial transport;
    • set -E traps in command substitutions.
  • Querying inside the SBOM (attributes.components.exists(c, …)) is blocked by swamp-club#4490: data query rejects CEL list macros. Filed with the cause and fix.

Sign in to post a ripple.