#3023 Design: verify the extension push adversarial review through the attestation (whole-extension rubric review as a local verification step, carried in the attestation, read by publish and the push gate)

Status

Design settled 2026-10-06 (session for Lab #3023, from Seth's rewritten brief of 2026-10-05). No code lands from this issue. The product is this design, one measurement, and two implementation issues, filed 2026-10-06 at Seth's request: (a) #3064, swamp-extensions, the verify-reviews step, attestation projection and publish adapter; (b) #3065, swamp, the push gate (and later the registry confirm) accepting attestation-backed evidence and the generated-provenance acceptance. Decisions taken with Seth in session: the design lives in this issue body, no PR; generated extensions are covered by the generator's review, not reviewed one by one.

Problem

Unchanged from the 2026-10-05 rewrite: the repo's adversarial review judges the pull request diff and submits one verdict through @swamp/review-record into the commit-bound attestation; swamp's push gate (extension_review_rules.ts, rule adversarial-review-report) wants a per-extension JSON report keyed by the package content hash with a non-pending verdict on every applicable dimension. Nothing writes that report, so every clean-runner publish carries the warning. A diff review cannot honestly fill whole-extension dimension verdicts.

Facts verified (swamp 20261005.144604.0, swamp origin/main 45184a1e, swamp-extensions main 98f67ba2e)

• The content hash is layout-bound, not content-only. computePackageCacheHash labels each packaged file by its path relative to the swamp repo dir (extension_push.ts sets rootDir to repoDir). The same unchanged extension hashed to 2497cac6 and eb13c1e8 from two scratch repo dirs here, and the swamp-side session measured three distinct hashes across layouts and identical hashes only when a swamp repo is initialised inside the extension directory. publish.yml does exactly that (cd into the extension dir, swamp repo init if no .swamp.yaml, push manifest.yaml), so repo dir = extension dir is the convention every producer of the hash must follow. There is no formatting normalisation; any byte change in any packaged file, and a version bump, moves the hash. Lab #660 only normalised path separators.
• The dry-run output is a stream of several JSON documents (#3017), the skeleton is a JSON string on the adversarial-review-report finding, and the content hash appears only in that finding's file path. A consumer parses it from there or recomputes it. When a report already exists at the path, neither the skeleton nor the hash is emitted.
• The gate is extension-level: content kinds present select the dimensions (8 universal; 7 model; 4 vault; 5 datastore), so a models-only extension has 15 verdicts whether it has 1 model (ssh) or 125 (gcp-compute). The report schema is not strict: extra top-level keys such as provenance parse on the installed binary and are ignored.
• The registry confirm payload carries nothing about the review today. Assessment finding S9 is structural: the review gate is author-asserted because the server never sees the review.
• Main uses squash merges, so a main commit has no attestation of its own: swamp-club returns Not found for the current main head while the pull request head has one. The Forgejo API maps a main commit to its pull request (GET repos/{owner}/{repo}/commits/{sha}/pull) without credentials, and swamp-club serves GET api/v1/admin/attestations?commit= without credentials (POST needs an admin session).
• The verify-reviews run record persists each step's stdout, stderr, exit code and command (checked on run ec169691), so a projection into the attestation can read a step's validated output after the worktree is cleaned up.

Measurement

One whole-extension rubric review each, same claude invocation shape as verify-reviews (claude -p, claude-opus-5-5, allowed tools Read, Glob, Grep and one Edit on the report file), the dry-run skeleton as input, the swamp skill's adversarial-review reference as the rubric. Nothing submitted or posted.

Two data points, both models-only extensions with 15 applicable dimensions; a vault, a datastore and a 125-model gcp service were not timed. A first run of each (114 s and 113 s, 0.78 and 0.71 USD) is not in the table because its report write was denied: an Edit rule with a single leading slash is project-relative in Claude Code, so the step needs the absolute form, which the existing steps' Edit(/$RECORD_FILE) already expands to. Both reviews filled every verdict (ssh: 9 issue, 5 pass, 1 na; digitalocean: 12 issue, 3 pass). The telling number is coverage, not cost: the 54-model generated review took the same time and turns as the 1-model review because it read the shared client and one or two models in full and sampled the rest by grep, by its own account. A whole-extension verdict on a large generated package at this cost is a sample, not a review, which is the second reason (beside multiplying the cost by the extension count on a regeneration) to cover generated packages by reviewing the generator.

Decisions

1. Cost: who gets the whole-extension review

Hand-written extensions (21 manifests outside model/) get the whole-extension rubric review whenever any file under their manifest directory changes on the branch. Generated extensions (682 manifests under model/) do not get a per-extension review. Their review is the existing adversarial-review step on the codegen diff (codegen/ is already in that step's guard), plus verify-build's idempotency check. The measurement supports the split: about two minutes and under one USD per hand-written extension per change, which is the same order as the existing diff review, while the generated review at the same cost sampled 2 of 54 models. On a regeneration that touches 100-plus extensions, per-extension reviews would multiply that by the extension count for verdicts the generator review already covers.

For the push gate, a generated package is evidenced by the generator's review. The gate accepts a report whose provenance says generated, naming the generator and the commit whose attestation carried the generator's adversarial review, and prints an informational line instead of the warning. Until swamp has that acceptance, generated packages keep the warning, waived by --yes and recorded in acceptedWarnings, which is the honest state.

2. Scope of re-review when one file of many changes

The unit is the extension. Any change under the manifest directory re-judges every applicable dimension for that extension; the reviewer receives the merge-base diff and the whole extension tree, so it can spend effort where the change is while still answering every dimension for the package. No per-model granularity: the gate keys one hash over everything packaged, kinds are detected per extension, and the hand-written corpus is small (the largest is kubernetes with 14 workflow files). Per-file hashes exist in swamp and could key a per-model report later; that is a new report shape, a new skeleton and new gate tests, and is deferred until a hand-written extension is large enough to need it.

3. Trust: what the verdicts inherit and what they do not

The report is one more projection of a verify-reviews step, so it inherits exactly the attestation's provenance: produced on the author's machine by scripts/build_attestation.ts from run records, posted by an admin session, validated by CI against the pull request head and the pinned harness files. It adds no claim of its own. What it does not have: proof that the attestation was produced by a step of the run it describes (the known gap in build_attestation.ts), and, until swamp's gate reads provenance, any way for swamp or the registry to tell an attestation-written report from a hand-written one. The short-term publish adapter therefore removes the warning without changing the trust model; the longer-term gate change is what makes the review server-verified (S9). The design keeps the two apart so the first is never mistaken for the second.

Guardrails inside the step: the record carries name, version and content hash per extension, and the step compares those against the dry-run values it computed itself before printing the marker line, failing the step on a mismatch, so a steered reviewer cannot relabel a report. Dimension notes are capped in the record schema and are never read as instructions by anything downstream; the CI validator already escapes attestation text.

4. Where the step runs and what prompt it uses

A new step extension-review in the reviews job of verification/workflow-verify-reviews.yaml, guarded by a marker from detect-changes listing changed hand-written extension directories (a changed path whose nearest manifest.yaml is outside model/). Same claude CLI, same allowlist shape the harness test pins (Read, Glob, Grep, one Edit on the record file, one Bash on the submit command), prompt at verification/review-prompts/extension-review.md, pinned in attestation.yaml. The prompt is the swamp skill's adversarial-review reference (mechanical checks, universal and type-specific dimensions, output format) with the harness's security note; dimension ids are not enumerated in the prompt, they come from the skeleton, so swamp's catalog stays the single source of truth.

Per changed extension the step: copies the extension directory to a scratch location, runs swamp repo init there (so the hash matches publish.yml and the verified worktree's diff is untouched), sets SWAMP_EXTENSION_REVIEW_DIR to an empty directory, runs swamp extension push manifest.yaml --dry-run --json --yes, parses the adversarial-review-report finding for the skeleton and the hash, and writes the skeleton into the reviewer's record file. The reviewer fills the verdicts and submits once. The record: @swamp/review-record gains an optional extensions array (name, version, contentHash, reviewedAt, dimensions with id, verdict pass|issue|na, note capped); submit rejects a pending verdict and an issue verdict with no finding; decide treats an issue verdict with a critical or high finding as fail. After decide passes, the step prints the validated record on one marker line; build_attestation.ts reads that step's stdout from the run record and projects it to an optional extensionReviews map in the attestation keyed by extension name, each entry carrying version, contentHash, the hash convention (repo dir = extension dir), reviewedAt, model, and the dimensions. validate_attestation.ts accepts the optional block. Extending review-record rather than adding a model is the smaller change because the harness test pins every reviews-job step to review-record's submit and decide; a separate model is the cleaner shape if that test is reworked.

Publish: how it obtains the report

Short term, with no swamp change: before each push, publish.yml resolves its commit to the merged pull request head through the Forgejo commit-to-pull API, fetches the attestation for that head from swamp-club, checks gate.allPassed, and for the extension being pushed writes the extensionReviews entry as a report file at SWAMP_EXTENSION_REVIEW_DIR/swamp-extension-review/-.json in swamp's report shape (extension, version, reviewedAt, dimensions) plus a provenance object (kind attestation, commit, attestation id) that today's binary strips. swamp's existing rule then compares hashes. When the merged content differs from the pull request head (another change to the same extension landed in between), the hashes differ and the warning returns; that is expected and honest, not an adapter bug. The push summary still cannot say where the report came from; that is the gate change.

Longer term, in swamp and swamp-club: swamp-club indexes attestations by extension name and content hash and serves the matching report; swamp's push gate looks it up (by name and hash, with the repo-dir convention stated) and the registry confirm payload carries the evidence reference so the registry verifies it server-side; the dry-run and completed summaries render the review source (author, attestation, generated) in log and JSON. This is the issue (b) draft.

Proposed shape of the generated declaration (proposal, not a decision; #3020 and #3021 touch the same field and the three must agree)

# manifest.yaml of a generated extension
generated:
  by: swamp-extensions/codegen/digitalocean   # which generator
  source: https://git.swamp-club.com/swamp-club/swamp-extensions
  commit: <commit of the codegen that produced these files>

Effects: #3020 silences or collapses testing-completeness for the package; #3021 lists it in the summary as a declared acceptance with its reason implied; this design's gate change accepts review evidence of kind generated when the declaration's generator and commit match an attestation that carried an adversarial review over codegen/. The field must be added to the manifest schema and to serializeManifestForHash so a changed declaration moves the hash.

Side findings (not filed; Seth's call)

From the measurement reviews, verified by reading the code afterwards:

• model/digitalocean: database_cluster.ts (and likely database_replica) marks password fields sensitive but not the uri and private_uri connection strings, which DigitalOcean builds with the password inside; a codegen/digitalocean fidelity gap that #3030 (secret-named fields) did not cover.
• model/digitalocean: extensions/models/security_secret.ts exists (last generated 2026-07-02) but is absent from manifest.yaml, so codegen does not remove files for models that disappeared from the spec.

From the ssh review, reported by the reviewer and not verified here: forward list matches other hosts by name prefix (operations.ts:890), open drops several connection settings (control_master.ts:218-246), forward cancel ignores a failed ssh -O cancel, open reports success after a failed connection, and AuthSchema.password is not marked sensitive (schemas.ts:59). The filled reports are in this session's scratchpad.

From the code reading:

• swamp: the report content hash is layout-bound (rootDir = repo dir). Either document the convention or hash relative to the manifest dir, which moves every cache key and report path.
• swamp: push --dry-run --json does not expose the content hash or report path as a field; consumers parse the finding's file name.
• The first-version ripple on this issue assumed a formatting-normalised hash; none exists.

Until then

The warning stays on every CI publish and is recorded as an accepted warning (#3047). Nothing here is urgent.

Related

#596, #660, #3015, #3017, #3018, #3020, #3021, #3047, the push assessment's S9.