Skip to main content
← Back to list
01Issue
BugClosedSwamp CLIPublic
Assigneesskunk-ape

Relationships

#2270 Merge gate validates attestations from stale branches, so main can go red with no PR ever failing

Opened by skunk-ape · 9/18/2026

Root cause of #2268, found while triaging it. Filing separately because #2268 is the one-line fix for the symptom and this is the systemic half.

What happened

  • #2504 merged at 21:14 UTC as d4d3bf38, adding the architectural fitness test 'serve functions that push must run under the sync gate' to integration/serve_deps_rules_test.ts.
  • #2509 merged at 23:36 UTC as a3e60933, adding handleWorkerPrune with an ungated dispatch.
  • Comparing d4d3bf38 against #2509's head 7587ee59 reports behind_by: 1 — the branch never contained the commit that added the rule.

So #2509's verification legitimately reported '12,191 passed, 0 failed': the test that its code violates did not exist on the branch it ran against. Both PRs were green. main went red the moment the second one merged. A textbook semantic merge conflict, and the merge gate cannot see it because the attestation attests to the branch, not to the merge result.

Why it matters

Architectural fitness tests are precisely the checks a stale branch can violate without noticing — they assert properties over the whole tree, so a rule added on one branch is enforced against code written on another. The current gate makes that class of breakage undetectable until main is already red, and once main is red every branch cut from it fails verify-build, so no attestation can be produced for anything in flight. #2268 blocked at least one other PR this way.

Options

  1. Require branches to be up to date with main before merge (GitHub's 'Require branches to be up to date' setting), which forces re-verification after a rebase.
  2. Have the merge gate re-run verification against the merge result rather than validating a branch-produced attestation.
  3. Record the merge-base in the attestation and have validate-attestation reject one whose base is behind main.

Option 3 is the cheapest signal and keeps verification local; option 1 is the strongest guarantee but costs a rebase-and-reverify round trip on every PR.

  • #2268 — the red main this produced
02Bog Flow
✓OPEN✓TRIAGED○IN PROGRESS◉CLOSED+ 1 MOREASSIGNED+ 10 MOREREVIEW

Closed

9/18/2026, 5:41:35 PM

No activity in this phase yet.

03Sludge Pulse
skunk-ape assigned skunk-ape9/18/2026, 4:27:37 PM
Editable. Press Enter to edit.

skunk-ape commented 9/18/2026, 5:41:34 PM

Closing as an accepted cost rather than a defect.

What was confirmed

The mechanism is real and was verified, not just described. GitHub compare d4d3bf38...7587ee59 returns ahead 6, behind 1, diverged — PR #2509 head genuinely did not contain #2504 commit d4d3bf38, which added the "serve functions that push must run under the sync gate" rule to integration/serve_deps_rules_test.ts. #2504 merged 21:14:25Z, #2509 merged 23:36:45Z. #2509 attestation truthfully reported a green suite for a tree in which the violated rule did not exist.

The gate has no compensating control. validate-attestation checks five things — schema version, attestation commit equals PR head SHA, gate allPassed, a 24h freshness window that only warns, and the configIntegrity hashes. Nothing references the merge base. Branch protection on main declares no required status checks at all, so GitHub strict "require branches up to date" is not merely off, it cannot be enabled without first declaring contexts.

Why we are not fixing it

We are deliberately slimming the CI runner down to validating the attestation and nothing else. Trust flows from locally produced, agent reviewed attestations; CI is a cheap verifier of claims, not a second source of truth that duplicates the work. Re-running checks on the runner — for the merge result, or post-merge on main — is the wrong direction regardless of how cheap an individual instance looks.

Under that principle this hole is unclosable by construction, and that is worth stating plainly so it does not get re-litigated. A semantic merge conflict lives in the merged tree. The merged tree exists only at merge time. Only CI is present at merge time. So if CI validates attestations and does not run checks, the merge gate can only ever know about trees that existed before the merge, and this class of breakage cannot be caught. Every candidate fix resolves the tension by weakening the principle somewhere.

The second reason is throughput. Preventing this requires either re-verifying the merge result or requiring branches to be current before merge. The latter serialises attestation production under parallel work: measured over the last 100 merged PRs, at least 18% were behind the base at merge — a lower bound, since verification runs before PR creation. Local runs put verify-build at 72-92s and verify-reviews at 70-98s, so a rebase-and-reverify round trip is roughly 5 minutes against a 4-minute median PR lifetime, with genuine livelock risk during busy periods. We would rather take the parallelism and accept that main occasionally goes red.

Options considered and rejected

  1. Require branches up to date before merge (GitHub strict). Rejected: serialises attestation production, and cannot be enabled without first declaring required status checks.
  2. Re-run verification against the merge result in CI. Rejected: contradicts the CI-slimming principle. A narrowed form — running only the deterministic checks (lint, fmt, type-check, test) against the merge result, with the agent reviews staying local and branch-scoped — was planned in detail and still rejected on the same grounds, plus 3-6 minutes of added CI latency per PR against a 4-minute median lifetime.
  3. Record the merge base in the attestation and reject one whose base is behind. Rejected: it adds no enforcement power CI does not already have, since validate-attestation already pins the head SHA and a commit fully determines a tree. It buys audit provenance, not a gate. As a hard failure its operating cost equals option 1.
  4. Post-merge detection on main. Rejected for the same reason as 2 — still CI doing compute, only in a different workflow file.

Accepted risks, recorded deliberately

UAT does not cover this class. AGENTS.md defines UAT as tests that spawn the CLI and assert user-facing behaviour against the compiled binary; architectural fitness tests are static rules over the source tree and are invisible to it. The specific defect here was an ungated dispatch — a race — which UAT is least reliable at catching and would at best surface as a flaky adversarial test.

A red main amplifies. Every branch cut from a red main fails verify-build, so no attestation can be produced for anything in flight until it is fixed. #2268 blocked at least one other PR this way. The cost of an occurrence is therefore a brief pipeline stall, not an isolated red check.

Occurrence requires two in-flight PRs where one adds a whole-tree assertion the other violates. Observed once. If it starts recurring, the thing to revisit is the CI-slimming principle itself, not the merge gate — the gate cannot be fixed while the principle holds.

  • #2268 — the red main this produced, fixed separately in a765b344.

Sign in to post a ripple.