Skip to main content
← Back to list
01Issue
FeatureIn ProgressSwamp ClubPublic
Assigneesswamp_lord

Relationships

#1703 No way to store code-review results — verdicts, findings, and cost are not persisted for any repo

Opened by swamp_lord · 8/18/2026

The problem

Every Lab issue is code-reviewed on its way to shipping, and the Lab keeps none of it.

Review is now a real phase of the issue lifecycle: the work behind an issue gets read by a set of agent reviewers, each judging one dimension and returning a verdict plus findings, usually over several rounds before anything ships. The Lab records the issue, its status transitions, its plan and its PR link — and nothing whatsoever about the review that actually decided whether it was fit to ship. There is no schema for a review record, no collection to hold one, and no ingest contract to submit one against.

swamp-club's own CI is one instance of the shape: seven review agents (review-web, review-telemetry, review-discord-bot, review-ui, review-observability, review-performance, review-ci-security), each an Opus invocation reading the diff against its own agent-constraints/*-dimensions.md, path-gated so a routes/** change draws four and a lib/** change three. Their verdicts are submitted as native GitHub reviews and read back by a marker-prefix gate. That is where they stop — and that is one repo. Every other repo with issues in the Lab has the same gap with a different roster.

What is stored today are two plan-level reviews — adversarial_review ({planVersion, findings[], critical, high, medium, low, blockers}) and code_conformance_review. Both judge the plan, before implementation, and neither carries a reviewer identity: the entry's actorUsername is the admin API key's owner, not the model that did the reviewing.

Three things have no representation in the data model at all:

  • Token usage and cost. A repo-wide grep across lib/, routes/, islands/, components/, extensions/ and shared/ for input_tokens, output_tokens, totalTokens or costUsd returns zero hits.
  • Which model reviewed, how long it took, and what it consumed. The model is pinned in workflow YAML; nothing records what actually ran against a given commit, how long it ran, or what it burnt — so a model change is unrecoverable in retrospect.
  • Commit shas and branches. PR linkage exists (githubPrUrl, githubPrNumber, pr_linked with an attempt counter), but there is no way to say which code a review read.

There is also no way to read across reviews even in principle: LabIssueRepository exposes only addLifecycleEntry and listLifecycleEntries(issueId) — no step filter, no date range, and a single {issueId, createdAt} index — so nothing today could aggregate review data across issues even if the data existed.

What spanning repos means for the storage shape

The Lab already tracks issues across repositories: githubRepoFullName sits on the issue aggregate, and POST /api/v1/lab/issues/ensure find-or-creates by githubRepoFullName + githubIssueNumber. Review storage has to work the same way, and the reviewer roster is not the same from one repo to the next — each has its own reviewers, its own dimensions, and its own gating. swamp-club's seven are one repo's answer, not the shape of the data.

Two consequences worth stating up front, because they are the difference between a design that survives the second repository and one that does not:

  • The reviewer set is open. It cannot be a fixed enum, a fixed column list, or anything else that has to be edited when a repo adds a reviewer or a new repo shows up with reviewers nobody has seen.
  • Reviews must land independently, and not landing must be fine. They run in parallel, finish out of order, get skipped by path gating, time out, or fail to submit. Storage should be an upsert of one review at a time, where a record is valid with reviews missing and re-submitting the same review does not duplicate it. A model that needs the complete set before it can store anything will hold nothing, because the complete set frequently never arrives — and is not knowable across repos anyway.

What a submitted review has to record

Each review that lands should carry the full account of itself, not just its outcome. At minimum:

  • Who reviewed — the reviewer's name/dimension, as a free-form identifier rather than a value from a fixed list.
  • What it reviewed — the repo, and the commit/branch/PR the review actually read.
  • The verdict — pass, warn, fail, and the distinct "did not run" state.
  • The findings themselves — severity and full content, not just counts. The content is the part that makes a review auditable after the fact, and it is the part currently lost.
  • Which model performed it, and at what version. Model choice changes review quality and cost, and today it is unrecoverable after the workflow file moves on.
  • How long it took — wall-clock duration of the review.
  • What it consumed — input/output tokens and the resulting cost.
  • When it happened.

What that then makes answerable

  • What a review cycle costs — per reviewer, per commit, per issue — and how that trends over time.
  • How many warnings and passes each reviewer produced, and the finding content behind them.
  • Which reviewers cleared which commit.
  • Which reviewers actually catch things versus always passing — which ones earn their spend, and whether one has quietly gone toothless after a dimensions-file edit.
  • Whether a model change made review better, worse, slower, or more expensive.

Why it matters

Several model invocations run per review round, several rounds per issue, across every issue in the Lab and every repo it spans — and none of it is measured. We cannot tell whether the spend is proportionate, whether a green verdict means "the code is clean" or "this reviewer stopped looking", or whether a change to a dimensions file made a reviewer better or worse. A regression in review quality is currently undetectable, and gets less detectable as more repos join.

It is also an audit-trail gap. "This shipped clean" is a claim the review record substantiates, and that record currently lives only in the reviewing platform's own conversation view — not where the issue lifecycle lives, in a different place per repo, and absent entirely for work that never goes through a PR.

Visibility: admin-only

All of it — the verdicts, the finding content, the model, the durations, the token counts and the cost — is admin-only, and that is a constraint on the storage design rather than something to be bolted on by whatever reads it later. It is operational and commercial data about how the platform is built and what that costs; it is not part of an issue's public record, and an issue's author seeing that their contribution triggered N reviews at some cost is not the intent.

/lab/insights already sets the precedent both ways: it is a public page whose paidLeadTimes block is admin-only because it is commercial data, and its one-corpus rule means a surface must never mix what a viewer may see with what they may not. Review records sit wholly on the admin side of that line, so no public or author-visible surface should derive anything from them — including counts, which leak the shape of what they are counting.

Scope

This asks for the ability to store and retrieve review records — the domain shape, the persistence, and an upsert-shaped way to submit one, across repos, without a fixed reviewer list, readable by admins only. It does not ask for any UI, and it does not ask for the CI-side work to actually emit the records. Both are worth doing and both are follow-on work; neither can start while there is nowhere for the data to live.

Questions the design will need to answer

  • What identifies a review round? Commit sha, PR number, or the lifecycle model's attempt counter. Nothing in the Lab holds a sha today, so whatever is chosen is new — and it has to be unique across repos, not just within one.
  • What is the upsert key — what makes two submissions the same review rather than two? Repo + sha + reviewer name is the obvious candidate, but that decides whether a re-run of a reviewer on the same commit replaces the earlier result or is recorded alongside it, and both are defensible.
  • What counts as a "pass" for a branch or PR — every reviewer green at least once across its commits, or every reviewer green on the final commit? A reviewer that passed an early commit never saw the last one. My instinct is the former, a funnel that ticks each column once, but it is a real decision with a real correctness cost either way, and it determines whether a stored record is per-commit or per-branch. Note it also needs to be answerable without a fixed reviewer list, since "every reviewer" means something different per repo.
  • How is "didn't run" stored? Reviewers are commonly gated — in swamp-club by changed paths and by whether the matching test job passed, so most reviewers skip most PRs, and other repos will gate differently. Is a reviewer that never ran a pass, a hole, or not-applicable? It needs a distinct stored state: today it is indistinguishable from a pass at the merge gate, and it is the common case rather than the exception.
  • Does a review attach to an issue, or must it be storable for a PR with no Lab issue behind it? That decides whether this hangs off the issue aggregate at all.
  • Who may submit one? The existing lifecycle ingest is admin-only, which suits reads; whether the write path stays on the same gate matters once multiple repos are submitting.
  • Retention. Finding content is large and there are several reviewers per commit. Is a review record permanent, or does it age out?
02Bog Flow
✓OPEN✓TRIAGED◉IN PROGRESS○SHIPPED+ 1 MOREASSIGNED+ 5 MOREREVIEW+ 3 MOREPR_LINKED

In Progress

8/18/2026, 2:40:44 AM

Click a lifecycle step above to view its details.

03Sludge Pulse
swamp_lord assigned swamp_lord8/18/2026, 1:48:45 AM

Sign in to post a ripple.