Relationships
#1481 extension quality --json exits 1 with 'rubric factors missing' even when a full score was computed (unearned factor, not uncomputable)
Opened by magistr · 7/30/2026· Shipped 7/30/2026
Summary
swamp extension quality <manifest> --json exits non-zero (1) and prints a
{"error":"Quality rubric factors missing: <factor>"} to stderr even when it
successfully computed a complete, valid score — the only problem being that
the extension did not earn full marks on one factor. It conflates "score
computed, some points unearned" (a normal B/B+ grade) with "score genuinely
uncomputable", giving both the same non-zero exit and the same "factors
missing" wording.
The word "missing" is overloaded and is the root of the confusion. In the JSON
a factor's status: "missing" means "evaluated, but the point was not
earned" (it even ships a remediation string). But the top-level stderr error
"Quality rubric factors missing: symbols-docs" reuses "missing" to mean "the
command failed", implying the factor was absent or the run broke. It was
neither — the factor was present and scored 0/1.
Reproduction
Run against any extension whose symbols-docs factor is below the 80% JSDoc
threshold (I used anime-cron from a swamp-workspace checkout; sources
unmodified), and two controls:
swamp extension quality anime-cron/manifest.yaml --json > out.json 2> err.txt; echo "exit=$?"
swamp extension quality porkbun/manifest.yaml --json > out.json 2> err.txt; echo "exit=$?"
swamp extension quality seanime/manifest.yaml --json > out.json 2> err.txt; echo "exit=$?"Observed (swamp 20260728.192016.0-sha.f2a1f6b9):
| extension | exit | stdout | stderr |
|---|---|---|---|
| porkbun (100%) | 0 | full score, allPassed:true |
(empty) |
| anime-cron (92%) | 1 | full valid score, percentage:92, 13/14 |
{"error":"Quality rubric factors missing: symbols-docs."} |
| seanime (uncomputable) | 1 | empty | {"error":"Extension has model upgrade chain errors..."} |
anime-cron's stdout is a complete grade:
"status":"failed", "rubricVersion":3, "earnedPoints":13, "maxEarnablePoints":14, "percentage":92, ...
{ "id":"symbols-docs", "label":"Most symbols documented", "earnedPoints":0, "maxPoints":1,
"status":"missing",
"remediation":"Add JSDoc to >=80% of exported symbols in entrypoints (current: 50%)." }Every other factor is "earned". The extension is not broken in any way — it is
a 92% (13/14) grade — yet the process exits 1.
Expected vs actual
Expected: when a score is computable, exit 0 and emit the JSON on stdout;
allPassed:false in the JSON already communicates "not perfect". Reserve a
non-zero exit + a top-level error for the genuinely-uncomputable case (e.g.
seanime's broken upgrade chain, where stdout is empty). And the top-level error
should not say a factor is "missing" when that factor was in fact scored.
Actual: any imperfect-but-scored extension exits 1 with a "factors missing" error, indistinguishable at the exit-code level from a truly ungradeable one. The only reliable discriminator today is that the computable case still writes the score to stdout while the uncomputable case writes nothing to stdout.
Impact
Any tooling that treats a non-zero exit as "no score" silently drops these
extensions. A concrete case: a score-ratchet that reads
swamp extension quality --json and throws on non-zero exit skips every
extension below the symbols-docs (>=80% JSDoc) threshold — so a whole class of
otherwise-fine extensions silently loses regression protection despite having a
perfectly good, computable score. (We can and will work around it consumer-side
by parsing stdout regardless of exit code, but the CLI contract is the surprise.)
Suggested fix
- Exit 0 whenever a score object is computable; carry "not perfect" via
allPassed:false/status:"failed"in the JSON, not via the process exit code. Keep non-zero only for the uncomputable path (no score object emitted). - If
extension pushneeds a hard "must be 100%" gate, make that an explicit flag/threshold on the gate path, not the default behavior ofquality --json. - Reword the top-level error: an unearned factor is
status:"missing"in the per-factor JSON, but the top-level "Quality rubric factors missing: X" reads as "factor absent / run failed". Distinguish "unearned" from "uncomputable".
Environment: swamp 20260728.192016.0-sha.f2a1f6b9, macOS.
Shipped
Click a lifecycle step above to view its details.