Skip to main content
← Back to list
01Issue
BugShippedSwamp CLITeam
Assigneesstack72

Relationships

#1481 extension quality --json exits 1 with 'rubric factors missing' even when a full score was computed (unearned factor, not uncomputable)

Opened by magistr · 7/30/2026· Shipped 7/30/2026

Summary

swamp extension quality <manifest> --json exits non-zero (1) and prints a {"error":"Quality rubric factors missing: <factor>"} to stderr even when it successfully computed a complete, valid score — the only problem being that the extension did not earn full marks on one factor. It conflates "score computed, some points unearned" (a normal B/B+ grade) with "score genuinely uncomputable", giving both the same non-zero exit and the same "factors missing" wording.

The word "missing" is overloaded and is the root of the confusion. In the JSON a factor's status: "missing" means "evaluated, but the point was not earned" (it even ships a remediation string). But the top-level stderr error "Quality rubric factors missing: symbols-docs" reuses "missing" to mean "the command failed", implying the factor was absent or the run broke. It was neither — the factor was present and scored 0/1.

Reproduction

Run against any extension whose symbols-docs factor is below the 80% JSDoc threshold (I used anime-cron from a swamp-workspace checkout; sources unmodified), and two controls:

swamp extension quality anime-cron/manifest.yaml --json > out.json 2> err.txt; echo "exit=$?"
swamp extension quality porkbun/manifest.yaml    --json > out.json 2> err.txt; echo "exit=$?"
swamp extension quality seanime/manifest.yaml    --json > out.json 2> err.txt; echo "exit=$?"

Observed (swamp 20260728.192016.0-sha.f2a1f6b9):

extension exit stdout stderr
porkbun (100%) 0 full score, allPassed:true (empty)
anime-cron (92%) 1 full valid score, percentage:92, 13/14 {"error":"Quality rubric factors missing: symbols-docs."}
seanime (uncomputable) 1 empty {"error":"Extension has model upgrade chain errors..."}

anime-cron's stdout is a complete grade:

"status":"failed", "rubricVersion":3, "earnedPoints":13, "maxEarnablePoints":14, "percentage":92, ...
{ "id":"symbols-docs", "label":"Most symbols documented", "earnedPoints":0, "maxPoints":1,
  "status":"missing",
  "remediation":"Add JSDoc to >=80% of exported symbols in entrypoints (current: 50%)." }

Every other factor is "earned". The extension is not broken in any way — it is a 92% (13/14) grade — yet the process exits 1.

Expected vs actual

Expected: when a score is computable, exit 0 and emit the JSON on stdout; allPassed:false in the JSON already communicates "not perfect". Reserve a non-zero exit + a top-level error for the genuinely-uncomputable case (e.g. seanime's broken upgrade chain, where stdout is empty). And the top-level error should not say a factor is "missing" when that factor was in fact scored.

Actual: any imperfect-but-scored extension exits 1 with a "factors missing" error, indistinguishable at the exit-code level from a truly ungradeable one. The only reliable discriminator today is that the computable case still writes the score to stdout while the uncomputable case writes nothing to stdout.

Impact

Any tooling that treats a non-zero exit as "no score" silently drops these extensions. A concrete case: a score-ratchet that reads swamp extension quality --json and throws on non-zero exit skips every extension below the symbols-docs (>=80% JSDoc) threshold — so a whole class of otherwise-fine extensions silently loses regression protection despite having a perfectly good, computable score. (We can and will work around it consumer-side by parsing stdout regardless of exit code, but the CLI contract is the surprise.)

Suggested fix

  1. Exit 0 whenever a score object is computable; carry "not perfect" via allPassed:false / status:"failed" in the JSON, not via the process exit code. Keep non-zero only for the uncomputable path (no score object emitted).
  2. If extension push needs a hard "must be 100%" gate, make that an explicit flag/threshold on the gate path, not the default behavior of quality --json.
  3. Reword the top-level error: an unearned factor is status:"missing" in the per-factor JSON, but the top-level "Quality rubric factors missing: X" reads as "factor absent / run failed". Distinguish "unearned" from "uncomputable".

Environment: swamp 20260728.192016.0-sha.f2a1f6b9, macOS.

02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 2 MOREREVIEW+ 4 MOREPR_MERGED+ 1 MORECONTRIBUTOR_NOTIFIED

Shipped

7/30/2026, 8:53:42 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack727/30/2026, 4:08:59 PM
Editable. Press Enter to edit.

stack72 commented 7/30/2026, 8:53:50 PM

Thanks @magistr for reporting this! The fix has been merged and a release is on its way. We appreciate your contribution to swamp.

Sign in to post a ripple.