Skip to main content
← Back to list
01Issue
BugOpenSwamp CLITeam
AssigneesNone

Relationships

#1427 workflow assert: a step whose CEL expression fails to evaluate is omitted from --junit output, so the XML reports a clean pass

Opened by magistr · 7/27/2026

An assert step whose CEL expression fails to EVALUATE is omitted entirely from --junit output, so the XML reports a clean pass. A cleanly-false assert is reported correctly; only the unevaluable one disappears.

The run's exit code is still 1, so exit-code-gated CI is unaffected. Artifact-gated CI — which is the reason --junit exists — reads the XML and goes green.

Repro

A workflow with two asserts, one that evaluates to false and one whose expression errors:

jobs:
  - name: verify
    dependsOn: []
    weight: 0
    steps:
      - name: assert-true
        dependsOn: []
        weight: 0
        allowFailure: false
        task:
          type: assert
          expr: 1 == 1
          message: "trivially true"
          severity: high
      - name: assert-missing-data
        dependsOn: []
        weight: 0
        allowFailure: false
        task:
          type: assert
          expr: data.latest("no-such-model", "no-such-spec").attributes.whatever == 0
          message: "references data that was never produced"
          severity: high
swamp workflow run <wf> --junit --out results.xml    # exit 1

Observed

<testsuites name="@magistr/spike-assert" tests="1" failures="0" time="0.0">
  <testsuite name="verify" tests="1" failures="0" time="0.0">
    <testcase name="assert-true" classname="@magistr/spike-assert.verify" time="0.0"/>
  </testsuite>
</testsuites>

tests="1" failures="0". The assert-missing-data testcase is not present in any form — not as a <failure>, not as an <error>, not as a skipped case. Any CI system consuming this artifact sees one test and zero failures.

Console output for the same run does report it, but not as an assertion:

verify │ done ✓ assert-true in 1ms
verify │ failed assert-missing-data in 1ms
system │ Assertions: 1 passed
system │ Invalid expression: No such key: attributes
>    1 | data.latest("no-such-model", "no-such-spec").attributes.whatever == 0
                                                      ^

Note Assertions: 1 passed — the errored assert is not counted as an assertion at all, which looks like where it drops out of the report.

Control: a cleanly-FALSE assert is handled correctly

Replacing only that step's expr with 1 == 2, same workflow, same flags:

<testsuites name="@magistr/spike-assert" tests="2" failures="1" time="0.0">
  <testsuite name="verify" tests="2" failures="1" time="0.0">
    <testcase name="assert-plainly-false" classname="@magistr/spike-assert.verify" time="0.0">
      <failure message="evaluates cleanly to FALSE" type="AssertionFailure">
severity: high
expr: 1 == 2
      </failure>
    </testcase>
    <testcase name="assert-true" classname="@magistr/spike-assert.verify" time="0.0"/>
  </testsuite>
</testsuites>

So the defect is specific to expressions that raise during evaluation, not to failing assertions generally.

Expected

An assert that could not be evaluated should appear in the XML — most naturally as <error> rather than <failure>, since JUnit already distinguishes "the test ran and failed" from "the test could not run". Whatever the encoding, it must not be absent, because absence is indistinguishable from success to every consumer of the artifact.

Why this shape matters

data.latest(...) returning nothing is not an exotic case — it is the normal result of an upstream step that did not produce its data, which is exactly when a verification workflow most needs to be loud. The failure mode is silent and directional: the assert that could not run is the one that disappears, so the report is cleanest precisely when the infrastructure is least healthy.

Underneath it there may be a model gap worth considering separately: severity expresses how much a failure MATTERS, but there is no state for "this could not be EVALUATED". An unevaluable assert therefore has nowhere to go. In our own gate tier we needed a four-state outcome (green / red / incomplete / could-not-run) for this reason — zero findings from a check that never ran is byte-identical to a clean check, and conflating them is how a gate reports green while red.

Also, a doc/CLI mismatch

https://swamp-club.com/manual/how-to/verify-infrastructure-with-assert-steps says severity high "Fails run by default" and presents --fail-on high as the setting that "allows only high-severity failures to fail the workflow". swamp help workflow run says --fail-on is the "Minimum assert severity that fails the run (low, medium, high). Default: low (any failure)." One of the two is wrong.

Environment

swamp 20260727.000815.0-sha.db74fcac, macOS (APFS, case-insensitive). Local repo, no --server.

02Bog Flow
◉OPEN○TRIAGED○IN PROGRESS○SHIPPED

Open

7/27/2026, 8:14:42 AM

No activity in this phase yet.

03Sludge Pulse

Sign in to post a ripple.