#3174 swamp-extensions CI: Review Integrity audit stalls silently on a large trust-root change until Forgejo kills the job

Parked (2026-10-12)

Investigation parked and the issue closed. No change landed: the work-in-progress was discarded, and diagnostic PR #472 is closed. Reopen this issue to resume.

What was found

Three diagnostic runs on throwaway PR #472 (Forgejo runs 2001, 2002, 2003). Each carried #468's four trust-root files, byte-identical, plus an instrumented audit step: a progress line per stream event, a 480 s timeout, and in the later runs a 30 s heartbeat and the job's cgroup limits.

• Not a rate limit. In every run, the API accepted the request and began streaming a response (init, then thinking-token events). A rate limit would have been refused with a 429 before any response streamed. No Console check is needed.
• The whole job goes dark, not just claude. 6 to 11 s into the response, all output stops: claude's progress, a separate heartbeat loop that is just a 30 s sleep and an echo (it never printed once), and the 480 s timeout (its error never reached the log). The step's completion never reached Forgejo, which reaped the job 10 to 13 minutes later.
• Ruled out, each with evidence:
  • Container limits: 8 GB memory with 1.2 GB in use, 4 CPUs, no memory pressure or OOM events.
  • claude's memory: a flat 236 MB for both small and large prompts.
  • Child processes holding the pipe: claude spawns none.
  • Shared runner home-directory state, retry behaviour and the autoupdater: run 2003 used its own claude config directory, capped retries and the autoupdater off, and froze the same way.
  • One bad machine: every attempt ran on a different ephemeral runner (forgejo-runner v13.0.0).
• Timing pattern: on 2026-10-08, audit jobs that finished within about 57 s of runner pickup passed (run 1929, attempt 2). Jobs still running at 48 to 90 s died, sometimes right as claude started (run 1992, attempt 1). On 2026-10-02, an 87 s audit passed (run 1929, attempt 1), so the runner environment may have changed in between.

Open questions for whoever picks this up

1. The runner or orchestrator logs for tasks 41896, 41898 and 41900 (about 15:04:40, 15:21:38 and 15:39:58 UTC on 2026-10-08): did the container exit, get killed, or lose its network? Did anything about the ephemeral runners change after 2026-10-02?
2. Since the runner itself goes dark, no in-step change can stop a silent failure. A comment needs a job outside the dying one, for example a follow-up job with needs: review-integrity and if: failure() that posts "did not complete". First confirm that Forgejo schedules dependent jobs after a reaped job.

Design that was built and tested locally (discarded, kept here for reference)

Changes to the "Audit trust-root changes" step in .forgejo/workflows/ci.yml:

• Streamed progress: claude runs with output-format stream-json and verbose. A small inline python filter (python3 -u, so it is unbuffered) prints one line per event, built only from typed fields: init (model, key source, version), thinking-token estimates, api_retry (attempt, status, delay), rate-limit events, token usage, and the result (subtype, is_error, api_error_status). It never prints the agent's text, and it redacts the key value and anything shaped like sk-ant-. The final result event's text becomes the review body.
• Time limit: timeout --kill-after=15s 480 around the claude call. The job shell is bash -eo pipefail, so claude's exit status leaves the pipeline through a file.
• Retry caps: CLAUDE_CODE_MAX_RETRIES=4 and API_TIMEOUT_MS=300000, both present in the pinned 2.1.282.
• Isolated config: CLAUDE_CONFIG_DIR pointing at a fresh directory under the work dir, plus DISABLE_AUTOUPDATER=1. The host executor runs as root with a shared /root/.claude, which the auditor should not load.
• On timeout: the body gets fixed text saying the audit did not finish within the limit, and the log gets an error naming the limit. claude's stderr is printed redacted, with every line prefixed.

Local results with the pinned binary and #468's real prompt:

• A normal run passed in 13 s, with progress lines arriving live.
• A 5 s timeout produced the fixed comment.
• A bad key surfaced the 401 immediately.
• A local fake API answering 429 logged four retry lines, then failed with the API's message.
• A hung API and a mid-stream stall both ended in a clean timeout.
• No output contained the key.

The in-step changes are still worth having (a real 429 or a hung API would now explain itself), but they do not fix this issue.

Reopening: hit again on 2026-10-15, now on a small trust-root PR.

PR #494 (swamp-club #2583) changes 5 files and about 60 lines, all under verification/, scripts/ and agent-constraints/. CI run 2067 for commit 7df1f347b: Validate Attestation passed in 35s. Review Integrity showed as running from about 15:01 UTC, with its reported duration stuck at 53s throughout. Forgejo marked it failed at about 15:16 UTC, and no comment was posted on the PR.

This makes PRs 468, 472 (diagnostic), 485 and 494 hit by the same hang. The 494 case shows it is not limited to large trust-root diffs. PR 494 is waiting on this, or on a maintainer override, to merge.

Possibly related: #3390, where the aws/ec2 publish job is killed after about 14 minutes on the same runners.