Skip to main content
← Back to list
01Issue
FeatureShippedSwamp Club
Assigneesstack72

Relationships

#912 Document the run tracker across all four diataxis verticals

Opened by stack72 · 7/1/2026· Shipped 7/2/2026

Problem

PR swamp-club/swamp#1725 shipped a SQLite-backed run tracker that tracks in-flight model method runs and workflow runs with heartbeat-based stale detection. It adds two new CLI command groups (swamp run history, swamp run doctor) and changes how swamp model cancel works internally. None of this is documented in the manual.

The run tracker is a cross-cutting feature that touches all four diataxis verticals:

  • Reference: the new CLI commands need flag tables, JSON output shapes, and exit codes
  • Explanation: why a run tracker exists, how heartbeat-based stale detection works, why runs are tracked in SQLite instead of the datastore
  • How-to: diagnose and fix stale runs, check what's running, query run history on a remote server
  • Tutorial: not needed standalone, but existing tutorials that show model method run or workflow run should mention swamp run history as the way to check status

Background: What the Run Tracker Does

Every model method execution (CLI, serve, workflow step) and every workflow run registers with a local SQLite database (.swamp/run_tracker.db). Each run heartbeats every 30 seconds. If a heartbeat stops and the originating process is dead, the run is considered stale and eligible for reaping.

New CLI Commands

swamp run history              # recent runs (last 24h)
swamp run history --active     # what's running right now
swamp run history --all        # full history (7-day retention)
swamp run history --server URL # query a remote server
swamp run doctor               # diagnose stale runs
swamp run doctor --fix         # auto-reap stale runs
swamp run doctor --server URL  # diagnose on a remote server

Changed Behavior

  • swamp model cancel now queries the run tracker instead of scanning output YAML files. The user-facing behavior is the same but the implementation is different — this matters for the reference docs.
  • Output YAML for model method runs is now write-once (terminal states only), preserving the findAllGlobalSince() mtime optimization.

CRITICAL: Do Not Fabricate CLI Output

Previous documentation PRs shipped errors because CLI command syntax, flag names, and behaviors were written from memory rather than verified. Every command and output shown must be real.

Mandatory Verification Process

  1. Create a sandbox repo: SWAMP_SANDBOX=$(mktemp -d) && cd "$SWAMP_SANDBOX" && swamp repo init
  2. Run each command and capture real output
  3. Verify all flag names against swamp help run history and swamp help run doctor
  4. Use the real output verbatim (truncate with ... but never edit or guess)
  5. Clean up: rm -rf "$SWAMP_SANDBOX"

For swamp run history --active and stale run scenarios, you'll need to start a long-running model method in the background to capture output showing active runs.

Detailed Plan

1. Reference: Add to content/manual/reference/operational-commands.md

The operational-commands page currently documents swamp audit, swamp summarise, swamp telemetry stats, and swamp source. Add a new top-level section for the swamp run command group.

swamp run history

Add a section documenting:

  • Usage: swamp run history [flags]
  • Description: Show recent model method and workflow runs from the run tracker. Defaults to the last 24 hours.
  • Flags: Build the flag table from swamp help run history. Expected flags include:
    • --active — show only currently running executions
    • --all — show full history (up to 7-day retention)
    • --hours <N> — override the time window
    • --server <url> — query a remote swamp serve instance
    • --token <token> — server authentication token
    • --json — JSON output
    • --repo-dir <dir> — repository directory
  • Output (text): Show real output from swamp run history after running a model method. Include both the "no runs" case and the "runs exist" case.
  • Output (JSON): Show real output from swamp run history --json. Document the JSON shape (fields: runId, type, modelName/workflowName, method, status, startedAt, duration, pid, etc.).
  • Examples: At least 3 examples:
    swamp run history                    # recent runs
swamp run history --active           # what's running now
swamp run history --server ws://...  # query remote server

VERIFY every flag name and description by running swamp help run history before writing the table. Do not guess flag names.

swamp run doctor

Add a section documenting:

  • Usage: swamp run doctor [flags]
  • Description: Diagnose stale or orphaned runs in the run tracker. A run is stale when its heartbeat is older than 90 seconds and the originating process is dead.
  • Flags: Build the flag table from swamp help run doctor. Expected flags include:
    • --fix — automatically reap stale runs instead of just reporting them
    • --server <url> — diagnose on a remote swamp serve instance
    • --token <token> — server authentication token
    • --json — JSON output
    • --repo-dir <dir> — repository directory
  • Output: Show real output for both the "no stale runs" and "stale runs found" cases. Show the output difference between swamp run doctor (report only) and swamp run doctor --fix (reap).
  • Exit codes: Document whether the exit code distinguishes "no stale runs" from "stale runs found" (verify by running the command and checking $?).

Update swamp model cancel documentation

In content/manual/reference/model-definitions.md, the ## Execution Cancellation section (around line 420) documents swamp model cancel. Verify whether the user-facing behavior or flags have changed. If the cancel semantics are identical (same flags, same output, same exit codes), no change is needed — the implementation change (tracker vs YAML scanning) is internal. If any flags were added or output changed, update accordingly.

VERIFY by running swamp help model cancel and comparing against what's currently documented.

2. Explanation: Add to content/manual/explanation/the-audit-system.md

The audit system explanation page (around lines 55-65) has a section about what audit is NOT, explaining that audit records commands but not outcomes. It currently says:

For execution outcomes, use swamp model method history or swamp workflow history.

This section should be updated to also mention swamp run history as the unified view of in-flight and recent runs. The distinction:

  • swamp run history — shows what's running RIGHT NOW and recent runs (heartbeat-tracked, real-time status)
  • swamp model method history / swamp workflow history — shows completed execution records (output artifacts, logs, data)

Add 1-2 sentences explaining swamp run history as the "what is happening right now" complement to audit's "what commands were issued". Also fix the existing broken links on lines 60 and 62 — they currently point to /manual/reference/operational-commands but should point to /manual/reference/model-definitions and /manual/reference/workflows respectively for the history commands.

3. Explanation: New section in content/manual/explanation/how-swamp-works.md

The "How Swamp Works" page is the overview that introduces all core primitives. Add a brief section (3-5 paragraphs) explaining the run tracker's role in the system:

  • What problem it solves: Before the tracker, there was no way to know what was currently running. swamp model method history only shows completed runs. If a method hung or a process crashed, the in-flight run was invisible.
  • How it works at a high level: SQLite database, heartbeat every 30s, stale detection when heartbeat stops and process is dead, automatic reaping on next CLI or serve invocation.
  • Why SQLite instead of the datastore: The run tracker is local to a machine (it tracks PIDs), not shared across machines. The datastore is the shared persistence layer. A run tracked on machine A has no meaning on machine B because PID 12345 on machine A is a different process than PID 12345 on machine B. For remote visibility, swamp run history --server queries the server's tracker over WebSocket.
  • 7-day retention: The tracker auto-prunes runs older than 7 days to keep the database bounded. This is not configurable — the tracker is an operational tool, not an audit log.

Place this section after the existing content, near the end of the page. Keep it short — this is supporting infrastructure, not a core primitive like models or workflows.

4. How-to: New guide content/manual/how-to/operational-tasks/diagnose-stale-runs.md

Note: This guide belongs under the "Operational Tasks" index page proposed in issue #910. If that index page doesn't exist yet when this work is done, create it following the pattern in #910. If it does exist, just add this sub-page.

What to cover, in order:

  1. Check what's running: swamp run history --active — show real output with at least one active run (start a long-running model method in the background to capture this)
  2. Spot a stale run: Explain what "stale" means (heartbeat >90s, process dead). Show what a stale run looks like in swamp run history output (the status field).
  3. Diagnose stale runs: swamp run doctor — show real output when stale runs exist
  4. Fix stale runs: swamp run doctor --fix — show real output of reaping
  5. Verify the fix: swamp run history --active — show that the stale run is gone
  6. On a remote server: swamp run doctor --server ws://... --token ... — note that server-side diagnosis checks the server's tracker

Prerequisites: Swamp installed, a repo initialized

Frontmatter:

---
title: "Diagnose Stale Runs"
description: "<150-160 chars>"
type: how-to
order: 5
---

5. Cross-reference updates

After all content is written, verify these cross-references:

  • explanation/the-audit-system.md lines 60-62: fix the broken links (currently point to /manual/reference/operational-commands, should point to /manual/reference/model-definitions and /manual/reference/workflows), and add swamp run history reference
  • explanation/how-swamp-works.md: new section should link to reference/operational-commands for the swamp run CLI reference
  • how-to/operational-tasks/diagnose-stale-runs.md: should link to reference/operational-commands for flag details and explanation/how-swamp-works for conceptual background
  • reference/operational-commands.md: should link to the how-to guide for the task-oriented walkthrough

6. Existing tutorial check

Read through the existing tutorials in content/manual/tutorials/:

  • hello-world.md — if it shows swamp model method run or swamp workflow run, consider adding a brief note about swamp run history to check status. Only add if it fits naturally; do NOT force it.
  • remote-execution.md — if it shows running methods on workers, swamp run history --server is relevant.
  • publish-an-extension.md — unlikely to be relevant.

Do NOT modify tutorials unless the addition is genuinely helpful and minimal (1-2 lines). Tutorials are learning journeys with a specific flow — don't interrupt them with tangential commands.

Implementation Notes

Meta descriptions

Per CLAUDE.md SEO rules, frontmatter description fields must be 150-160 characters. Verify character count for every new and modified page. Previous PRs shipped descriptions outside this range.

Terminology

Per CLAUDE.md:

  • "Swamp" capitalized in prose, swamp lowercase in command invocations
  • "operative" not "operator" for users
  • "collective" not "organization"
  • "agent" lowercase in running prose

Source material

  • The PR description at swamp-club/swamp#1725 explains the feature
  • The design doc at ~/code/swamp-club/swamp/design/run-tracker.md contains implementation rationale
  • The CLI help at swamp help run is the authoritative source for flags and arguments

Acceptance Criteria

  • swamp run history and swamp run doctor documented in reference/operational-commands.md with flag tables from swamp help, real JSON output shapes, and examples
  • swamp model cancel reference checked and updated if needed
  • Run tracker concept explained in explanation/how-swamp-works.md (3-5 paragraphs)
  • explanation/the-audit-system.md updated with swamp run history reference and broken links fixed (lines 60-62)
  • New how-to guide diagnose-stale-runs.md created under operational-tasks
  • Every CLI command in every page has been run in a sandbox repo with real output captured
  • Every flag name verified against swamp help run history and swamp help run doctor
  • Cross-references between all new/modified pages are valid
  • Meta descriptions 150-160 chars
  • deno task check passes
  • Terminology rules followed
02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 2 MOREREVIEW+ 4 MOREPR_MERGED+ 1 MORENOTIFICATION_SKIPPED

Shipped

7/2/2026, 12:40:14 AM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack727/1/2026, 11:59:17 PM

Sign in to post a ripple.