Skip to main content

Fleet Health Loki

@aaronge/fleet-health-lokiv2026.11.05.3· 16d agoBEHAVIORSWORKFLOWS
01README

Evidence-preserving instant LogQL queries against Loki — an explicit time argument rather than an implicit now, so a captured result is reproducible and auditable.

02Release Notes

fleet_health_unit_up counts a running service (e.g. swamp-serve) as up.

03Behaviors1
@aaronge/fleet-health-lokiv2026.10.01.1loki.ts

Global Arguments

ArgumentTypeDescription
baseUrlstringLoki base URL, e.g. http://localhost:3100
timeoutSeconds?number
fn query-at(logql: string, time: string, name: string)
Evaluate a LogQL expression at an explicit timestamp
ArgumentTypeDescription
logqlstringFull LogQL expression, e.g.
timestringRFC3339 or Unix-nanosecond timestamp to evaluate at. No implicit
namestringInstance name this result is stored under (lowercase, digits, hyphens, underscores).

Resources

queryResult(7d)— Parsed result of an instant LogQL query
04Workflows1
@aaronge/fleet-healthe8ddf8f3-c995-4d26-aa34-0ad47f414abd

Cross-cutting health check for the host's swamp scheduling, spanning both Grafana dashboards (fun-stuff and rpi-metrics-bridge). Every scheduled @aaronge/* workflow runs inside one `swamp serve` (the ops repo's swamp-serve.service user unit), so that one unit is what's monitored. Checks three things: (1) each monitored unit's state via @aaronge/systemd-panel, restarting anything found failed (this only helps transient failures; a unit failing from a persistent external cause just re-fails identi

nowCapture a single, reproducible evaluation timestamp for this run.
1.capture
probeGather current state from every unit, Prometheus, and Loki.
1.sync-${{ self.unit }}${{ self.unit }}.sync
2.freshnessprom.query-at
3.new-errorsloki.query-at
assertFail the run when a gated unit is in a failed state, a textfile export has gone stale, or new (unexplained) error-level log lines appeared.
1.unit-ok-${{ self.unit }}
2.freshness-ok
3.no-new-errors
remediateRestart any unit found in a failed state during probe. Runs regardless of the assert job's outcome — it reacts to each unit's own live status, not to the aggregate gate.
1.restart-${{ self.unit }}${{ self.unit }}.restart
exportWrite /var/lib/prometheus/node-exporter/fleet-health.prom for both Grafana dashboards — same metric names, labels and HELP/TYPE lines the old wrapper script wrote. Runs whatever the assert job decided, since a failing check is exactly when the panel matters. Everything is rendered by CEL before the shell starts, so if any probe data is missing the step fails and the existing file is left alone, never replaced by an empty one. Unit series are written in `units` order; labels come from this repo's
1.writeTextfile
05Previous Versions3
2026.11.05.2

Documentation, descriptions and setup brought up to date with running every workflow under a single swamp serve (2026-11-05).

2026.11.05.1

Ship the workflow in the package so a single ops repo can pull it and run it under swamp serve's own scheduler.

Modified 1 behaviors. Added 1 workflows. updated dependencies. updated platforms

2026.10.01.1
06Stats
A
93 / 100
Downloads
32
Archive size
10.2 KB
  • Has README or module doc2/2earned
  • README has a code example1/1earned
  • README is substantive1/1earned
  • Most symbols documented1/1earned
  • No slow types (deprecated)1/1earned
  • Dependencies pass trust audit2/2earned
  • Docs keep secrets in a vault1/1earned
  • Has description1/1earned
  • Platform support declared (or universal)2/2earned
  • License1/2partial

    no SPDX license declared

  • Verified public repository2/2earned
07Platforms