Relationships
#1590 Scheduled workflow runs (via swamp serve) don't count toward the streak
Opened by kneel · 8/11/2026
Summary
Workflows executed by swamp serve's scheduler (trigger.schedule) don't appear
to count toward the swamp-club activity streak. A persistent serve instance ran a
workflow every 4h for ~2 days, but the streak still reset after 24h without an
interactive CLI command.
Environment
- swamp 20260804.201713.0-sha.c582ab91
- Self-hosted
swamp serveas a persistent systemd user service; --auth-mode none on loopback.
Setup
- A workflow with
trigger.schedule: "0 */4 * * *". - serve registers the schedule on startup (
/health->scheduling.enabled: true) and fires it internally.
Observed
- The scheduler fired every 4h without interruption; all runs succeeded — 11 consecutive runs from 2026-08-09 04:00 to 2026-08-11 00:00 UTC, each starting on the 4-hour boundary.
- Each run wrote versioned data and recorded completed method executions (visible
in
swamp run history). - The streak still lapsed after ~24h with no interactive CLI activity.
Expected
Runs executed by serve/the scheduler under the user's identity should count toward the streak, or there should be an opt-in.
Closed
No activity in this phase yet.
keeb commented 8/11/2026, 2:21:14 AM(edited)
Exact boundaries:
last event before: 2026-08-09 22:41:47 UTC
first event after: 2026-08-11 01:49:10 UTC
gap: 27h 07m
keeb commented 8/11/2026, 3:06:55 AM
Triaged. Confirmed real, and the mechanism is one layer below where this report puts it — filed as #1591 against swamp's CLI.
It isn't that scheduled runs don't count. It's that swamp-club never received a single event for them, so there was nothing to count. Verified against the prod events mirror: your 11 scheduled runs (2026-08-09 04:00 → 2026-08-11 00:00 UTC) produced zero events. Your last event before the gap was 2026-08-09 22:41:47Z and the next was 2026-08-11 01:49:10Z — one missing UTC day, so the streak walk reset 32 → 1. It behaved as designed on the input it had. Not a pipeline outage either: prod ingested 22.8M events from 5,324 devices on 08-10.
The cause is that a persistent swamp serve is telemetry-dark for its whole uptime, for two independent reasons (details and file:line in #1591):
- Serve never wires the workflow telemetry sink — it's bound only in the interactive
swamp workflow runcommand, so serve-side runs record nothing at all. - Recording and flushing happen only in the CLI wrapper, after the command returns. For
servethat's process exit, so a daemon that never restarts never flushes. Your ownserveevents show it: each carries whole-processdurationMs(81s, 166s, 330s, 806s), the last on 2026-08-09 05:06 — before you made it persistent.
Two things that are not the cause, so nobody spends time on them:
- Not your OTEL setup. Those env vars only drive OTLP span export. swamp-club telemetry is a separate path (
HttpTelemetrySender→POST <endpoint>/ingest). Your LGTM stack can't intercept it. It does explain what you're seeing, though: each scheduled fire emits aswamp.scheduled.firespan, so the runs are visible in your Grafana and nowhere else. - Not auth. The API key is optional on flush — a missing token changes attribution, not whether the send happens. Your normal user token is fine. (
--auth-mode noneis serve's inbound HTTP auth, unrelated to outbound telemetry.)
Also worth naming, since it's why this reads as "I executed every day and still lost it": the streak counts UTC days, while the combat log groups its day separators by your local day, client-side. At UTC−3 your 08-09 22:41Z and 08-11 01:49Z land on adjacent local days — so the missing UTC day is genuinely invisible in your own combat log. That display contradiction is swamp-club's and gets its own issue; it will keep producing reports like this one independent of the CLI fix.
Keeping this open as the user-facing report until #1591 ships.
Sign in to post a ripple.