UNDERSTANDING GRAFANA AUTOMATION IN SWAMP

Grafana has two kinds of state: the configuration it serves (dashboards, alert rules, contact points, notification policies) and the telemetry it proxies (Prometheus metrics, Loki logs). The @keeb/grafana extension handles both through one behavior type. The @shrug/grafana-ext extension adds a separate type for the server-admin work of managing service accounts. This page covers how those pieces fit with Swamp's behaviors, types and methods, and the decisions they leave to a team.

One behavior per Grafana instance

@keeb/grafana/instance takes grafanaUrl as its only required global argument. A behavior of this type therefore stands for one Grafana server. Its methods are things you can do to that server. A team with a staging and a production Grafana creates two behaviors, one for each URL, rather than one behavior that is told which server to use on each run.

This keeps the target of every run explicit. When you run pushDashboard on the production behavior, there is no argument that could quietly point it somewhere else. Each behavior also keeps its own data history, so results from staging and production never mix. The cost is a little repetition. If you have many instances, the factory pattern is one way to reduce it.

Dashboards and alert rules as files

pushDashboard and pushAlertRule read JSON from files in the repository. exportDashboard writes a Grafana dashboard back out to a file. Together they let Git hold a record of what Grafana should contain, so changes can be reviewed and reverted like any other code.

This raises a question the extension does not answer for you: which side is the source of truth? Some teams treat the repository as authoritative. They edit JSON, push it, and treat UI changes as drift. Others build in the UI, where Grafana's editor is most convenient, and export to the repository as a snapshot. Both work. Mixing them without agreement does not, because a push overwrites edits made in the UI, and an export overwrites uncommitted edits to the file. getDashboard sits between the two. It stores the full dashboard content, including panels and variables, as data without touching any file, which helps you compare what Grafana holds with what the repository says it should.

What each method writes

Every method writes a named resource. discover writes instance. pushDashboard and exportDashboard write dashboard. getDashboard writes dashboardContent. pushAlertRule writes alert, and createAnnotation writes annotation. queryMetrics writes queryResult and queryLogs writes logResult.

The two query methods matter most here. A PromQL or LogQL result that would normally live only in a Grafana panel becomes versioned data in the repository. Later workflow steps can read it, and you can query it after the fact. The Data Layer covers how those versions are kept, and the Data reference describes how to query them.

queryLogs is described in @keeb/grafana as the Loki sibling of queryMetrics. The readme of @shrug/grafana-ext describes adding a queryLogs to @keeb/grafana/instance alongside queryMetrics. The current @keeb/grafana lists queryLogs as its own method, so log queries no longer depend on the second extension.

Credentials

The two behavior types treat credentials differently. On @keeb/grafana/instance, apiKey is optional, because some Grafana servers on a private network accept anonymous API calls. On @shrug/grafana-admin/instance, adminPassword is required, because creating service accounts needs server-admin rights over basic auth.

In both cases the value belongs in a vault. The definition refers to it with ${{ vault.get('<vault-name>', '<key>') }} and never holds the value itself. Swamp then redacts the secret from output and stored data. The Storing Secrets tutorial walks through this, and the Vaults reference covers the expression and the vault types.

The admin control plane

ensureServiceAccount is idempotent. Given an org ID, a name and a role, it makes sure a service account with that role exists. If one already exists, a repeat run leaves it as it is. It is a dry run unless confirm is true. Without confirm, the serviceAccount resource records the plan, so you can read what would change before anything does.

When mintToken is set, the new token is printed once and never persisted. The serviceAccount resource holds state and plan but never the secret. This is a deliberate gap. Swamp does not keep a long-lived Grafana credential in its data store, so whoever runs the method has to decide where the token goes next, often into a vault for an @keeb/grafana/instance behavior to use as its apiKey. Because the admin password and that decision both carry weight, many teams limit who can run this behavior. On a shared server, the swamp serve access controls are one way to do that.

Alert routing as a sequence

Alerting in @keeb/grafana takes three methods, and their order matters. configureContactPoint creates or updates a Discord webhook contact point. configureNotificationPolicy sets the default notification policy to route to a named contact point. pushAlertRule pushes rules whose notifications follow that policy. If a policy names a contact point that does not exist yet, or a rule fires before its route is in place, notifications go nowhere.

Running the three as steps in a workflow fixes that order and records each run together. The webhook URL for the contact point is a secret like any other and fits the same vault pattern.

Annotations and queries inside workflows

The methods that write to Grafana and the methods that read from it combine well around a change. A deployment workflow can call createAnnotation to mark the moment on a dashboard, or on every dashboard if the annotation is global. Later steps can run queryMetrics or queryLogs to check the service. Because the query results are stored as data, assert steps can check them and stop the workflow when a check fails.

Repeated runs need some care. Pushing the same dashboard or rule again converges to the same state, but each createAnnotation call adds a new annotation. A workflow that is retried can therefore leave duplicate marks on a dashboard. Make Workflows Idempotent covers ways to deal with steps like this.