Skip to main content

UNDERSTANDING HONEYCOMB AUTOMATION IN SWAMP

Honeycomb is configured through a mix of its UI and its v1 Configuration API. Most teams start in the UI: they define an SLO, add a burn alert and wire a trigger to a Slack channel. Later they find that nobody remembers what the alert thresholds are, or why. The @hivemq/honeycomb extension moves that configuration into Swamp behaviors, so each change is a method run with recorded inputs and results. This page covers how the extension is structured and the decisions behind that structure.

Two behavior types for two scopes

The extension provides two behavior types. @hivemq/honeycomb covers the objects that belong to a dataset and its reliability goals: SLOs, SLI derived columns, burn alerts, queries, query annotations and triggers. @hivemq/honeycomb/datasets covers the datasets themselves: finding the ones that are no longer written to, or that were created by probe traffic, and deleting them.

The split follows scope. An SLO or a trigger belongs to one dataset. A cleanup pass looks at every dataset in the team. Putting both in one type would mean either requiring a dataset slug that the cleanup methods ignore, or making the slug optional and letting SLO methods run without one. Separate types keep each set of global arguments matched to the work its methods do.

One behavior per dataset

datasetSlug is a required global argument on @hivemq/honeycomb, so each behavior is bound to one dataset. A special value, all with two underscores on each side, covers SLOs that span several datasets. Several methods (createDerivedColumn, createQuery, createTrigger and the query annotation methods) also take their own datasetSlug input. updateSliExpression takes sliDatasetSlug. These inputs override the global value for a single run.

That gives teams a choice. One behavior per dataset keeps every run clearly scoped, and the behavior name (checkout-slo, payments-slo) tells you where a change will land. One behavior per service, using per-method overrides for the occasional neighbouring dataset, means fewer behaviors to maintain, but you have to check the inputs to know where a change went. When services and datasets map one to one, the two approaches are the same. When they diverge, binding to the dataset is the more predictable default, because Honeycomb itself scopes SLIs and triggers to datasets.

Credentials and regions

Every call sends the Configuration Key as the X-Honeycomb-Team header. The key can change SLOs and triggers, and the datasets type can delete data, so it belongs in a vault and not in the definition:

globalArguments:
  configKey: "${{ vault.get('honeycomb', 'config-key') }}"
  region: us
  datasetSlug: checkout

The definition stores only the expression. The value is resolved at run time and redacted from output, as shown in Storing Secrets. The region argument selects the us or eu instance. Honeycomb runs these as separate services, and a key from one does not work against the other. The vault guides show how to back the honeycomb vault with a cloud secrets manager when several people or machines run these behaviors.

A Honeycomb key's permissions are set on the Honeycomb side. Some teams keep a narrower key for SLO and trigger work and a separate key for dataset cleanup, each in its own vault entry. This takes more setup. The benefit is that a mistake in a trigger workflow cannot delete datasets.

SLIs and SLOs are separate objects

In Honeycomb, an SLO does not contain its SLI expression. The SLI is a derived column, a calculated field in the dataset, and the SLO refers to it by alias. The extension keeps the same split. createDerivedColumn and updateSliExpression work on the column, and createSlo and updateSlo work on the SLO.

This matters when you change what counts as success. Tightening the expression from "status below 500" to "status below 500 and latency under one second" is an updateSliExpression run. The SLO's ID, target and burn alerts stay as they are. Moving an SLO to a different column entirely is an updateSlo run with a new sliAlias, and that column must already exist. Keeping the two changes separate makes it clear in the run history whether the definition of success changed or the goal did.

What each run leaves behind

Each method writes a resource: slo, derivedColumn, burnAlert, query, queryAnnotation, trigger, or datasetCleanupRun for the cleanup type. These resources are Swamp data. They are versioned and queryable, and they keep the IDs Honeycomb returned.

Later runs depend on those IDs. deleteBurnAlert and deleteBudgetRateBurnAlert take the SLO ID and the alert's defining parameters, not the alert ID, and get the alert ID from the stored burnAlert resource. Triggers work differently. syncTrigger, updateTrigger and deleteTrigger use local state when it exists, and otherwise look the trigger up in Honeycomb by name. So a repository that has never run createTrigger can still take over triggers that were made by hand in the UI. With burn alerts, the extension does not search Honeycomb for an alert you did not create through it.

Idempotency differs by method

The Configuration API is not uniform, and the extension does not hide that. Before you run a method in a loop or a retried workflow, check how it behaves when run twice:

  • createTrigger checks for an existing trigger with the same dataset and name, so running it again does not create a duplicate.
  • syncBudgetRateBurnAlert reconciles a single alert. It creates one when none exists and updates it in place when one does, keeping its ID and routing.
  • createQuery and createQueryAnnotation are not idempotent. Honeycomb queries cannot be changed or deleted, and each call returns a new ID.
  • Delete methods for triggers and query annotations treat 404 and 410 as success and mark the local state as deleted.

For workflows that may be retried, the reconcile and guarded methods are the safer choice, and the non-idempotent ones belong behind a condition. Make Workflows Idempotent covers the general techniques.

Composing into workflows

The methods have a natural order. A derived column must exist before an SLO can refer to it, and an SLO must exist before a burn alert can attach to it. A workflow that runs createDerivedColumn, then createSlo, then syncBudgetRateBurnAlert encodes that dependency, and the workflow execution model ensures each step sees the data written by the previous one.

Dataset cleanup fits a different pattern. Preview and apply are separate runs because deleting a dataset cannot be undone. Some teams run the preview on a schedule and look at the candidates when they have time. Others put an approval gate between preview and apply, so the person who reviews the list is the one who allows the deletion. Running apply mode unattended on a schedule is also possible, but only when the filter (olderThanDays, slugPrefix) has been tested against several previews first.