Skip to main content

MONITOR SERVER HEALTH

This guide shows you how to monitor a running swamp serve instance using its health endpoints.

Prerequisites

  • A running swamp serve instance with authentication enabled
  • A valid server token for the health endpoints. Admin access is needed only for the full snapshot and for full run history.

Mint a monitoring token (token mode)

If your server uses --auth-mode token, mint a dedicated token for monitoring:

swamp access token mint health-monitor --principal user:monitoring

Retrieve the token plaintext with the swamp access token reveal command shown in the mint output. The token needs no grant to read the health endpoints.

Query the health snapshot

curl https://swamp.example.com/api/v1/health \
  -H "Authorization: Bearer health-monitor.secret"

The response is a JSON object with instance identity, active runs, throughput metrics, worker status, scheduling, webhooks, and component health.

A token without admin sees a narrowed snapshot: only the runs, schedules and webhooks of models and workflows it may read, without run principals, workers or components. To see everything, grant the monitoring principal admin:

swamp access grant create --subject user:monitoring --allow admin --on "access:*"

See REST API — GET /api/v1/health for the full response schema.

Stream health updates

To receive health snapshots as a continuous Server-Sent Events stream:

curl -N https://swamp.example.com/api/v1/health/stream \
  -H "Authorization: Bearer health-monitor.secret"

The stream sends an initial snapshot immediately, then pushes updates every 5 seconds by default.

To change the push interval (in milliseconds, between 1000 and 60000):

curl -N https://swamp.example.com/api/v1/health/stream?interval=10000 \
  -H "Authorization: Bearer health-monitor.secret"

To resume after a disconnect, pass the last received event ID:

curl -N https://swamp.example.com/api/v1/health/stream \
  -H "Authorization: Bearer health-monitor.secret" \
  -H "Last-Event-ID: 42"

When the token is revoked, rotated, or expires, the stream ends with a final session-ended event and closes:

event: session-ended
data: {"code":4003,"reason":"Session revoked: token revoked"}

Reconnect with a new token. A token may hold 10 open streams; beyond that the server answers 429 with Retry-After: 30. See REST API — GET /api/v1/health/stream for the close codes and limits.

Query full run history

The /internal/runs endpoint returns all run records, including completed and failed runs. It is disabled by default, and requires an admin token.

To enable it, start the server with any of:

swamp serve --enable-internal-api
# swamp-serve.yaml
enable-internal-api: true
export SWAMP_ENABLE_INTERNAL_API=true

Then query it with an admin token, such as one minted for a principal in --admins:

curl https://swamp.example.com/internal/runs \
  -H "Authorization: Bearer paul-token.secret"
← PreviousLoad External Grants at StartupNext →Enable Managed Config for Multi-Instance Serve