Skip to main content

SWAMP SERVE

swamp serve turns a Swamp repository into a remote API. This page explains the architectural decisions behind the server, the security constraints it enforces, and how admin access is managed.

Architecture

A swamp serve instance is a single process that holds a Swamp repository and exposes it over a WebSocket API. Clients connect to run model methods, execute workflows, query data, and manage access. Workers connect to receive dispatched steps for remote execution.

The server is the single source of truth. It owns the datastore, resolves vault secrets, stores model definitions, and manages the policy snapshot. Clients and workers are stateless — they issue requests and receive responses but hold no durable state.

This is a deliberate centralization. A Swamp repository is a coherent unit: definitions reference each other, workflows chain model outputs, and vault secrets are scoped to the repo. Distributing this state would introduce consistency problems that the current model avoids by not having them.

Hard refusals

swamp serve refuses to start in certain configurations. These are not warnings — they are hard failures that prevent the server from binding.

Off-loopback without TLS and authentication. If --host is set to anything other than 127.0.0.1 or ::1, the server requires both --cert-file / --key-file and --auth-mode other than none. This is not configurable.

The reasoning is straightforward: swamp serve is a control plane. It accepts arbitrary model method execution, workflow runs, and vault reads. An unauthenticated, unencrypted control plane on the network is arbitrary remote execution — anyone who can reach the port can run any command. The hard refusal prevents this misconfiguration from reaching production.

On loopback, the risk model is different. The server is only reachable from the local machine, where the user already has access to the repository. TLS and authentication add friction without meaningful security benefit in this context.

That said, --auth-mode none is deprecated. The loopback exception made sense as a convenience for early adoption, but in practice it creates a gap between local and production setups. Workflows, grants, and token-scoped permissions behave differently under none than under token or oauth — issues that surface only at deployment time. Requiring authentication everywhere, including loopback, closes that gap and makes local development a reliable predictor of production behavior.

The token and oauth modes also require Swamp authentication (swamp auth login or an API key with serve:* scope) to start the server itself — not just to connect as a client. The reason is the same trust boundary that drives the off-loopback refusal: these modes turn a local Swamp instance into a network-accessible service that accepts model execution, workflow runs, and vault reads from remote clients. Knowing who stood up the service is part of the accountability chain. The none mode is exempt because it is loopback-only by design and carries no remote access risk.

Remote-only mode

By default, swamp serve runs a local loopback executor alongside the remote dispatch system. Steps without placement run locally on the orchestrator; steps with placement are dispatched to workers. This is convenient for mixed workloads where some steps need remote execution and others do not.

Remote-only mode (--remote-only) disables the local loopback executor entirely. Every step must declare explicit placement — target, labels, or platform — so it can be dispatched to a remote worker. A step without placement fails immediately with:

Step '<name>' has no placement but the server is running in remote-only mode.
Add a placement block (target, labels, or platform) to the workflow step, job,
or workflow so it can be dispatched to a remote worker.

The error is deliberate: in remote-only mode, a missing placement is a configuration mistake, not something to silently run locally. The server treats it as a hard failure rather than falling back.

When to use remote-only mode

Remote-only mode is for dedicated orchestrator instances that should never consume local compute. The common deployment pattern is a lightweight orchestrator — a small VM or container that holds the repository, manages the datastore, and coordinates workers — with all actual execution happening on purpose-built worker machines.

The motivation is resource isolation. The orchestrator process handles WebSocket connections, evaluates access policies, proxies capability requests from workers, and serves the REST API. When it also runs workflow steps locally, those steps compete for the same CPU, memory, and file descriptors. A long-running model method or a memory-intensive step can degrade the orchestrator's ability to serve its coordination role. Remote-only mode eliminates this contention by construction: the orchestrator does coordination, workers do execution.

This is distinct from simply not having any unplaced steps. A workflow that declares placement on every step works the same way with or without --remote-only. The flag's value is enforcement: it catches workflows that accidentally omit placement, which would otherwise silently run on the orchestrator and defeat the isolation.

Configuration

Remote-only mode can be enabled through any of the three configuration surfaces:

  • CLI flag: swamp serve --remote-only
  • Config file: remote-only: true in .swamp/serve.yaml
  • Environment variable: SWAMP_REMOTE_ONLY=true

The same flag is available on swamp serve daemon enable for daemon deployments.

The remoteOnly field is reported in both the root GET / response and the GET /api/v1/health response, so monitoring tools and clients can detect the mode programmatically.

See Serve Flags for the full flag reference, and Set Up a Remote-Only Orchestrator for a practical setup guide.

Admin materialization

The --admins flag lists principal IDs that receive full admin access. On every server start, the server materializes a grant of admin on access:* for each listed principal. This grant implies all actions on all resources.

Materialization is reconciled on every boot. If a principal is removed from --admins, its materialized grant is removed on the next start. If a new principal is added, it receives the grant on the next start. The config is the source of truth — not the grant store.

This design serves two purposes.

Bootstrap. When a server starts for the first time with --auth-mode token, someone needs to be able to mint the first token and create the first grants. The --admins flag provides that bootstrap identity without requiring a separate setup step.

Recovery. If an admin accidentally revokes their own access or corrupts the grant store, restarting the server with the correct --admins flag restores admin access. The config always wins.

Policy snapshot and reload

Grants and group memberships are compiled into a policy snapshot — a precomputed structure that the server evaluates on every access check. The snapshot is rebuilt on swamp access reload or automatically on every change when --grant-reload auto is active.

The default manual reload mode is a safety mechanism. Grant changes are staged until an admin deliberately applies them. A bad grant — an overly broad allow, a deny that locks out the admin — does not take effect until reload. This gives the admin a chance to review and revoke before applying.

The auto mode removes this safety net in exchange for convenience. Teams that trust their grant-creation workflow and want immediate effect use auto. Teams that prefer a review step use manual.

Deny-wins evaluation

When both allow and deny grants match a request, deny wins. This is not configurable.

The alternative — most-specific-wins or last-writer-wins — creates reasoning problems at scale. With deny-wins, a compliance team can create a deny grant that blocks access to sensitive resources, and no amount of allow grants from other teams can override it. The deny is a hard boundary.

This makes the system predictable: if you see a deny grant, you know it applies regardless of what else exists. See the authorization reference for the full evaluation model.

OAuth and collective-based admission

Token authentication requires admins to mint and distribute credentials for every user. OAuth replaces this with self-service login — users authenticate themselves through swamp-club, and the server decides admission based on collective memberships. No tokens to distribute, no credentials to rotate per user.

The device grant flow

swamp serve uses the OAuth 2.0 Device Authorization Grant (RFC 8628) for authentication. The device grant exists for exactly this scenario: a CLI tool running on one machine, with the user approving access in a browser that may be on a different machine.

The flow works like this:

  1. The CLI requests a device code from swamp-club
  2. swamp-club returns a verification URL and a user code
  3. The CLI displays both and polls for approval
  4. The user opens the URL in a browser, signs in, and enters the code
  5. On approval, swamp-club issues an access token
  6. The CLI receives the token and stores it locally

This is the same pattern used by GitHub CLI and other tools that need browser authentication from a terminal. The security boundary is in the browser approval step — the user sees exactly what they are authorizing and can refuse.

The bootstrap client

On first start with --auth-mode oauth, the server needs to register itself as an OAuth client with swamp-club and resolve admin usernames to principal IDs. There are two paths through this — interactive and headless — and the server picks between them automatically based on whether SWAMP_API_KEY is set.

Interactive bootstrap (device grant). Without SWAMP_API_KEY, the server uses the OAuth 2.0 Device Authorization Grant to register. A well-known public client ID baked into the binary initiates this registration — the same pattern GitHub CLI uses. Client ID secrecy is not the security mechanism; the browser approval step is. The admin opens a URL, signs in, and approves the registration.

Headless bootstrap (API key). When SWAMP_API_KEY is set and the token carries the oauth:manage scope, the server registers itself without any browser interaction. It validates the key against swamp-club, confirms the scope, and calls the OAuth client registration endpoint directly. The same key is used to resolve --admins usernames to principal IDs — another step that would otherwise require a device grant.

The decision tree on each boot:

Stored OAuth client credentials in vault?
├─ Yes → use stored credentials (no registration needed)
└─ No  → SWAMP_API_KEY with oauth:manage scope?
         ├─ Yes → headless registration via API
         └─ No  → interactive device grant flow

The headless path exists for containerized and unattended deployments — Kubernetes pods, CI runners, systemd services — where no browser is available. The oauth:manage scope gates client registration because registering an OAuth client is a privileged operation: it creates a credential that can initiate user authentication flows against the collective's swamp-club account. The serve:* scope, required separately for instance registration and heartbeats, does not grant this.

Both paths end in the same state: client credentials stored in the vault. Subsequent starts load the stored credentials and skip registration entirely, regardless of which path created them. SWAMP_API_KEY is read fresh from the environment on each boot, so key rotation is transparent — replace the secret in the deployment environment and the next restart picks it up.

See Deploy with Headless OAuth for the practical setup guide.

Collective-based admission

Collectives in swamp-club serve as the admission boundary. The admin declares which collectives grant access, and swamp-club membership controls who gets in. There is no user management on the swamp serve side — the admin manages team membership in swamp-club, and the server enforces it.

This inverts the traditional approach where the server maintains its own user list. Instead, identity and group membership live in swamp-club, and swamp serve is a policy consumer. Adding or removing a team member is a swamp-club operation, not a server operation.

Collectives on the token

A user's collective memberships are snapshotted on the server token at login time. The server uses these memberships for idp-group: grant matching — connecting swamp-club collectives to swamp serve authorization grants without requiring the server to call swamp-club on every request.

The tradeoff is staleness. If a user is removed from a collective in swamp-club, their server token still carries the old membership until the next group refresh cycle updates it. The --group-refresh-interval (default: 4h) controls how often the server re-fetches memberships from the OAuth provider, bounding the staleness window. Token expiry provides a secondary refresh — when the token expires, re-authentication captures the latest memberships from scratch.

Collectives vs groups

Collectives and IdP groups serve different purposes in the swamp serve authorization model, even though both originate from swamp-club.

Collectives are the admission gate. The --allowed-collectives flag controls who can connect to the server at all. A user must be a member of at least one allowed collective to authenticate. This is a coarse, binary decision: you are in or you are out.

Groups are for grant matching. Once admitted, a user's fine-grained permissions come from grants. Grants can target idp-group:<collective-slug> subjects, which match users whose OAuth token includes that group. This connects swamp-club collective membership to swamp serve authorization without the server maintaining its own group directory.

The separation is deliberate. Admission is a policy the server admin controls (which teams can access this server). Authorization is a policy the access control system controls (what each team can do). Conflating the two — using collective membership as both the admission check and the permission boundary — would mean every collective member gets the same access, which breaks down as soon as teams need different permission levels on the same server.

IdP group refresh and deprovisioning

When a user authenticates via OAuth, the server snapshots their group memberships from the provider's userinfo response. These memberships inform idp-group: grant matching. The question is: what happens when memberships change after login?

The server runs a background refresh loop controlled by --group-refresh-interval (default: 4h). On each cycle, it re-fetches userinfo for every active session and updates the policy snapshot with current memberships. This bounds the staleness window — a user removed from a collective in swamp-club loses matching grants within one refresh interval, not at their next login.

The refresh loop distinguishes two failure modes:

Transient errors are fail-open. If the provider is temporarily unreachable (network timeout, 5xx, DNS failure), the server retains the last known group memberships and retries on the next cycle. The reasoning: a brief provider outage should not revoke access for every authenticated user. The staleness window extends by one cycle, which is acceptable because the memberships were valid at the last successful refresh.

401 Unauthorized is a hard revocation. If the provider responds with 401, the user's session is no longer valid at the identity layer — their token has been revoked, their account disabled, or their consent withdrawn. The server invalidates the session immediately and all idp-group: grants for that user stop matching. This is not a transient condition, and retaining stale memberships would mean the server continues authorizing a user that the identity provider has explicitly rejected.

This asymmetry — fail-open on transient errors, hard-close on 401 — matches the trust relationship between swamp serve and the OAuth provider. The provider is authoritative on identity. When it says "this user is no longer valid," the server honours that immediately. When the provider is just unreachable, the server falls back to its last known state rather than making an irreversible decision based on a network hiccup.

See Serve Flags for the --group-refresh-interval flag reference, and Authorization — IdP group refresh for the behavioral specification.

Two authentication targets

swamp auth login and swamp auth server-login --server authenticate to different systems. swamp auth login authenticates to swamp-club — the platform. swamp auth server-login --server authenticates to a specific swamp serve instance — a team's server. They are different trust relationships stored in different credential files.

A user who is logged in to swamp-club is not automatically authenticated to any swamp serve instance. Each server requires its own login, which issues its own token with its own expiry and collective snapshot.

Token transports on WebSocket

swamp serve accepts tokens on the WebSocket upgrade request via three transports: the Authorization: Bearer header, the Sec-WebSocket-Protocol subprotocol, and the ?token= query parameter. The reason for three is that different client environments have different constraints on what they can attach to a WebSocket upgrade.

The Authorization: Bearer header is the standard HTTP mechanism. CLI clients, server-side code, and tools like curl can set it directly. It keeps the token out of URLs entirely — no query string, no path fragment — so it does not appear in proxy access logs, browser history, or Referer headers that the server or intermediaries might emit.

The browser WebSocket API cannot set custom headers. new WebSocket(url) does not accept an Authorization header — the browser attaches only the URL and an optional list of subprotocols. The Sec-WebSocket-Protocol transport exists for this case: the client passes bearer.<name>.<secret> as a subprotocol value, and the server extracts the token from it. This is a well-known workaround used by other WebSocket-authenticated systems.

The query parameter (?token=<name>.<secret>) is the simplest transport — it works everywhere — but it embeds the token in the URL. URLs are logged by proxies, cached in browser history, and leaked via Referer headers on subsequent navigations. For environments where neither headers nor subprotocols are available, the query parameter still works, but the other two transports avoid this exposure.

When multiple transports are present on the same upgrade request, the server picks the first match in priority order: Authorization header, then Sec-WebSocket-Protocol, then query parameter. The priority favors the transport that leaks the least.

See Serve Flags — WebSocket token transports for the protocol-level reference, and Connect with a token for connection examples.

Webhooks and event-driven workflows

swamp serve --webhook registers an HTTP endpoint that receives external events, verifies their signatures, and triggers workflow runs. This turns Swamp from a pull-based system — where cron schedules and manual runs drive automation — into a reactive one that responds to external events as they happen.

The workflow triggered by a webhook receives the incoming payload through webhook.* CEL variables: webhook.body for the parsed payload, webhook.headers for request headers, and webhook.route for the matched endpoint path. These variables are available in every step of the workflow, allowing steps to branch on event type, extract resource identifiers, or pass payload fields to model methods. See the workflow reference for trigger syntax.

Provider schemes and signature verification

Each SaaS platform signs webhooks differently. GitHub uses HMAC-SHA256 over the raw body. Stripe prepends a timestamp to prevent replay attacks. Linear, Slack, and others each have their own algorithm. Rather than forcing a single verification model, Swamp supports named provider schemes — github, stripe, linear, slack, and generic — each implementing the correct algorithm for its platform.

The generic scheme exists for platforms not explicitly supported. It accepts a configurable header name and optional prefix, covering most HMAC-based verification patterns without requiring a new scheme per provider. The webhooks reference documents the full list of schemes and their configuration.

Secret source indirection

Webhook secrets can be provided as literal strings, @env=VAR (read from an environment variable), @file=/path (read from a file), or @vault=<name>:<key> (read from a named vault). The env and file forms exist because process arguments are visible in ps output — passing a secret as a literal --webhook argument leaks it to anyone who can list processes on the machine. Environment variables and file paths are opaque in process listings.

The vault form goes further: environment variables are per-machine and file paths are per-host, so both require out-of-band provisioning on every machine that runs the server. A vault secret lives in the repository itself (encrypted at rest) and travels with it — any machine that can decrypt the vault can resolve the secret, with no additional setup. The @vault= prefix works with any configured vault backend (local encryption, AWS Secrets Manager, etc.) transparently; the webhook configuration does not change when the backend does. See the vaults reference for vault types and configuration.

Webhooks vs cron triggers

Cron triggers (trigger.schedule in workflow YAML) run workflows on a time schedule. Webhooks run workflows in response to external events. The distinction is pull vs push: cron is for periodic sweeps (check every hour for drift), webhooks are for immediate reaction (deploy when code is pushed).

In practice, many workflows combine both. A webhook provides the fast path — respond to events within seconds — while a cron schedule acts as a fallback sweep to catch events that were missed due to delivery failures or downtime. Neither replaces the other; they complement.

Multi-instance behavior and continuous reconciliation

A single swamp serve instance owns its runs. If that instance dies — process crash, node eviction, OOM kill — its in-flight runs become orphans. Nobody notices, nobody recovers them. For single-instance deployments this is acceptable: the operator restarts the process, and boot-time reconciliation (comparing the run tracker against WorkflowRun YAML files) cleans up. But behind a load balancer with multiple instances sharing a datastore, one instance's death is invisible to the others unless they actively look for it.

HA mode is now auto-detected based on the datastore configuration. When the datastore supports a control plane (e.g. S3), runs are automatically detached from the requesting client connection and tracked in the shared control-plane store rather than tied to a single process. This is the prerequisite for multi-instance reconciliation — without it, runs die with their connection and there is nothing for a peer to adopt.

The --detach-runs flag is deprecated — it is still accepted for backwards compatibility but has no effect. HA mode is determined entirely by the datastore.

Why heartbeats, not health checks

The question is: how does instance B know that instance A is dead? External health checks (a load balancer probe, a Kubernetes liveness check) answer a different question — "is the process responding to HTTP?" — and their failure signal goes to the orchestrator, not to peer instances. A peer needs to detect death itself, without relying on external infrastructure that may not exist or may define liveness differently.

Instance heartbeats solve this. Each instance writes a timestamp to the shared store on a regular interval (--heartbeat-interval, default 30s). A peer that sees a stale heartbeat — one older than --stale-ttl (default 90s) — considers that instance dead and eligible for reconciliation. The constraint that --stale-ttl must be at least 2× --heartbeat-interval provides a margin for transient delays: a single missed heartbeat (GC pause, network blip, busy disk) does not trigger a false positive.

This is a protocol-level decision, not an implementation convenience. The alternative — registering with a coordination service (etcd, ZooKeeper) — would introduce an external dependency that most swamp serve deployments do not have and should not need. The shared datastore already exists; heartbeats reuse it.

Reconciliation and per-instance claiming

When an instance detects a dead peer, it scans for that peer's orphaned runs and adopts them. The scan interval is --reconciliation-interval (default 60s). In practice, a dead instance's runs are detected within roughly one stale TTL plus one reconciliation interval — about 150 seconds with defaults.

Adoption is per-instance and claimed. When instance B decides to reconcile instance A's runs, it claims them atomically in the store before starting work. This prevents a race where instances B and C both detect A's death and both try to reconcile the same runs. Only one claim succeeds; the other instance skips those runs and moves on.

Crash safety and heartbeat deletion

A subtlety: a dead instance's heartbeat record is not deleted until after its orphaned runs have been reaped. The ordering matters for crash safety. If the reconciling instance (B) deletes A's heartbeat first and then crashes mid-reap, A's remaining orphaned runs have no dead-peer signal pointing to them — they become invisible to future reconciliation scans. By deferring the heartbeat deletion, any instance that picks up after B's crash still sees A as dead and can resume the reap.

The run tracker and WorkflowRun files

Two data structures track runs, and they serve different purposes.

The run tracker (SQLite, .swamp/run_tracker.db) is the per-instance liveness authority. It records which runs a specific instance is executing right now, heartbeats them, and detects stale processes by PID. It is inherently local — a PID is meaningful only on the machine where the process runs.

WorkflowRun YAML files (in the shared datastore) are the durable run history. They record the full lifecycle of a run — created, started, steps completed, outcome — and survive instance death. Boot-time reconciliation compares the run tracker against these files to detect runs that were in-flight when the process last died.

In a multi-instance deployment, the shared datastore's WorkflowRun files are what a surviving instance reads during reconciliation. The dead peer's SQLite database is inaccessible (it is on a different machine or in a terminated container), but the WorkflowRun files are shared and authoritative.

See Serve Flags for the full flag and environment variable reference, and Run a Multi-Instance Deployment for a practical setup guide.

Config management and multi-instance convergence

Heartbeats and reconciliation solve run recovery — but multi-instance deployments have a second coordination problem: configuration. When an operative creates a model on instance A, instance B needs to see it. Without a shared config mechanism, operatives must distribute definition files to every instance manually, which is error-prone and scales poorly.

Managed config

The managedConfig flag in .swamp.yaml solves this by storing configuration — model definitions, workflow definitions, vault configs, the extension lockfile, and pulled extension sources — in the datastore tier alongside runtime data. The datastore becomes the single source of truth for config, and each swamp serve instance reads from it instead of from local directories.

This is an opt-in migration, not a default. Existing deployments continue reading config from local directories until swamp datastore config migrate is run. The migration copies config into the datastore's config subdirectory, and sets managedConfig: true in .swamp.yaml.

ConfigPoller

Once managed config is enabled, each swamp serve instance runs a ConfigPoller — a background loop that checks the datastore for config changes at a 30-second default interval. The mechanism is pull-based: instances poll the datastore rather than receiving push notifications.

The poller uses a commitSeq fast-path to avoid unnecessary work. Each config write increments a sequence number in the datastore. On each poll cycle, the poller compares the remote sequence number against its last-seen value. If they match, no config has changed and the cycle completes without reading any files. If they differ, the poller pulls the changed config.

This design makes the polling cost proportional to the change rate, not the config size. A deployment with hundreds of model definitions but infrequent changes pays only the cost of reading one sequence number every 30 seconds.

Cold start

On startup, a swamp serve instance with managedConfig: true performs an early config pull before loading extensions. This ensures the instance starts with the latest shared config rather than whatever was on disk when the container image was built or last deployed. The cold-start pull is a full pull — it does not use the commitSeq fast-path because there is no prior sequence to compare against.

Write propagation

When an operative modifies config through a running swamp serve instance — creating a model, pulling an extension, updating a vault — the instance pushes the change to the datastore immediately rather than waiting for the next sync cycle. This means the change is available to other instances on their next ConfigPoller cycle, typically within 30 seconds.

The push is fire-and-forget from the write handler's perspective: the local write succeeds regardless of whether the push completes. If the push fails (network partition, datastore outage), the change exists locally and will be pushed on the next successful sync. Other instances eventually converge but may lag during the outage.

Why polling, not push notifications

The ConfigPoller uses polling rather than push notifications for the same reason heartbeats use the shared datastore rather than a coordination service: it avoids an external dependency. Push notifications would require a messaging layer (SQS, Redis pub/sub, a WebSocket fanout service) that most swamp serve deployments do not have. The shared datastore already exists, and the 30-second polling interval is acceptable for config changes, which are infrequent and operator-initiated.

The tradeoff is propagation latency. A config change takes up to one poll interval to reach other instances, plus any datastore sync latency. For the intended use case — operatives deploying new model definitions or pulling extensions — this is indistinguishable from immediate.

See Datastore Configuration — Managed Config for the reference, and Enable Managed Config for the setup guide.

← PreviousWorker FleetsNext →The Audit System