Skip to main content

UNDERSTANDING SSH AUTOMATION IN SWAMP

SSH automation tends to start as a shell loop over a list of hostnames and grow from there. Swamp's SSH extensions replace that loop with a behavior. The list of hosts becomes declared data, each connection becomes a method call, and each host's outcome becomes a record that can be queried later. This page looks at why the extensions are shaped that way and where they differ from each other.

One behavior per fleet

@swamp/ssh represents a fleet, not a host. All the machines live in the hosts global argument, and the settings they share live in transport: user, port, identity, proxy jump, authentication mode and ControlMaster behaviour. Any host can override any of these. A bastion-only host can set its own proxyJump, and an old appliance can use a different port, while the rest keep the defaults.

The alternative is one behavior per host. That approach suits @keeb/ssh, which addresses a single machine. For anything run across many machines at once, though, it scatters the configuration and makes "run this everywhere" a matter of orchestration rather than selection. Keeping the fleet in one behavior means the definition is the inventory. Behaviors, Types, and Methods covers how global arguments and methods relate in general.

Selecting hosts

Every @swamp/ssh method that connects takes a hosts selector. The selector can be all, a list of names, a bare name or tag, a value with a name: or tag: prefix, or a CEL predicate with the cel: prefix. The predicate is evaluated against each host with helpers such as cidrContains and matchesRegex available. A bare CEL string without the prefix still works but is deprecated, because a bare value is ambiguous between a name, a tag and an expression.

The resolve method exists because selectors can be subtly wrong, and a wrong selector on an exec with sudo is expensive. resolve answers "which hosts would this match?" without opening a connection. It returns zero matches as data, count: 0, rather than as an error. An empty match is often a legitimate state, such as no canary hosts this week, and a workflow can decide what it means. CEL Expressions describes the expression language.

Two transports

OpenSSH hosts use ControlMaster. The first connection to a host opens a master socket, and later commands reuse it, which avoids repeated handshakes when a workflow makes several calls in a row. open, check and close make the socket's lifecycle explicit, so a workflow can open sockets once, do its work and tear them down.

Tailscale SSH hosts authenticate with tailnet identity and are reached through tailscale ssh, so there is no ControlMaster to manage, and open does nothing for them. Because the transport is a per-host setting, one fleet can contain both kinds. That fits estates that are partway through a migration.

Results as data

Each host a method touches writes its own run-<method>-<host> resource, tagged with fleet, host, method and exitCode. Hosts run in parallel up to defaultParallel, and one failed host fails the method. With failFast set, the first failure stops the rest.

A per-host record matters more than a combined log. Combined output tells you that something failed. A record per host tells you which host failed, with what exit code, and lets you compare that host's result with its previous run. The Data Layer covers how these records are versioned and queried.

Exit codes are where SSH automation usually gets awkward. Many useful commands, such as test -f or grep -q, answer with exit 1, and treating that as failure forces wrappers like || true that also hide real errors. okExitCodes lets the caller say which codes count as answers. The setting replaces the default [0] instead of extending it. An explicit list is safer than any because OpenSSH reports its own failures, such as an unreachable host or a rejected key, as exit 255. With any, those would be accepted too.

Credentials stay in vaults

Key authentication can use a key file, ssh-agent, an IdentityAgent socket, or key content supplied through identityContent. Password authentication goes through sshpass, with the password supplied as a vault.get() reference so that it never sits in the definition. Swamp redacts vault values from output, and the host and selection resources hold addresses, tags and attrs only, never credentials. Storing Secrets shows how vault references behave.

The @hivemq extensions follow the same principle. @hivemq/ssh/keygen writes the private half of a generated key straight into a vault and returns only the public key as data. @hivemq/ssh/rotate-config records which vault key names a rotation workflow should read, not the keys themselves.

Fleets in workflows

Because every step leaves data behind, a workflow can gate on results instead of parsing text. One common sequence resolves a selector, stops if count is 0, runs exec on the matched hosts and then collects host keys. The count tag on the selection resource lets a step check the match count without reading the resource body. The Workflow Execution Model covers how steps, dependencies and gates interact.

Choosing an extension

The SSH extensions overlap less than their names suggest:

  • @swamp/ssh handles fleet operations: commands, scripts, file copies, port forwards and host key collection across selected hosts.
  • @keeb/ssh handles one host at a time, with exec, upload and a wait-for-connection method that suits provisioning, where a machine has only started booting.
  • @hivemq/ssh/authorized-keys is declarative. It describes which keys should exist and converges files toward that list. Its seal method deletes any key not on the list, which suits locked-down hosts but makes it a poor fit where users manage their own keys.
  • @hivemq/hetzner/ssh/authorized-keys is a read-only variant for Hetzner servers that adds password authentication.
  • @evrardjp/ssh-ca replaces key distribution with certificates. Hosts trust a CA, and access is granted by issuing certificates and withdrawn through key revocation lists.

The declarative authorized_keys approach and the certificate approach are alternatives more than complements. Managing key lists is simpler to adopt on an existing estate. A CA scales better as the number of users and hosts grows, at the cost of running the CA itself. The extensions reference lists each type in full.