UNDERSTANDING KUBERNETES AUTOMATION IN SWAMP

Swamp drives a Kubernetes cluster through behaviors. Each behavior is an instance of a behavior type from an extension, usually @swamp/kubernetes. The behavior is bound to a kubeconfig, a context and, for namespaced types, a namespace. A run calls one of the behavior's methods, which talks to the Kubernetes API and stores what it read as Swamp data. The general relationship between types, behaviors and methods is covered in Behaviors, Types, and Methods. This page looks at how that relationship fits Kubernetes in particular.

One behavior type per resource kind

The @swamp/kubernetes extension follows the shape of the Kubernetes API. There is a type for each resource kind people work with day to day: pod, deployment, statefulset, service, ingress, configmap, secret, rbac, pvc, netpol, hpa, job, event, node and namespace. The sixteenth type, pod-summary, is different. It aggregates data and never touches a resource.

The methods on each type match the kubectl verbs for that kind. Pods have list, get, getLogs and exec. Deployments add scale, restart, pause, resume and getRolloutStatus. Nodes have cordon, uncordon, taint and untaint. Anyone who knows kubectl will recognise these operations. The difference is that each Swamp run is recorded and its results are kept.

Node and namespace are cluster-scoped, as they are in Kubernetes, so these two types have no namespace global argument. A few types group related kinds together. The rbac type covers Roles, ClusterRoles, both kinds of binding, and ServiceAccounts. The job type covers both Jobs and CronJobs. The event type is read-only and is organised around the object an event concerns: a pod, a deployment, a service, or warnings across the namespace.

Choosing scope with global arguments

The namespace, context, kubeconfig and labels global arguments are fixed when the behavior is created, and they define its default reach. Most namespaced methods also accept a namespace input, which overrides the default for a single run.

This gives teams two broad ways to organise their behaviors. Some create one behavior per namespace and per cluster, such as shop-pods on prod-cluster and shop-pods-staging on staging. With this approach the behavior name says exactly what a run will touch, and the stored data for each namespace stays separate. Others create fewer behaviors and pass the namespace input when they need it. That keeps the repository smaller, but it moves the decision about scope from the definition to the command line, where it is easier to get wrong.

The context argument deserves the most thought. A behavior pinned to a production context can only reach production, and a behavior pinned to a development context cannot reach production by accident. For mutating methods such as update, scale, cordon or delete, this fixed binding is often what you want.

What runs write to data

Read methods store the resources they read. A pod list writes a pod record for each pod. A deployment getReplicaSets writes the ReplicaSet history. An rbac listClusterRoleBindings writes every cluster-wide binding. Because these records are kept in the data layer, later behaviors and workflows can read them without contacting the cluster again.

The pod-summary type relies on this. Its sourceName points at a pod behavior, and summarize computes counts by phase and node, restart totals and a healthy/unhealthy split from that behavior's stored pods. The summary is therefore only as current as the last pod list. This is a deliberate split. Collecting from the cluster and analysing the results are separate steps, and either can be repeated without the other.

Node list works a little differently. It syncs the inventory: live nodes are upserted, and the records of nodes that have left the cluster are pruned. The stored node set therefore tracks the cluster as it is now, rather than collecting every node that has ever existed. How long other records last is covered in Data Lifetimes.

Secrets are kept apart from the rest. Secret list writes secretMeta records, which contain only the type and key names. Secret get decodes the values and stores them in a vault, not as ordinary data. Listing secrets for an audit therefore never copies their contents.

How changes are made

The update methods on configmap, secret, service, ingress, deployment and namespace work by reading the resource and then replacing it. This is simple and predictable, but it is not a merge on the server. If someone else changes the resource between the read and the write, their change is overwritten. For resources that several people or controllers edit, keep that window in mind, or serialise changes through a single workflow.

Restarts depend on the kind. Deployment restart and statefulset restart set the restartedAt annotation on the pod template, which triggers an ordinary rolling restart. Pod restart deletes the pod and leaves its controller to recreate it. For pods managed by a Deployment or ReplicaSet, the replacement gets a new name, so a later get with the old name finds nothing. Running list again picks up the new pod.

Bundled workflows

The extension's 14 workflows combine these methods into diagnostics that can be repeated. Some look at the whole cluster: cluster-health, cluster-summary and storage-health. Some look at one area of concern: deployment-status, service-connectivity, autoscaling-status and batch-jobs-status. Some are audits: security-audit, rbac-audit and network-audit. The rest gather inventory or debug a namespace: pod-inventory, pod-health-check, namespace-debug and debug-namespace-deep. The largest of these discovers every resource in a namespace and then iterates over what it found.

These workflows put into a fixed form the sequence of checks an operator would otherwise type by hand during an incident. Since every step stores its results, a run leaves behind a record of the cluster's state at that moment. That record is useful when a problem has already cleared by the time anyone looks. The Workflow Execution Model describes how steps run, and Run Workflows on a Schedule turns a health check into a regular sweep.

Choosing extensions

@swamp/kubernetes is the main toolkit, but it is not the only one.

@john/k8s is a separately published toolkit with 15 types and mostly the same methods, named under @john/. It has no statefulset type. Its ingress, service and netpol create methods do not take a namespace input. Its pod restart takes a waitSeconds input and waits for the controller's replacement to reach Running. A behavior's type is part of its definition, so moving from one toolkit to the other means recreating the behaviors, not editing them.

@bixu/k8s/exec does one job. It runs a command in one pod, or in parallel across every pod that matches a label selector, with a configurable concurrency limit. It requires the kubeContext global argument. It suits fleet-wide checks that the pod exec method, which targets one pod per run, would make repetitive.

@twonines/k8s-fleet works across clusters, not within one. It iterates over every kubeconfig context, or a chosen subset, and probes each for node conditions and pod phases. It reports unreachable clusters in the results and does not stop the run because of them. For teams running many clusters, this answers a different question from cluster-health.

Trust in a third-party extension is a supply-chain decision as well as a question of features. Extension Trust and the Supply Chain discusses how to weigh it.

Credentials and where Swamp runs

Kubernetes behaviors authenticate through the kubeconfig on the machine that runs them. Swamp does not take over cluster credentials. It uses whatever the selected kubeconfig and context allow. This keeps cluster access under the control of the existing Kubernetes RBAC and identity setup. It also means the question of which clusters a run can reach is really a question of which machine runs it.

This becomes important when runs move off a laptop. With remote execution, a run can be placed on a worker. Which kubeconfig each worker holds then decides what each run can touch. Workers running inside a cluster, as in Run a Fleet on Kubernetes, or a server deployed as in Deploy swamp serve on Kubernetes, raise the same question in a different form. Giving each worker only the contexts it needs is the Kubernetes equivalent of pinning a behavior to one context.