DEPLOY SWAMP SERVE ON KUBERNETES

This guide deploys swamp serve with raw Kubernetes manifests. It starts with the official Swamp image, uses managed config for repository definitions, and leaves datastore, credential, and certificate management to systems you operate separately.

The initial vault, model, workflow, and extension configuration must be complete before you enable managed config. This guide treats that migrated configuration as an immutable release: it does not create or edit repository definitions after migration.

Prerequisites

Prepare:

A Kubernetes cluster and kubectl
A free swamp-club.com account with owner or admin access to a collective
A Swamp repository with a stable repoId, initialized with the Swamp release you will deploy
A sync-capable datastore service and extension that support managed config
Datastore credentials that the provider's credential chain can resolve from the Serve Pod
Access to the official swampclub/swamp:latest image
An optional persistent external vault provider when repository definitions use vault-backed secrets
An external system for creating Kubernetes Secrets
An ingress or load balancer that supports WebSockets
Network access from each new Serve Pod to the Swamp extension registry, unless you build a custom image containing every pinned extension

Use a Swamp CLI release compatible with the Serve image when preparing and migrating the repository.

Set variables for the commands in this guide:

export NAMESPACE=swamp-system
export SWAMP_URL=wss://localhost:9090
export SWAMP_COLLECTIVE=your-collective-slug

The manifest uses swamp-system in internal DNS names because Kubernetes does not expand shell variables in YAML. If you choose another namespace, replace every swamp-system occurrence in swamp-serve.yaml before applying it.

Log in with your personal account, then create the collective token used by Serve. The account must be an owner or admin of $SWAMP_COLLECTIVE:

swamp auth login
swamp auth token create \
  --collective "$SWAMP_COLLECTIVE" \
  --scopes 'serve:*' \
  --name k8s-serve \
  --json

The JSON key is shown once. Store it immediately in your external secret system as SWAMP_API_KEY; do not commit it or put it in a manifest. serve:* authorizes Serve startup.

Create the namespace:

kubectl create namespace "$NAMESPACE" \
  --dry-run=client --output yaml | kubectl apply --filename -

Rollout milestones

Treat the deployment as a sequence of independently verifiable milestones. Do not enable the next layer until the current gate passes.

Select the image. Start with swampclub/swamp:latest, inspect its Swamp version, and pin a tested release before production. Gate: Keep the publishing CLI and Serve versions compatible.
Complete the initial repository. Pin required extensions and validate all models and workflows before enabling managed config. Create any persistent vault definitions at this stage too. Gate: Require every definition and optional vault lookup to pass locally.
Publish managed config once. Configure the datastore and run the config migration from one instance. Gate: Confirm the datastore identity and namespace, and copy the resulting .swamp.yaml exactly for Serve bootstrap.
Deploy Serve. Supply datastore credentials, authentication configuration, and internal TLS. Gate: Require a successful rollout and HTTP 200 from /ready. Check logs for completed hydration, not merely a live process.
Verify migrated config. Query the model, workflow, and optional vault inventories in the hydrated repository. Gate: Require the expected migrated items to appear.
Prove Serve recovery. Delete the Serve Pod and wait for its replacement to hydrate. Gate: Require /ready and the managed-config inventory checks to pass again.
Release normal traffic. Apply the Ingress after the recovery gate passes. Gate: Verify WebSocket upgrades through the public endpoint, and retain the previous manifest, image reference, and datastore snapshot for rollback.

The datastore is not an optional post-deployment attachment. You can create the Serve Deployment before publishing application workloads, but managed config and its bootstrap credentials must be reachable before Serve can pass /ready.

Use the official image

Use the official image for the Deployment initially:

image: swampclub/swamp:latest
imagePullPolicy: Always

For production, replace latest with a tested release tag or manifest digest.

Inspect the version embedded in an image before initializing or upgrading the publishing repository:

docker run --rm swampclub/swamp:latest --version

After deployment, swamp version --server "$SWAMP_URL" --json reports the running Serve version. The swampVersion in .swamp.yaml belongs to the repository and must not be reconstructed from an image label.

Prepare the initial managed configuration

Configure the remote datastore from one checked-out copy of your Swamp repository. This example uses S3; use the type and configuration required by your datastore provider:

swamp datastore setup extension @swamp/s3-datastore \
  --config '{"bucket":"operations-swamp","prefix":"swamp","region":"us-east-1"}' \
  --namespace operations \
  --hydration-strategy lazy

The provider's credential resolver must work both here and in the Serve Pod. Depending on the provider and cluster, that can mean a workload identity, projected identity, mounted credential file, or environment variables. Test the same mechanism from a Pod before migration; Kubernetes successfully starting a Pod does not prove that the datastore SDK can resolve credentials.

Complete all model, workflow, vault, and extension definitions before migration. Pin each pulled extension to a version tested with the selected Serve image.

Model definitions can obtain secrets through either of these paths:

Environment-backed CEL: inject variables into the appropriate Deployment and reference them as ${{ env.MY_API_KEY }}.
Persistent vault provider: create an external vault before migration and reference it with ${{ vault.get("configured-vault-name", "item-name") }}.

Do not use local_encryption for Kubernetes Pods because its key and encrypted values live in the ephemeral home directory. For a persistent provider, follow one of the Vault how-to guides, verify a secret lookup, and ensure any required runtime binary or certificate is available in the Serve image.

Validate every initial definition while the repository still uses file-backed configuration:

swamp model validate
swamp workflow validate

After the model, workflow, optional vault, and extension lock are complete, migrate the repository configuration into the datastore:

swamp datastore config migrate --json

Run the migration from one instance only. Models, workflows, optional vault definitions, the extension lock, and pulled extension sources move into the managed config tier. Secret values remain in environment variables or the external vault. The bootstrap .swamp.yaml does not move.

Important

Do not create or edit vaults, models, workflows, or extension configuration after this migration for the deployment procedure in this guide. Affected Swamp releases can report a successful managed-config mutation while later reads still return the previous configuration. The one-time migration is not a general publish command, and swamp datastore sync --push does not publish repository definitions into managed config.

Warning

Preserve repoId, datastore coordinates, namespace or prefix, and existing definition IDs during every release. Changing them points Serve at a different configuration, data, and authentication history.

Do not use a vault expression to obtain datastore bootstrap credentials when managed config is enabled. The vault definition is not available until after the datastore has initialized. Supply those credentials through a provider-supported Pod identity, mounted credential file, or the Serve environment Secret instead. See Datastore Configuration for supported backends and bootstrap limitations.

Prepare TLS and environment secrets

Serve must terminate TLS when it listens off loopback. Obtain an internal certificate whose subject alternative names cover at least:

swamp-serve
swamp-serve.<namespace>
swamp-serve.<namespace>.svc
swamp-serve.<namespace>.svc.cluster.local
localhost when following the port-forward checks in this guide

Have cert-manager, your certificate controller, or another external process create a kubernetes.io/tls Secret named swamp-serve-internal-tls. The certificate in tls.crt must be an end-entity server certificate with basicConstraints=critical,CA:FALSE; do not use a CA certificate as the Serve leaf certificate.

For a temporary self-signed test certificate, create a leaf with the required SANs and CA:FALSE:

cat > internal-tls.cnf <<EOF
[req]
distinguished_name = subject
x509_extensions = server_cert
prompt = no

[subject]
CN = swamp-serve.${NAMESPACE}.svc.cluster.local

[server_cert]
subjectAltName = @alt_names
basicConstraints = critical,CA:FALSE
keyUsage = critical,digitalSignature,keyEncipherment
extendedKeyUsage = serverAuth

[alt_names]
DNS.1 = swamp-serve
DNS.2 = swamp-serve.${NAMESPACE}
DNS.3 = swamp-serve.${NAMESPACE}.svc
DNS.4 = swamp-serve.${NAMESPACE}.svc.cluster.local
DNS.5 = localhost
EOF

openssl req -x509 -newkey rsa:2048 -nodes -days 7 \
  -keyout internal-tls.key \
  -out internal-tls.crt \
  -config internal-tls.cnf

Create the serving Secret:

kubectl create secret tls swamp-serve-internal-tls \
  --namespace "$NAMESPACE" \
  --cert internal-tls.crt \
  --key internal-tls.key \
  --dry-run=client --output yaml | kubectl apply --filename -

Operator clients also need a trust anchor for the internal Serve certificate. For the temporary self-signed leaf, trust that exact certificate:

kubectl create secret generic swamp-serve-client-ca \
  --namespace "$NAMESPACE" \
  --from-file=ca.crt=internal-tls.crt \
  --dry-run=client --output yaml | kubectl apply --filename -

For a CA-issued Serve certificate, put the issuing CA chain in the ca.crt key instead. Publicly trusted certificates do not require a custom CA file.

Have your secret controller create the required swamp-serve-environment Secret. It must contain SWAMP_API_KEY, using the collective token created earlier. Add any environment variables required by the datastore credential resolver, vault-provider authentication, or environment-backed CEL expressions. Pod identity or mounted credentials can provide datastore access instead, but they do not replace SWAMP_API_KEY.

Do not commit plaintext credentials to a manifest. Kubernetes Secret values are only base64 encoded unless your cluster adds encryption at rest.

Deploy Serve

Create swamp-serve.yaml with the following resources. Replace the repository identity, datastore coordinates, public host, and authentication administrator before applying it. Copy the post-migration values from the publishing repository's .swamp.yaml; do not derive swampVersion, initializedAt, or repoId from the container image. Add any annotations or projected volumes required by your datastore provider's credential mechanism.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: swamp-serve
automountServiceAccountToken: false
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: swamp-serve-config
data:
  .swamp.yaml: |
    swampVersion: "<copy-from-post-migration-.swamp.yaml>"
    initializedAt: "<original-repository-initialization-timestamp>"
    repoId: "<stable-repository-uuid>"
    tools: []
    gitignoreManaged: false
    autoGc: false
    # S3 example only. Replace this block with the datastore you configured.
    datastore:
      type: "@swamp/s3-datastore"
      managedConfig: true
      namespace: "operations"
      hydrationStrategy: "lazy"
      config:
        bucket: "operations-swamp"
        prefix: "swamp"
        region: "us-east-1"
  serve.yaml: |
    host: "0.0.0.0"
    port: 9090
    tls:
      cert-file: /tls/tls.crt
      key-file: /tls/tls.key
    trusted-hosts:
      - swamp.example.com
      - swamp-serve
      - swamp-serve.swamp-system
      - swamp-serve.swamp-system.svc
      - swamp-serve.swamp-system.svc.cluster.local
      - localhost
    trust-proxy: true
    auth:
      mode: token
      admins:
        - user:cluster-admin
    hydration-timeout: 120s
---
apiVersion: v1
kind: Service
metadata:
  name: swamp-serve
spec:
  type: ClusterIP
  selector:
    app.kubernetes.io/name: swamp-serve
    app.kubernetes.io/component: serve
  ports:
    - name: https
      port: 9090
      targetPort: https
      protocol: TCP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: swamp-serve
spec:
  replicas: 1
  revisionHistoryLimit: 3
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app.kubernetes.io/name: swamp-serve
      app.kubernetes.io/component: serve
  template:
    metadata:
      labels:
        app.kubernetes.io/name: swamp-serve
        app.kubernetes.io/component: serve
      annotations:
        swamp.club/release: "initial"
        swamp.club/secrets-revision: "1"
        swamp.club/tls-revision: "1"
    spec:
      serviceAccountName: swamp-serve
      automountServiceAccountToken: false
      terminationGracePeriodSeconds: 60
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        runAsGroup: 1000
        fsGroup: 1000
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: serve
          image: swampclub/swamp:latest
          imagePullPolicy: Always
          args:
            - serve
            - --repo-dir=/repo
            - --config=/config/serve.yaml
            - --hot-reload
          env:
            - name: HOME
              value: /home/swamp
            - name: DENO_DIR
              value: /home/swamp/.cache/deno
            - name: SWAMP_REPO_DIR
              value: /repo
            - name: SWAMP_LOG_LEVEL
              value: info
            - name: SWAMP_DATASTORE_SYNC_TIMEOUT_MS
              value: "600000"
          envFrom:
            - secretRef:
                name: swamp-serve-environment
          ports:
            - name: https
              containerPort: 9090
              protocol: TCP
          volumeMounts:
            - name: repo
              mountPath: /repo
            - name: config
              mountPath: /repo/.swamp.yaml
              subPath: .swamp.yaml
              readOnly: true
            - name: config
              mountPath: /config
              readOnly: true
            - name: home
              mountPath: /home/swamp
            - name: internal-tls
              mountPath: /tls
              readOnly: true
          readinessProbe:
            httpGet:
              path: /ready
              port: https
              scheme: HTTPS
            initialDelaySeconds: 5
            periodSeconds: 5
            timeoutSeconds: 3
            failureThreshold: 3
          livenessProbe:
            httpGet:
              path: /health
              port: https
              scheme: HTTPS
            initialDelaySeconds: 30
            periodSeconds: 15
            timeoutSeconds: 5
            failureThreshold: 4
          startupProbe:
            httpGet:
              path: /ready
              port: https
              scheme: HTTPS
            periodSeconds: 5
            timeoutSeconds: 3
            failureThreshold: 120
          resources:
            requests:
              cpu: 250m
              memory: 2Gi
            limits:
              cpu: "1"
              memory: 4Gi
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: [ALL]
      volumes:
        - name: repo
          emptyDir: {}
        - name: home
          emptyDir: {}
        - name: config
          configMap:
            name: swamp-serve-config
        - name: internal-tls
          secret:
            secretName: swamp-serve-internal-tls
            defaultMode: 0440

Apply the Serve resources and wait for the rollout:

kubectl apply --namespace "$NAMESPACE" --filename swamp-serve.yaml
kubectl rollout status deployment/swamp-serve --namespace "$NAMESPACE"

Tune hydration-timeout and the startup probe together. Raising only the Kubernetes probe budget does not stop Serve from exiting when its own hydration timeout is reached.

In one terminal, forward the Service port:

kubectl port-forward service/swamp-serve 9090:9090 --namespace "$NAMESPACE"

In another terminal, check the unauthenticated readiness endpoint:

kubectl get secret swamp-serve-client-ca --namespace "$NAMESPACE" \
  --output jsonpath='{.data.ca\.crt}' | openssl base64 -d -A \
  > internal-ca.pem
export SWAMP_CA_CERT="$PWD/internal-ca.pem"
curl --cacert internal-ca.pem \
  --resolve "swamp-serve.$NAMESPACE.svc.cluster.local:9090:127.0.0.1" \
  "https://swamp-serve.$NAMESPACE.svc.cluster.local:9090/ready"

For all Serve configuration fields, see Use a Config File and Serve Flags.

Each cold Serve Pod may resolve missing pinned extensions from the registry before hydration completes. If egress is intentionally blocked, build and test an image that already contains those exact extension versions.

Verify migrated configuration

Confirm that the Serve Pod hydrated the expected pre-migration model, workflow, and optional vault inventories. Inventory search is used here because affected managed-config releases can list definitions even when an individual get or validate read incorrectly reports that the same definition is missing:

kubectl exec deployment/swamp-serve --namespace "$NAMESPACE" -- \
  swamp model search --json
kubectl exec deployment/swamp-serve --namespace "$NAMESPACE" -- \
  swamp workflow search --json
# Run this only when the migrated repository includes vault definitions:
# kubectl exec deployment/swamp-serve --namespace "$NAMESPACE" -- \
#   swamp vault search --json

Compare each result with the source repository's inventory. Do not continue if an expected item is missing or Serve logs do not show completed managed-config hydration. Do not try to repair a missing result by creating or editing it in the running managed repository.

Prove Serve replacement

Delete the Serve Pod to test the recovery path Kubernetes will use during node loss, eviction, and later upgrades:

kubectl delete pod \
  --namespace "$NAMESPACE" \
  --selector app.kubernetes.io/name=swamp-serve,app.kubernetes.io/component=serve
kubectl rollout status deployment/swamp-serve --namespace "$NAMESPACE"

The existing port-forward exits when its selected Pod is deleted. Start it again after the replacement is ready, repeat the /ready request, and repeat the inventory searches from the previous section:

kubectl port-forward service/swamp-serve 9090:9090 --namespace "$NAMESPACE"

Continue only if managed config hydrates from the same datastore namespace, Serve becomes ready without changing credentials, and the expected inventory is still present after replacement.

Publish the Ingress

After Serve replacement passes, create ingress.yaml. This NGINX example terminates public TLS and uses HTTPS to the Serve Service:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: swamp-serve
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: HTTPS
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - swamp.example.com
      secretName: swamp-example-com-tls
  rules:
    - host: swamp.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: swamp-serve
                port:
                  number: 9090

Apply it after replacing the host and public certificate Secret:

kubectl apply --namespace "$NAMESPACE" --filename ingress.yaml
export SWAMP_URL=wss://swamp.example.com
unset SWAMP_CA_CERT

Other ingress controllers must preserve WebSocket upgrades and long-lived connections. When an ingress controller re-encrypts traffic to Serve, force the backend connection to HTTP/1.1. See TLS and Proxies.

Treat the migrated configuration as immutable

This procedure supports an initial managed-config deployment whose vaults, models, workflows, and extension dependencies were complete before migration. It intentionally does not prescribe post-migration configuration mutation.

In affected Swamp releases, vault and model create or edit commands can report success while subsequent reads return missing or stale configuration. Do not use those commands as a release mechanism, do not hand-edit the hydrated cache, and do not use swamp datastore sync --push to publish definitions. The migration sentinel also prevents swamp datastore config migrate from acting as a repeat publication command.

Before adopting post-migration changes, upgrade to a release where your exact vault, model, workflow, and extension create-edit-read sequence passes against a disposable managed-config namespace. Snapshot the datastore with provider-native versioning before that test. Swamp does not provide a generic managed-config snapshot or restore command.

Image-only and Kubernetes-only changes do not mutate managed config. Update the image references and swamp.club/release annotations, apply the manifests, and repeat the Serve readiness, migrated-inventory, and Serve-replacement gates from this guide. Keep the previous manifest, image reference, and provider-native datastore snapshot for rollback.

Know when to restart Pods

Secrets resolved with vault.get(...) are read from the vault provider at execution time, so changing the stored value does not require a rollout.

Kubernetes Secret values injected as environment variables are fixed when a container starts. This includes environment-backed CEL values and credentials that authenticate Serve to the vault provider itself. After changing those values, update the affected Deployment's swamp.club/secrets-revision annotation and apply the manifest.

After changing swamp-serve-internal-tls, update swamp.club/tls-revision on the Serve Deployment. The annotation itself has no runtime meaning; changing it changes the Pod template and causes Kubernetes to replace the Pod.

Account for ephemeral storage

The manifests create no PersistentVolumeClaims:

Serve's /repo and home directories are emptyDir volumes.
Managed configuration and runtime records must survive in the remote datastore.
Every cold Serve Pod hydrates its local cache again.

Allow enough startup-probe and Serve hydration time for legitimate cold starts. The example probe allows ten minutes while hydration-timeout is two minutes. If hydration legitimately needs longer, raise both budgets after measuring datastore latency and metadata volume.

Rollout FAQ

Is `/health` enough to continue?

No. /health proves that the Serve process is alive. /ready proves that startup and managed-config hydration completed. Use /ready as the rollout gate and keep /health as the liveness probe.

How do I know Serve connected to the right datastore?

Check repoId, datastore type, bucket or endpoint, prefix, and namespace against the publishing repository, inspect the hydrated inventory, and check Serve logs for the completed managed-config pull. An empty repository that is healthy but has different coordinates is not a successful deployment.

What should stop the rollout immediately?

Stop on repository identity drift, hydration errors, a non-ready Serve Pod, unexpected data location, missing migrated definitions, or WebSocket upgrade failures. Do not compensate by raising timeouts until the underlying failure is understood.

What must be available for rollback?

Keep the previous Serve image reference, manifest, and a tested datastore snapshot or provider-native version. Preserve repository IDs, definition IDs, and authentication records. Rolling back only the image is not a complete rollback when managed config changed.

Recover or remove the deployment

To stop compute without deleting external state:

kubectl scale deployment/swamp-serve --namespace "$NAMESPACE" --replicas 0

Use your datastore provider or release system to restore the previous managed-config snapshot, then apply the previous manifests. Do not recreate or seed token definitions during a configuration rollback; preserve the durable authentication records already in the datastore.

To remove the Kubernetes resources, delete the manifests you applied:

kubectl delete --namespace "$NAMESPACE" --filename ingress.yaml
kubectl delete --namespace "$NAMESPACE" --filename swamp-serve.yaml

This does not delete the externally managed TLS and credential Secrets or the remote datastore.

Enable Managed Config - migrate definitions into a shared datastore
Datastore Configuration - choose a backend and hydration strategy
API Key Authentication - create, scope, rotate, and revoke the SWAMP_API_KEY
TLS and Proxies - configure internal and public TLS correctly
Monitor Server Health - authenticated health snapshots and streams

← PreviousSet Up a Remote-Only Orchestrator Next →Set Up Extension Trust