WORKFLOW PLACEMENT

Placement fields control where workflow steps execute. Steps without any effective placement run locally on the orchestrator. Steps with placement are dispatched to a matching remote worker.

For the underlying executor model, see Remote Execution. For the full workflow YAML schema, see Workflows.

Placement Fields

Four fields control placement. They can be set at the workflow, job, or step level. A step may declare target or labels, but not both.

target

Dispatches the step to a specific worker by name or instance UUID.

The value must match the worker's registered name (set at enrollment) or its instance UUID. If the string is a valid UUID, it is matched against instance UUIDs first, then worker names.

labels

Dispatches the step to any connected worker whose labels are a superset of the selector.

Every key-value pair in the selector must be present on the worker's label set for the worker to match. Workers register labels at connect time via the --label flag on swamp worker connect — see Worker Commands.

platform

Dispatches the step to a worker running the specified operating system.

The value must match the worker's reported platform (e.g., linux, darwin, windows). Workers report their platform automatically at connect time.

queueTimeout

Overrides the server's default queue timeout for this step.

When a step is queued waiting for a matching worker, it times out after this many seconds. Set to 0 to disable the timeout. The per-step value overrides the server's --queue-timeout flag.

Inheritance

Placement fields follow a three-level inheritance chain: workflow → job → step. Each level inherits from its parent on a per-field basis.

• Omitting a field at a level inherits the parent's value.
• Setting a field at a level overrides the parent's value for that field only — other fields still inherit.
• Setting a field to an explicit empty value (e.g., labels: {}) clears the inherited value. If all placement fields resolve to empty, the step runs locally.

Inheritance is resolved per step before execution:

effective = merge(merge(workflow, job), step)

For each field independently: if the child defines the field, use the child's value; if the child omits it, use the parent's value.

Example: workflow-level defaults

All steps inherit the workflow's labels. The report job clears them, so its steps run locally.

name: deploy-pipeline
labels:
  pool: gke
  fleet: platform-fleet-v3

jobs:
  - name: build
    steps:
      - name: compile
        task:
          type: model_method
          modelIdOrName: builder
          methodName: run
      - name: test
        task:
          type: model_method
          modelIdOrName: tester
          methodName: run
  - name: report
    labels: {}
    steps:
      - name: summarize
        task:
          type: model_method
          modelIdOrName: reporter
          methodName: run

The compile and test steps inherit labels: {pool: gke, fleet: platform-fleet-v3} from the workflow and dispatch to matching workers. The report job sets labels: {}, clearing the inherited labels, so summarize has no placement and runs locally.

Example: job-level override

A job narrows the inherited labels. A step within it pins to a specific worker.

name: ml-pipeline
labels:
  pool: gke
platform: linux

jobs:
  - name: preprocess
    steps:
      - name: transform
        task:
          type: model_method
          modelIdOrName: transformer
          methodName: run
  - name: train
    labels:
      pool: gke
      gpu: "true"
    steps:
      - name: fit
        task:
          type: model_method
          modelIdOrName: trainer
          methodName: run
      - name: evaluate
        target: gpu-box-1
        task:
          type: model_method
          modelIdOrName: evaluator
          methodName: run

• transform inherits labels: {pool: gke} and platform: linux from the workflow.
• fit inherits platform: linux from the workflow and gets labels: {pool: gke, gpu: "true"} from the job (the job's labels replaces the workflow's labels entirely — it is not merged key-by-key).
• evaluate sets target: gpu-box-1, overriding placement for that step only. It still inherits platform: linux from the workflow and labels: {pool: gke, gpu: "true"} from the job.

Example: step-level only

Steps can still declare placement directly, with no workflow or job defaults.

jobs:
  - name: build
    steps:
      - name: compile
        task:
          type: model_method
          modelIdOrName: builder
          methodName: run
        target: build-node
      - name: test
        task:
          type: model_method
          modelIdOrName: tester
          methodName: run
        labels:
          region: us-east
          gpu: "true"

The compile step is dispatched to the worker named build-node. The test step is dispatched to any connected worker whose labels include both region: us-east and gpu: "true".

Scheduling Behavior

When a worker matches

The orchestrator assigns the step to a matching worker immediately. If multiple workers match a labels selector, the orchestrator selects one from the available pool.

When no worker matches

If no connected worker matches the placement, the step is queued. It remains queued until a matching worker connects or the queue timeout expires.

If no matching worker appears within the timeout, the step fails with an error naming the unmet placement requirement (e.g., "Timed out waiting for a worker matching labels gpu=true to become available").

The timeout is layered: per-step queueTimeout overrides per-job, which overrides per-workflow, which overrides the server --queue-timeout flag (default 10m). 0 at any layer disables the timeout — the step queues indefinitely until a matching worker enrolls or the workflow is cancelled.

Affinity

When affinity: true is set on a workflow or job, all remote steps within that scope run on the same worker node. The dispatch service picks a worker for the first placed step using normal target/labels/platform matching, then pins all subsequent placed steps to that same worker for the duration of the scope.

name: stateful-pipeline
labels:
  pool: gke
affinity: true

jobs:
  - name: process
    steps:
      - name: download
        task:
          type: model_method
          modelIdOrName: fetcher
          methodName: run
      - name: transform
        task:
          type: model_method
          modelIdOrName: transformer
          methodName: run
        dependsOn:
          - step: download
            condition:
              type: succeeded
      - name: upload
        task:
          type: model_method
          modelIdOrName: uploader
          methodName: run
        dependsOn:
          - step: transform
            condition:
              type: succeeded

All three steps run on the same worker. The download step selects the worker via labels: {pool: gke}, and transform and upload are pinned to it.

Inheritance

affinity follows the same workflow → job inheritance as other placement fields. A workflow-level affinity: true applies to all jobs unless a job explicitly sets affinity: false.

affinity is not available at the step level — it applies to a group of steps, not an individual step.

Interaction with placement

Affinity requires at least one placement field (target, labels, or platform) to be effective. Without placement, steps run locally on the orchestrator and there is no worker to pin to — affinity: true is a no-op and a validation warning is emitted.

When a step within an affinity scope sets its own target, that step is dispatched to the named worker and opts out of the affinity group. The remaining steps still share their pinned worker.

Worker selection

The first placed step in the affinity scope selects the worker through the normal scheduling path — least-loaded among workers matching the placement selector. Once selected, the worker identity is recorded and all subsequent placed steps in the scope are dispatched to that specific worker, bypassing the normal selection process.

Failure semantics

If the pinned worker disconnects while steps in the affinity group are still pending, those steps fail with WorkerAffinityLostError instead of being silently re-dispatched to another node. This is deliberate: affinity exists because steps share node-local state (filesystem, GPU memory, caches), and re-dispatching to a different node would produce silent data corruption or missing-file errors rather than a clean failure.

A WorkerAffinityLostError is a step failure — downstream steps with succeeded conditions do not trigger, but completed and always conditions do. The workflow can be resumed with swamp workflow resume after the worker reconnects.

Concurrency trade-off

Affinity serializes steps onto a single worker's dispatch slots. A job whose steps would normally fan out across multiple workers runs them sequentially (or limited by the pinned worker's --concurrency slots). For example, a job that completes in 2.6s across two workers may take 5.1s on one.

This is the expected trade-off: affinity trades parallelism for node-local consistency. Use it only when steps need shared node-local resources.

When to use affinity

Affinity is appropriate when steps within a scope need:

• Shared filesystem state — an earlier step writes files that a later step reads, without an intermediate upload/download cycle.
• GPU memory — a model loaded into GPU memory by one step is reused by the next.
• Node-local caches — build caches, dependency caches, or other artifacts that are expensive to recreate on a different node.
• Licensing constraints — software licensed per-node that must not run on multiple nodes simultaneously.

Do not use affinity when steps are independent and can benefit from parallel execution across multiple workers.

Interaction with Non-Placed Steps

Steps without effective target, labels, or platform run locally on the orchestrator through the local loopback executor. Placed and non-placed steps can coexist in the same job. The orchestrator applies the same dependency resolution and execution ordering regardless of where each step runs.

Output, reports, and follow-up actions behave identically regardless of where the step executes. The orchestrator proxies all data access — a remote worker streams results back through the data plane, and downstream steps see the same output shape as if the step had run locally.