Skip to main content
← Back to list
01Issue
FeatureShippedSwamp Club
Assigneesstack72

Relationships

#948 Docs: update remote execution references for elastic queueing

Opened by stack72 · 7/3/2026· Shipped 7/3/2026

Summary

PR1 of worker-fleets phase 1 (issue #947) changes the scheduling behavior for placed steps: an empty or unmatched worker pool now queues the step instead of failing it immediately. Three documentation pages need updates to reflect the new behavior before this ships.

Important: the current user-facing behavior is unchanged for the tutorial and how-to paths (worker is connected before the workflow runs, so steps never queue). These doc changes describe the new timeout mechanics and the new queueTimeout step field so users who encounter the queueing behavior understand what's happening.

Changes needed

1. content/manual/reference/swamp-serve/serve-flags.md

Add --queue-timeout to the flags table:

Flag Default Description
--queue-timeout <duration> 60s How long a placed step queues for a matching worker before timing out. Accepts seconds (60), explicit units (2m, 10m), or 0 to wait forever

No environment variable exists for this flag yet (that lands with phase 2's env-var configuration work).

2. content/manual/reference/remote-execution/workflow-placement.md

Add a queueTimeout field to the "Placement Fields" section (after labels):

### `queueTimeout`

Overrides the server's default queue timeout for this step.

| Property | Value  |
| -------- | ------ |
| Type     | number (seconds) |
| Required | No     |
| Default  | Server default (60s) |

When a step is queued waiting for a matching worker, it times out after this
many seconds. Set to `0` to wait indefinitely. The per-step value overrides
the server's `--queue-timeout` flag.

Update the "When no worker matches" section to mention the timeout behavior. The current text says the step "waits" but doesn't explain what happens when the wait ends. Add:

  • The step times out with an error message naming the unmet placement requirement (e.g., "Timed out waiting for a worker matching labels gpu=true to become available")
  • The timeout is layered: per-step queueTimeout > server --queue-timeout flag > 60s default
  • 0 at either layer means wait indefinitely

3. content/manual/reference/remote-execution/failure-semantics.md

Add a new section "Queue timeout" (before or after "Cooperative cancellation") covering:

  • When a placed step is queued because no matching worker is available, it waits for a configurable timeout
  • If no matching worker appears within the timeout, the step fails with an error naming the unmet placement requirement
  • The timeout is layered: per-step queueTimeout (seconds, in workflow YAML) > --queue-timeout flag on swamp serve > default 60s
  • 0 means wait forever — the step queues indefinitely until a matching worker enrolls or the workflow is cancelled
  • A queue timeout is an ordinary step failure — downstream steps with dependsOn: failed conditions will fire

Files verified as needing NO changes

  • content/manual/explanation/remote-execution.md — conceptual architecture, no timeout/scheduling specifics
  • content/manual/tutorials/remote-execution.md — tutorial connects the worker before running the workflow, so the step never queues; behavior is identical before and after this change
  • content/manual/how-to/remote-execution.md — covers deployment and troubleshooting, not scheduling

Context

These docs should ship alongside or before PR1 of #947 lands. The behavior change is backward-compatible (steps that previously dispatched immediately still do; steps that previously failed immediately now queue for up to 60s), but users who encounter the new queueing behavior need to understand why their step is waiting and how to configure the timeout.

02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 2 MOREREVIEW+ 4 MOREPR_MERGED+ 1 MORENOTIFICATION_SKIPPED

Shipped

7/3/2026, 11:53:38 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack727/3/2026, 11:39:55 PM
Editable. Press Enter to edit.

stack72 commented 7/3/2026, 11:07:17 PM

PR #1747 adds two things that affect these doc changes:

1. SWAMP_QUEUE_TIMEOUT env var now exists. Add to the env var table in serve-flags.md:

| SWAMP_QUEUE_TIMEOUT | Default for --queue-timeout (default: 60s) |

Remove the note in this issue that says "No environment variable exists for this flag yet."

2. Validation warning for queueTimeout without placement. Add a note to the queueTimeout field description in workflow-placement.md:

swamp workflow validate warns when queueTimeout is set on a step that has no placement fields (target, labels, or platform). The timeout has no effect without placement — the step runs locally on the orchestrator.

Also minor: help text says "0 to disable" (not "0 to wait forever") for parity with --ws-idle-timeout. Match that wording in the docs.

stack72 commented 7/4/2026, 12:42:58 AM

PR2 (#949) adds one more doc change needed:

content/manual/reference/remote-execution/worker-commands.md — add a swamp worker queue section between swamp worker list and swamp serve. Also update the page description frontmatter to include the new command.

Content:

## `swamp worker queue`

List steps currently queued for dispatch, waiting for a matching worker.

\`\`\`
swamp worker queue [--server <url>] [--repo-dir <dir>]
\`\`\`

When steps are waiting:

\`\`\`
$ swamp worker queue
\`\`\`

\`\`\`
REQUIREMENT          STEP              MODEL          QUEUED AT                   AGE
labels gpu=true      train-model       ml/trainer     2026-07-04T10:32:15.000Z    3m
target build-node    compile           builder/go     2026-07-04T10:34:02.000Z    1m
\`\`\`

When the queue is empty:

\`\`\`
$ swamp worker queue
No steps are currently queued.
\`\`\`

### JSON output

\`\`\`
$ swamp worker queue --json
\`\`\`

\`\`\`json
[
  {
    "queueId": "a1b2c3d4-...",
    "requirement": "labels gpu=true",
    "stepName": "train-model",
    "modelType": "ml/trainer",
    "methodName": "run",
    "workflowName": "training-pipeline",
    "queuedAt": "2026-07-04T10:32:15.000Z",
    "ageMs": 180000
  }
]
\`\`\`

### JSON fields

| Field          | Type     | Description                                        |
| -------------- | -------- | -------------------------------------------------- |
| `queueId`      | `string` | Unique identifier for this queue episode           |
| `requirement`  | `string` | Human-readable placement requirement               |
| `stepName`     | `string` | Workflow step name (if available)                   |
| `modelType`    | `string` | Model type being dispatched                        |
| `methodName`   | `string` | Method being invoked                               |
| `workflowName` | `string` | Workflow containing the step (if available)         |
| `queuedAt`     | `string` | ISO 8601 timestamp when the step entered the queue |
| `ageMs`        | `number` | Milliseconds since the step was queued             |

Steps appear here when they are waiting for a matching worker and disappear
once dispatched, timed out, or cancelled. Use this to diagnose why a workflow
is stalled and to inform autoscaling decisions.

The same data is queryable via `swamp data query`:

\`\`\`bash
swamp data query 'modelType == "swamp/pending-dispatch" && attributes.state == "waiting"'
\`\`\`

This is the last doc change for phase 1. After this, #948 covers everything users need.

Sign in to post a ripple.