Skip to main content
← Back to list
01Issue
BugShippedSwamp CLI
Assigneesstack72

Relationships

#1462 Remote step stdout is not streamed to the workflow runner (log artifact is transferred correctly)

Opened by jasonbarnett · 7/29/2026· Shipped 7/29/2026

When a workflow step is placed on a remote worker with target:, the step's stdout is not streamed into the workflow runner's output. The same step run locally streams it fine. The log artifact itself is captured and transferred back correctly — byte-for-byte identical — so this is purely a live-streaming gap in the runner/gateway, not data loss.

This is what the remote execution tutorial tells readers to expect ("Workflow runner — shows the step completing successfully, with output like: │ executed on a worker"), so following the tutorial gives you a step that looks like it produced nothing.

Environment: swamp 20260727.193526.0-sha.03dabbb5, linux/aarch64, fresh swamp repo init repo, --auth-mode none.

Note: wall-clock timestamps in the sample output are shown as <ts> — the Lab's redaction pass rewrites real HH:MM:SS values as [IP-n].

Reproduction

Set up per the tutorial: a command/shell model whose execute runs echo "executed on a worker", an orchestrator (swamp serve), and a worker enrolled as build-node.

Then run the same model and step through two workflows that differ only in the presence of target:.

Remote (remote-demo):

jobs:
  - name: build
    steps:
      - name: echo-on-worker
        target: build-node
        task:
          type: model_method
          modelIdOrName: remote-echo
          methodName: execute
$ swamp workflow run remote-demo --server ws://localhost:9090

 system │ Starting workflow remote-demo · <ts> UTC

  build │ start <ts> UTC
  build │ step echo-on-worker · remote-echo · execute · start <ts> UTC
  build │ done echo-on-worker in 729ms · <ts> UTC        <-- no stdout line
  build │ completed in 729ms · <ts> UTC

 system │ Completed workflow remote-demo succeeded in 750ms · <ts> UTC

Local control (local-demo, identical but with target: removed):

$ swamp workflow run local-demo

 system │ Starting workflow local-demo · <ts> UTC

  build │ start <ts> UTC
  build │ step echo-on-worker · remote-echo · execute · start <ts> UTC
  build │ executed on a worker                               <-- present
  build │ done echo-on-worker in 78ms · <ts> UTC
  build │ completed in 79ms · <ts> UTC

 system │ Completed workflow local-demo succeeded in 102ms · <ts> UTC

Expected vs actual

Expected: │ executed on a worker appears in the runner output for the remote step, as it does for the local step and as the tutorial documents.

Actual: the line is absent for remote steps. The step is otherwise reported as succeeded.

The data is intact — only the stream is missing

Both dispatch sides log the execution correctly:

orchestrator: Dispatching "command/shell"."execute" (step "echo-on-worker") to worker "build-node" ["ea076d50-538c-490a-a922-4812ef9b4c96"]
worker:       Dispatch "ea076d50-538c-490a-a922-4812ef9b4c96" finished: "command/shell"."execute" "succeeded" in 599ms

result.stdout for the remote run is correct:

$ swamp data query 'modelName == "remote-echo" && specName == "result"' --select 'attributes.stdout'
executed on a worker

And the log artifact from the remote run is checksum-identical to both local runs, which is the strongest evidence that capture and transfer are fine and only the live relay is missing. v1 = local direct method run, v2 = remote workflow run, v3 = local workflow run:

$ swamp data versions remote-echo log
{
  "versions": [
    { "version": 3, "createdAt": "2026-07-29T20:21:25.207Z", "size": 29,
      "checksum": "[REDACTED-SECRET-1]", "isLatest": true },
    { "version": 2, "createdAt": "2026-07-29T20🔞10.162Z", "size": 29,
      "checksum": "[REDACTED-SECRET-1]", "isLatest": false },
    { "version": 1, "createdAt": "2026-07-29T20:17:13.626Z", "size": 29,
      "checksum": "[REDACTED-SECRET-1]", "isLatest": false }
  ]
}

Note the log spec is marked "streaming": true on the stored artifact, which suggests the streaming path is expected to be wired through for remote dispatches too.

Impact

Low severity — no data loss, and the logs are retrievable after the fact via swamp data get remote-echo log. But it's a real observability gap: for long-running remote steps you get no live progress output, and the difference between local and remote execution is supposed to be transparent. It also makes the tutorial's documented output unreachable.

02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 5 MOREREVIEW+ 4 MOREPR_MERGED+ 1 MORECONTRIBUTOR_NOTIFIED

Shipped

7/29/2026, 9:30:20 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack727/29/2026, 8:31:40 PM
Editable. Press Enter to edit.

jasonbarnett commented 7/29/2026, 8:25:39 PM

Companion docs issue: #1461 covers the stale worker token list output and the inline @swamp/workflow-summary block on the same tutorial page. Item 3 there is this issue.

Two notes on the redaction pass that rewrote this body:

  • The three checksum values were replaced with [REDACTED-SECRET-1]. They all collapsed to the same placeholder, which happens to preserve the point — the log artifact from the remote run is byte-identical to both local runs. The real value was a SHA-256 (868f2f74…17030a, 29-byte log), not a secret.
  • Timestamps are shown as <ts> because real HH:MM:SS values get rewritten as [IP-n].

Happy to re-post the raw output another way if the redaction makes any of the evidence hard to read.

stack72 commented 7/29/2026, 9:30:29 PM

Thanks @jasonbarnett for reporting this! The fix has been merged and a release is on its way. We appreciate your contribution to swamp.

jasonbarnett commented 7/29/2026, 9:59:35 PM

Confirmed fixed on 20260729.212638.0-sha.45d9ed58. Thanks for the quick turnaround.

Re-ran the tutorial from a clean swamp repo init with the same A/B setup — identical model and step, one workflow with target: build-node and a control with target: removed. The remote run now streams the step's stdout:

 system │ Starting workflow remote-demo · 21:57:54 UTC

  build │ start 21:57:54 UTC
  build │ step echo-on-worker · remote-echo · execute · start 21:57:54 UTC
  build │ executed on a worker
  build │ done echo-on-worker in 675ms · 21:57:54 UTC
  build │ completed in 675ms · 21:57:54 UTC

 system │ Completed workflow remote-demo succeeded in 695ms · 21:57:54 UTC

Still a genuine remote dispatch rather than a local fallback — both sides logged it:

orchestrator: Dispatching "command/shell"."execute" (step "echo-on-worker") to worker "build-node" ["6ba825ce-458b-4f47-9fab-e09e43715c87"]
worker:       Dispatch "6ba825ce-458b-4f47-9fab-e09e43715c87" finished: "command/shell"."execute" "succeeded" in 552ms

And the log artifacts are still byte-identical across local-direct (v1), remote (v2), and local-workflow (v3), so the fix added the live relay without disturbing the transfer path:

$ swamp data versions remote-echo log
{ "version": 3, "createdAt": "2026-07-29T21:58:29.971Z", "size": 29,
  "checksum": "868f2f7475b9a29b2cab235f3f951c44fff20fdf98d41c07913744a95617030a" }
{ "version": 2, "createdAt": "2026-07-29T21:57:54.830Z", "size": 29,
  "checksum": "868f2f7475b9a29b2cab235f3f951c44fff20fdf98d41c07913744a95617030a" }
{ "version": 1, "createdAt": "2026-07-29T21:57:30.621Z", "size": 29,
  "checksum": "868f2f7475b9a29b2cab235f3f951c44fff20fdf98d41c07913744a95617030a" }

Incidentally, this comment is also a check on #1463 — it's posted with redaction enabled and contains literal HH:MM:SS timestamps and full SHA-256 digests, both of which needed obfuscating to survive submission before. If it reads cleanly above, that fix is confirmed too.

One knock-on to flag, raised on #1461 rather than here: the tutorial's sample runner output was updated to drop the │ executed on a worker line at about the same time this fix restored it, so the docs and the CLI have now crossed over.

Sign in to post a ripple.