Skip to main content
← Back to list
01Issue
BugOpenSwamp CLIPublic
AssigneesNone

Relationships

#1750 Dispatched steps over 15 minutes cannot write results: dispatch credentials expire and never refresh

Opened by skunk-ape · 8/20/2026

Summary

A dispatched model method that runs longer than 15 minutes cannot write its resource. The data-plane call fails with 401 Missing or invalid session credential and the dispatch is recorded as failed — after the work has already succeeded on the worker.

This is the worst shape a failure can take: the side effects happened, the host was changed, and swamp records the step as failed with nothing persisted.

Cause

src/domain/remote/session_credential.ts:

export const DEFAULT_SESSION_TTL_MS = 15 * 60 * 1000;

refresh() in the same file explicitly refuses to renew a dispatch credential:

/**
 * Slide the window forward: re-issue against a currently valid
 * control-channel credential. Does not affect dispatch credentials.
 */
refresh(credential: string): SessionCredentialRecord | null {
  const result = this.verify(credential);
  if (result === null) return null;
  if (result.dispatchId) return null;   // <-- dispatch credentials never renew
  ...

So a dispatch credential is minted once, lives 15 minutes, and cannot be slid forward. src/serve/data_plane.ts:160 then rejects the write:

const auth = this.#authenticate(req);
if (auth === null) {
  return errorResponse(401, "Missing or invalid session credential");
}

I could not find a swamp serve flag that changes the TTL.

Reproduction

  1. swamp serve on the orchestrator; enroll a worker with a label.
  2. Author a workflow whose step is placed on that worker by label, calling a model method that takes more than 15 minutes and then calls ctx.writeResource(...) at the end.
  3. Run it through the orchestrator.

Observed twice, with a model method that installs a large upstream project (Node, Python toolchain, and a Chromium download):

Dispatch d7dcb302-6d53-46c6-8a2f-6361baa02f0d on worker "hermes-vm" "failed" in 1519199ms
✗ Dispatch d7dcb302: @skunk-ape/hermes.install failed in 1519194ms
  — Data plane POST /data/resource failed (401): {"error":"Missing or invalid session credential"}

and again at 27m40s on a second run. The install itself succeeded both times — the binary, the pinned checkout, and the downloaded browser were all correct on the host afterwards. Only the bookkeeping failed.

Why this matters beyond one extension

The remote-execution design record makes the worker the right place to run host-shaped work: installs, builds, image pulls, migrations. Those are exactly the operations that routinely exceed 15 minutes. The current behaviour puts a hard ceiling on how long any dispatched step can be and still record anything, and the ceiling is invisible until you cross it — there is no warning at 14 minutes, and the error names a credential rather than a timeout, so it reads as an auth misconfiguration rather than a duration limit.

It also interacts badly with retries. A no-write step re-dispatches automatically, but a step that failed at its write is treated as having written, so it does not re-dispatch — and there is nothing to resume from.

Suggested directions

Any one of these would resolve it:

  1. Renew while the dispatch is live. The orchestrator already tracks whether a dispatch is active (#dispatchesWithWrites, releaseDispatch). If the dispatch is still open, sliding its credential forward does not widen the trust window in any meaningful way — the credential dies with the dispatch either way.
  2. Mint the credential against the step's own timeout, so a step allowed to run for an hour gets a credential that lasts an hour.
  3. Make the TTL configurable on swamp serve, as a stopgap.

Option 1 seems strictly best: it needs no new configuration and it ties the credential's lifetime to the thing it actually authorizes.

Smaller, separate observation

While diagnosing this I also hit a related sharp edge that may be worth a mention in the remote-execution docs: a dispatched method that shells out can have its wall-clock inflated by orphaned grandchildren holding its stdout pipe open. timeout(1) signals only its direct child, so a killed step can leave descendants alive, and anything that reads the child's output to EOF then blocks long after the command finished. In my case that added thirteen minutes and was what pushed the step past the fifteen-minute ceiling in the first place. That part is my bug to fix, not swamp's, but the combination is what makes the 401 easy to misdiagnose.

Environment

  • swamp 20260820.011410.0-sha.23de2d5a
  • Orchestrator: macOS 15 (arm64), swamp serve --host 127.0.0.1 --port 9099, --auth-mode none
  • Worker: Ubuntu on Lima vz, linux/aarch64, enrolled via swamp worker connect under systemd
02Bog Flow
◉OPEN○TRIAGED○IN PROGRESS○SHIPPED

Open

8/20/2026, 4:39:20 PM

No activity in this phase yet.

03Sludge Pulse

Sign in to post a ripple.