Relationships
#2922 Tell timeouts apart from cancels in method-run records, and review the hidden 30 s fallback timer for step-called model methods
Opened by hammz · 10/1/2026
Background
Found while fixing swamp-club#2910 (PR #2779). After that change, a workflow step's method run that the run's abort stops is recorded cancelled, in its method-run output and its tracker row, as a standalone swamp model method run already was. Any abort of the run's signal counts as a cancel, so timeouts are recorded as cancels too.
A reason-based distinction was tried and dropped. Three different things abort the step's signal with the same TimeoutError reason:
- workflow run --timeout: abortOnTimeout (src/cli/abort_on_timeout.ts) deliberately aborts the run's controller with the TimeoutError reason that AbortSignal.timeout gives, so the workflow run records the timeout as its cancel_reason. The run itself ends cancelled.
- The cleanup grace: after a cancel, always/completed cleanup steps run under AbortSignal.timeout(CLEANUP_GRACE_TIMEOUT_MS), 30 s (src/domain/workflows/execution_service.ts).
- A fallback timer: buildModelMethodDelegate gives model methods called from a step options.signal ?? AbortSignal.timeout(30_000) (execution_service.ts, added with the step guard feature, a809c490).
Treating every TimeoutError abort as failed would have recorded a workflow stopped by --timeout as cancelled while its steps' method runs were failed, inconsistent with standalone model method run --timeout (which aborts with no reason and records cancelled).
Proposal
- Give the internal timers (cleanup grace, delegate fallback) their own abort reason, so the step executor can tell a user timeout or cancel apart from swamp cutting a step off.
- Record timeouts on method-run outputs distinguishably. The lighter option keeps status cancelled and records the cause (timed out vs aborted) as the output's reason, mirroring the workflow run's cancel_reason. A new timed_out status is the heavier option: it touches ExecutionStatuses (model_output.ts), ActiveRunStatuses and the tracker's SQL guards, the three WorkflowRun status enums, terminal-status sets, trigger conditions (does a timed-out dependency count as failed or completed), serve protocol, history filters, JSON output, swamp-uat schemas and the manual. Decide with a DDD pass.
- Find out when buildModelMethodDelegate runs with no options.signal, and therefore hits the 30 s fallback. If it is reachable, it is an undocumented limit on model methods called from a step: document it, make it configurable, or drop it.
Current behaviour (after #2910)
A method cut off by any of the three timers is recorded cancelled, with error message aborted. Only the fallback-timer case is arguably wrong today.
Open
No activity in this phase yet.
Sign in to post a ripple.