Relationships
#3519 Remote worker step marked failed ('aborted due to timeout') when the result report times out, though the method succeeded and its data was saved
Opened by swamp_lord · 10/15/2026· Shipped 10/16/2026
Summary
A remote worker step whose method completed successfully, and whose resource data was persisted, is recorded as failed with The operation was aborted due to timeout when the result report to serve times out. Nothing retries the report. The stored data then says the step succeeded while the run says it failed. Any workflow that gates on step status fails even though the work was done.
Environment
- serve
20261015.173228.0-sha.95cd2318, local filesystem datastore, about 36 remote workers - The workflow fans out one
model_methodstep per worker; each method runs a test shard (about 2 to 17 minutes) and writes a resource
What happened (2026-10-15, UTC)
Workflow run f73eba6f-caad-4f67-8187-8101047ad752 (@swamp/ci-uat/run) had 36 steps. 20 of them failed with The operation was aborted due to timeout. For each failed step, the resource that the step's own method wrote shows that it completed:
| step | step status | persisted resource |
|---|---|---|
| cli-1 | failed (948 s) | ok=true, passed=208, failed=0, timedOut=false, durationSeconds=775 |
| serve-4 | failed | ok=true, passed=21, failed=0 |
| club | failed | ok=true, passed=57, failed=0 |
| cli-3, cli-4, cli-5, cli-6, serve-5, worker-1, … | failed | ok=true, failed=0 |
Serve logged Dispatch "<id>" on worker "<w>" "failed" in <ms> for each of them, at the moment the method finished. Each step's failure time matches its method's completion time; there is no fixed cut-off. The trigger was serve being stalled by a long GC pass (filed separately). Every step that finished during the stall was lost this way.
Expected
One of:
- The worker retries the result report (with backoff) until serve accepts it or a much longer deadline passes, so a slow or briefly stalled serve doesn't turn a completed step into a failed one.
- If the report truly can't be delivered, the step should end in a distinct state, for example "result unknown / report timed out". A generic
failedhides that the method succeeded and that its data was written.
Step status and persisted data should agree.
Impact
A release pipeline gated on this workflow failed three times in a row with every test passing. The error message (The operation was aborted due to timeout) points at the tests rather than at serve.
Shipped
Click a lifecycle step above to view its details.
swamp_lord commented 10/15/2026, 9:13:25 PM
Trigger for this incident: serve stalled by a 26-minute GC pass, filed as #3518.
Sign in to post a ripple.