#718 forEach with concurrency intermittently fails child workflow runs with 'Bad resource ID'

Description

Workflow forEach steps with concurrency > 1 intermittently fail child workflow runs with error: Bad resource ID before any step executes. The failure is non-deterministic — different items in the same forEach.in list hit it on different cycles. The error string matches Deno's runtime error for an invalid resource handle, suggesting a race in the workflow engine's child-run dispatch, not in the workflow definition.

Affected workflow (excerpt) — fans out one child workflow per Linear ticket, three at a time:

- name: lifecycle-${{ self.ticket.identifier }}
  allowFailure: true
  forEach:
    item: ticket
    in: '${{ data.latest("asdlc-linear", "eligible-" + inputs.projectId).attributes.tickets }}'
  concurrency: 3
  task:
    type: workflow
    workflowIdOrName: '@hivemq/asdlc/issue-lifecycle'
    inputs:
      identifier: ${{ self.ticket.identifier }}
      dryRun: ${{ inputs.dryRun }}

What's recorded in the run YAML for a failed item:

- stepName: lifecycle-PLT-1057
  status: failed
  startedAt: '2026-06-20T14:50:11.292Z'
  completedAt: '2026-06-20T14:50:11.455Z'
  error: Bad resource ID

The child run record exists on disk but has status: failed, no startedAt, and every step pending:

id: 80fd29db-9c50-4217-963d-7cbc1b1f7314
workflowId: 15c02584-6322-4556-b1ae-20f17907faea
workflowName: '@hivemq/asdlc/issue-lifecycle'
status: failed
completedAt: '2026-06-20T14:50:11.446Z'
jobs:
  - jobName: triage
    status: pending
    steps:
      - stepName: fetch-issue
        status: pending
      ...

The error never reaches the daemon log (swamp-server.log) — it's only persisted in the parent run YAML.

Steps to reproduce

1. Define a workflow with a forEach step that triggers type: workflow children, concurrency: 3+, list of 5+ items.
2. Trigger it on a cron (e.g. * * * * *).
3. Observe parent run YAMLs over many cycles — some item invocations fail with error: Bad resource ID and a child run record with all-pending steps.

In our case, across ~50 project-lifecycle runs, every cycle has 1–2 failed children. The same item identifier fails on some cycles and succeeds on others — purely a race.

Environment

• swamp 20260619.201050.0-sha.c1708295 (arm64 macOS)
• Deno-based runtime — error string matches Deno's Bad resource ID (resource table miss for fd/pipe/handle)
• Workflow type: forEach with task.type: workflow (sub-workflow trigger)
• concurrency: 3 on the forEach step

Suspected cause

Deno-level resource (fd/pipe) is closed by one in-flight child dispatch before another sibling reads from it. The forEach scheduler may be sharing a writer/reader handle across concurrent child spawns. Setting concurrency: 1 is the obvious workaround but defeats the parallelism.

Impact

Tickets in the affected forEach list get silently skipped on individual cycles. In our case, an "approved plan, In Progress ticket" can wait many cron cycles before its child run actually executes the implement step. The error is opaque (Bad resource ID with no context) and only visible in the run YAML, not the daemon log — diagnosis took longer than necessary.