#3665 Serve: execute independent webhook-triggered workflows concurrently

Request

Support concurrent webhook-triggered workflow executions in swamp serve. Two independently accepted deliveries should be able to start immediately without waiting for the first workflow to finish. This is a core serve dispatcher feature, not a request to change application-level signal correlation or triage.

Current behavior and evidence

Verified in deployed swamp 20261019.172240.0-sha.148237ef, src/serve/webhook.ts, WebhookHandler processQueue():

if (this.processing) return;
this.processing = true;
while (this.runQueue.length > 0) {
  // persist/replay bookkeeping omitted
  const entry = this.runQueue.shift()!;
  await withoutVaultAccess(() => this.executeWorkflow(entry));
}

The dispatcher awaits the entire workflow before starting the next delivery. Workers and workflows otherwise support parallel execution, and scheduled workflows have configurable concurrency, but this webhook path is serial with no webhook concurrency setting in the inspected implementation.

On ops.swamp-club.com, the github-event-router workflow validates a delivery then invokes a nested GCP build → UAT → publish workflow. The parent stays running throughout. Swamp PR #3220 (registry transient-failure retries) merged at 2026-10-20 14:25:31 UTC. Its merge webhook reached ops at 14:25:33 UTC and received HTTP 200 with:

{"status":"queued","workflow":"github-event-router"}

Its release had still not started more than 15 minutes later because an earlier release was running UAT. Earlier release run: 7c4f241b-1e3d-429b-ac96-f2817235e170, binary 20261020.142238.0-sha.158c9fbf; nested UAT run: d107af6d-9ae8-4dd6-a2d7-5ec887fc98fd. The target merge is b8bb7ecc (PR #3220). Per-run UAT VMs and model instances already isolate executions, so the webhook dispatcher is the bottleneck.

This also affects alert/signal intake: a long-running investigation should not prevent another accepted signal from starting its workflow. Combining signals into one incident is separate application logic and is not part of this request.

Desired behavior

• Start independent accepted deliveries concurrently, including deliveries to the same route/workflow when their executions are isolated.
• A slow webhook workflow must not globally block unrelated webhook routes.
• Do not replace the hard-coded serial behavior with a mandatory low concurrency cap. If optional admission/concurrency controls are exposed, allow operators to explicitly choose concurrent execution without an artificial single-run bottleneck; document defaults and semantics.
• Keep existing model-level locking and application-level ordering guarantees; concurrency does not imply that shared mutations or stable release publication may race unchecked.

Acceptance criteria

• Deterministic test: delivery A starts a workflow that remains blocked; delivery B starts and completes before A is released. Cover both same-route and different-route deliveries.
• Each run retains its own principal/vault scope, inputs, delivery/run identity, cancellation state, traceparent and tracestate. The processor must not leak the first delivery context into another execution.
• Preserve durable pending-run bookkeeping, replay/recovery and existing delivery deduplication semantics. Starting overlapping runs must not silently lose or double-dispatch pending entries.
• Preserve drain/shutdown behavior: reject new admissions as appropriate, stop launching pending entries during drain, track/await or abort all in-flight executions, and retain pending entries for recovery under the existing guarantees.
• A failed workflow must not stall other executions or leave the dispatcher permanently marked busy.
• Expose active versus pending webhook executions so operators can distinguish admission delay from worker capacity or model-lock waits.
• Document the concurrency behavior, relevant configuration and any compatibility implications.

A supported durable detached-workflow launch could be useful separately, but operators should not have to build a custom scheduled dispatcher or background shell process merely to process independent webhook deliveries in parallel.

Known workaround and why we are requesting native support

We understand there is a potential workflow-level workaround: keep the webhook workflow short, durably record each accepted request, then return. A separate dispatcher can claim those requests and launch independent workflows concurrently. A supported durable detached launch, if available, could serve the same purpose. We are not claiming that parallel downstream work is impossible today.

The workaround is architecturally unclean for this use case: it adds an application-owned dispatch layer solely to bypass serve’s serial webhook execution. We would need to maintain request storage, claiming/deduplication, crash recovery, delivery-to-run correlation, and dispatch latency. A shell-level fire-and-forget launch is even less suitable because its durability and lifecycle guarantees are weaker.

This feature asks swamp to own concurrency at the natural boundary—the webhook dispatcher—rather than requiring each deployment to reconstruct that orchestration in its workflows. The workaround is acknowledged; native support is preferable for clarity, reliability, and operational simplicity.