Skip to main content
← Back to list
01Issue
FeatureOpenSwamp CLIPublic
AssigneesNone

Relationships

#3046 serve: run scheduled workflows concurrently (per-workflow serialization), and report queue delay

Opened by randybias · 10/5/2026

Problem

swamp serve runs every scheduled workflow through one global FIFO queue, one run at a time (src/libswamp/workflows/scheduled_execution.ts:687-689, "workflows execute one at a time to avoid lock contention"). No setting changes this. Workers do not change it: a worker with --concurrency 4 sits idle while scheduled runs wait.

On our serve (20261003.192630.0, one worker, s3-datastore on Ceph RGW), 7 scheduled workflows share the queue. Each run spends about 37 s in serve's serialized datastore sync (median pull 32.3 s before a dispatch and push 5.0 s after it; src/serve/sync_gate.ts), so a run of 1 s of work takes 40-110 s. When one per-minute schedule was added, the queue saturated: a 40 */6 * * * fire started 2 h 49 min late, and a 2-59/5 schedule ran half its fires.

Request

  1. Serialize scheduled runs per workflow, not globally (a workflow never overlaps itself; different workflows may run together). Make the global limit a serve setting, e.g. scheduling.maxConcurrentRuns, default 1 for today's behaviour.
  2. Report queue delay: the time from fire to start, per schedule, in /health or the health stream. Today running is the only state, so a 49-minute delay is invisible.
  3. Related: the sync gate makes every dispatch pay a full pull serially. A pull that is scoped to the models the step reads would remove most of the 32 s.

Why per-model locks are enough

Two different workflows that touch the same model already contend on the per-model lock. That is the correct exclusion, and a global queue adds nothing to it.

02Bog Flow
◉OPEN○TRIAGED○IN PROGRESS○SHIPPED

Open

10/5/2026, 8:10:08 PM

No activity in this phase yet.

03Sludge Pulse

Sign in to post a ripple.