Relationships
#3046 serve: run scheduled workflows concurrently (per-workflow serialization), and report queue delay
Opened by randybias · 10/5/2026· Shipped 10/8/2026
Problem
swamp serve runs every scheduled workflow through one global FIFO queue, one run at a time (src/libswamp/workflows/scheduled_execution.ts:687-689, "workflows execute one at a time to avoid lock contention"). No setting changes this. Workers do not change it: a worker with --concurrency 4 sits idle while scheduled runs wait.
On our serve (20261003.192630.0, one worker, s3-datastore on Ceph RGW), 7 scheduled workflows share the queue. Each run spends about 37 s in serve's serialized datastore sync (median pull 32.3 s before a dispatch and push 5.0 s after it; src/serve/sync_gate.ts), so a run of 1 s of work takes 40-110 s. When one per-minute schedule was added, the queue saturated: a 40 */6 * * * fire started 2 h 49 min late, and a 2-59/5 schedule ran half its fires.
Request
- Serialize scheduled runs per workflow, not globally (a workflow never overlaps itself; different workflows may run together). Make the global limit a serve setting, e.g.
scheduling.maxConcurrentRuns, default 1 for today's behaviour. - Report queue delay: the time from fire to start, per schedule, in
/healthor the health stream. Todayrunningis the only state, so a 49-minute delay is invisible. - Related: the sync gate makes every dispatch pay a full pull serially. A pull that is scoped to the models the step reads would remove most of the 32 s.
Why per-model locks are enough
Two different workflows that touch the same model already contend on the per-model lock. That is the correct exclusion, and a global queue adds nothing to it.
Shipped
Click a lifecycle step above to view its details.
stack72 commented 10/8/2026, 11:15:46 PM
@randybias thanks for the detailed report on scheduled runs piling up in serve. The numbers made it easy to see what was happening.
Scheduled runs no longer have to wait behind each other. A new serve setting, max-concurrent-scheduled-runs, lets different scheduled workflows run at the same time. It can go in serve.yaml, the SWAMP_MAX_CONCURRENT_SCHEDULED_RUNS env var, or the --max-concurrent-scheduled-runs flag. A workflow still never overlaps itself. The default stays at 1, so nothing changes until you raise it. It's separate from max-concurrent-runs, which only limits API runs.
You can also see the waiting now. Each schedule in /health, the health stream and the dashboard reports how many fires are queued, when the oldest one fired, and how long the last run waited to start.
Each run still pays its datastore sync time, about 37 s on your setup. Most of the pull before each step is spent inside the S3 datastore extension, which lists and walks the whole namespace even for a one-model pull. We're tracking that separately in swamp-club#3198.
This ships in the next release.
Sign in to post a ripple.