Relationships
#4337 Parallel steps on one model instance lose seconds per lock handoff, and waiters slow the holder
Opened by mtaylor · 11/3/2026· Shipped 11/3/2026
Summary
When parallel workflow steps in one process call the same model instance, they take turns on its per-model lock, as expected. But each turn costs far more than the method: a waiter picks up a released lock up to ~10 s late, and the holder's own method runs several times slower while others wait. Five parallel steps whose methods take ~0.6 s each took 41–60 s; the same five steps run in sequence took 5.1 s.
Observed
swamp 20261022.021804.0-sha.983e2929, Windows 11, local filesystem datastore, one repo, no namespace. A workflow job runs five steps in parallel (dependsOn: []), each a method on one model instance (an HTTP API extension: four searches and a connection check). Times are from OTel traces and --log-level debug lines, seconds since start, run 3 of 3:
| t (s) | event |
|---|---|
| 2.0 | five steps start; one acquires the lock, four log Waiting for lock … held by <same host> (pid <same pid>) |
| 5.0 → 9.1 | holder resolves a vault.get expression (4.1 s; ~10 ms in later steps) |
| 10.3 → 20.8 | holder's HTTP call: 10.5 s |
| 22.5 | holder's method completes |
| 33.1 | next waiter acquires the lock (10.6 s after release) |
| 33.2 → 38.1 | its HTTP call: 4.9 s |
| 40.6 → 46.8 | method done → next waiter acquires (6.2 s gap) |
| 47.6 → 50.2 | last three steps: HTTP calls 0.9, 0.7, 0.6 s, gaps ~0.1 s |
One waiter logged Acquired lock … after 37 retries (41859ms).
The same workflow with the five steps chained by dependsOn, run straight after:
| step | lock acquire | method |
|---|---|---|
| 1 | 6 ms | 0.9 s |
| 2–5 | 2 ms | 0.5–0.6 s |
Whole workflow 5.1 s. With only two parallel steps on one model (another workflow) the overhead was ~0.3 s, so the cost grows with the number of waiters.
Two problems
- Handoff gap. Waiters poll the lock file with a growing backoff, so after a long wait the next retry can be ~10 s after the lock is free. The holder and all waiters are the same process here, so swamp could hand the lock over in process instead of polling.
- Holder slowed by waiters. The holder's
vault.getand HTTP call ran 7–17× slower with four waiters than with none (10.5 s vs 0.6 s for the same kind of call), and got faster as waiters left. Something in the wait loop seems to compete with the holder; we have not found what.
Expected
Parallel steps on one model instance should cost roughly the sum of their methods, as they do when run in sequence. The Lock on behavior … took 41859ms — concurrent runs against one behavior instance take turns on its lock warning explains the queue but not the extra time, which is the larger part.
Related
#520 (lock backoff), #2553 (lock held across datastore pull), #2941 (slow-lock warning advice; mentions #2870 for the backoff, which we can't open).
Shipped
Click a lifecycle step above to view its details.
zack commented 11/3/2026, 4:09:31 PM
Thanks @mtaylor for reporting this! We shipped: Stop lock waiters from freezing swamp on Windows. Every waiter retry checks whether the lock holder is alive; on Windows that check spawns tasklist synchronously, freezing the whole process, including the holder when it is the same process. Fix: a liveness probe of this process's own pid answers alive without probing (no semantic change, removes every probe from the reported scenario), and the Windows probe asks kernel32 in process (OpenProcess + GetExitCodeProcess), falling back to tasklist only when the process cannot be opened. Reproduced in a Windows 11 VM: with 450 processes, 5 parallel steps took 8.6-9.5s vs 5.2s chained; a build with only the own-pid skip took 5.4-5.6s.. The fix has been merged and a release is on its way. We appreciate your contribution to swamp.
Sign in to post a ripple.