Skip to main content
← Back to list
01Issue
BugClosedSwamp CLIPublic
Assigneeszack

Relationships

#4639 Parallel steps on one model instance leave its lock idle between turns: hand it over in process, with a fairness cap for other processes

Opened by zack · 11/11/2026

Description

When several workflow steps in one process call the same model instance, they take turns on its per-model lock. That is intended, since a step holds the lock for its whole method run. But the next step doesn't start when the lock is released: every waiter polls the lock file with backoff that starts at 25 ms and doubles to a 250 ms cap (MODEL_LOCK_RETRY_INTERVAL_MS / MODEL_LOCK_MAX_BACKOFF_MS in src/cli/repo_context.ts). With a queue of waiters, most of them are mid-sleep when the holder lets go, so the lock sits free between turns.

#2870 cut that idle time by capping the backoff at 250 ms. #4337 asked for an in-process handoff ("the holder and all waiters are the same process here, so swamp could hand the lock over in process instead of polling"), but only its Windows tasklist problem was fixed. This issue covers the handoff.

Impact

The bundled build was measured with 50 parallel steps that run echo on one command/shell instance (fresh repo, quiet machine):

  • 49 of the 50 steps logged Waiting for lock. The last acquired the lock after 11 retries and 2.1 s.
  • The acquisitions come in bursts, with 100–230 ms gaps between bursts while every waiter sleeps.
  • Median wall time: 2.96 s in parallel vs 2.05 s for the same steps with concurrency: 1. About 0.9 s, roughly a third of the run, is a free lock with nobody taking it.
  • Run-to-run variance is large. Over 3 runs each, 50 parallel steps took 2.9–6.0 s and 10 parallel steps took 1.2–1.8 s.
  • The overhead grows with queue depth: 10 parallel steps took 1.27 s against 0.62 s serial.

Steps to reproduce

  1. swamp repo init --tool none --no-vault, then swamp behavior create command/shell sh1 and give it an execute method that runs echo bench.
  2. Create a workflow with 50 model_method steps on sh1 / execute, with no dependsOn, so they all start together.
  3. Run swamp workflow run <name> a few times and time it. Then add concurrency: 1 to the workflow and time it again.

Proposed fix

Put an in-process FIFO queue, one per model lock key, in front of the model lock. It wraps the lock that createModelLock returns, so it covers the file lock, serve's lock and custom datastore locks alike.

  • Only the first waiter in a process tries the real lock. Everyone behind it waits on an in-memory promise and doesn't touch the lock file.
  • When a step releases, it wakes the next waiter in the process, which tries the lock immediately instead of finishing its sleep.
  • Every step still really releases and re-takes the lock. The lock file, holder information, heartbeat and TTL, stale-holder cleanup and the global-lock check after acquire stay as they are. So does the datastore pull and push that happens when a model lock is taken and released.
  • Steps get the lock in the order they asked for it, not at random.

Fairness cap

The lock is still shared with other processes: a CLI behavior run, another workflow, serve runs, team members. Without a cap, the next step in this process would always win the race against a waiter in another process that is up to 250 ms into its sleep, so a long workflow could lock everyone else out until it finished.

  • After N consecutive handoffs within the process (for example 8), or once a time budget is used up (for example 2 s of continuous in-process ownership), the next waiter sleeps one retry interval before trying, so outside waiters get a chance.
  • A waiter from outside that wins resets the count.
  • The cap should be a constant next to the existing model-lock settings, with a test showing that an outside waiter acquires within a bounded time while a long in-process queue is draining.

The existing protection against a structural command holding the global lock is unchanged, because every step still re-checks the global lock after acquiring.

Things to get right

  • Cancellation and timeouts: a step that is cancelled, or runs out its wait, while queued leaves the queue cleanly and never blocks the steps behind it. Its 60 s wait still counts from when it started waiting, not from when it reached the front.
  • Fits with the existing layers: the own-lock gate (passOwnLockGate), slow-lock registration (registerDatastoreSyncNamed) and the onLost / lease-lost abort path.
  • Wait warnings and progress: today each queued step logs Waiting for lock … held by <same host> (pid <same pid>). Decide whether a wait behind a step in the same process logs nothing, once per queue, or as now. The run-progress state waiting_for_lock needs the same decision. Waits on another process should keep warning as they do now.
  • Scope: this helps waiters in the same process: workflow steps, forEach iterations and concurrent runs inside one serve. Separate CLI processes contending on one model still poll.

Not proposed

  • Handing the held lock straight to the next step without releasing it. The holder information would go stale. Custom datastores push data only on release, so other machines wouldn't see data until the queue drained. A structural command could wait behind the whole queue.
  • A shorter backoff cap. With 50 waiters, a 50 ms cap means about 1,000 lock attempts a second (exclusive create, holder read, liveness check), and it only removes part of the gap.
  • Waking waiters through filesystem notifications. Unreliable on network filesystems and in containers, every release wakes all the waiters at once, and it still needs polling as a fallback.

#4337 (handoff gap reported, not fixed), #2870 (250 ms backoff cap), #4638 (the other cost that grows with step count in long workflow runs)

Environment

Current main, Linux x86_64, local filesystem datastore, both the default and the --bundle compiled builds.

02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS◉CLOSED+ 1 MOREASSIGNED+ 5 MOREREVIEW+ 13 MORECODE_CONFORMANCE_REVIEW

Closed

11/11/2026, 6:18:24 PM

No activity in this phase yet.

03Sludge Pulse
zack assigned zack11/11/2026, 2:20:44 PM
Editable. Press Enter to edit.

zack commented 11/11/2026, 6:18:24 PM

Closing this without shipping a fix.

We built an in-process handoff for a model instance's lock: parallel steps in one process took turns in memory instead of polling the lock file, with a 5 s fairness yield so other processes could still get in. It passed verification, but we could not demonstrate a real improvement. On a 50-step parallel repro, the timings stayed within the noise of the existing polling behaviour. The only consistent difference was that the per-step "Waiting for lock" warnings went away. That is not enough to justify the extra locking machinery, so we are not merging it.

If this comes back with a reproduction that shows a measurable slowdown, the approach can be picked up again.

Sign in to post a ripple.