Relationships
#520 Per-model LockTimeoutError at 60s causes cascading failures under concurrent access
Opened by webframp · 6/1/2026· Shipped 6/24/2026
Summary
Under concurrent access (25 parallel subagents hitting the same model instance), a single long-running model method or workflow step holds the per-model datastore lock for >60s, causing all other concurrent callers to receive LockTimeoutError. This cascades — once one process times out, the next queued caller also hits the 60s limit, creating a chain of failures.
Environment
- swamp version:
20260530.005533.0-sha.1c117111 - OS: Linux (WSL2)
- Single model instance of type
@webframp/gitlabwith ~45 extension methods - Local filesystem datastore (no Postgres/S3)
Steps to Reproduce
- Create a model instance and register ~45 extension methods on it.
- Spawn 10+ concurrent
swamp model method run <model> <method>calls targeting the same model with varying methods (some fast at ~1s, some slow at ~3.5s). - Meanwhile, spawn 3+ concurrent
swamp workflow runcalls that also target the same model across multiple steps. - Observe lock contention within 1-2 minutes.
Observed Behavior
WRN datastore·lock Waiting for lock "data/@webframp/gitlab/.../.lock"
held by "user@host" (pid XXXXX, acquired XXXms ago)
FTL error LockTimeoutError: Lock "data/@webframp/gitlab/.../.lock"
held by user@host (pid XXXXX) — timed out after 60130msA single workflow process (pid 4130092 in our test) held the lock across multiple minutes (several consecutive lock acquisitions), causing every other concurrent process to time out at 60s. The lock holder would release after ~3 rounds of lock acquisition (~3 minutes), at which point another queued process would take over and repeat the pattern.
Workflow runs wrapped this as:
FTL error Error: 'Workflow execution failed: Lock ".../.lock" held by user@host (pid XXXXX) — timed out after 60135ms'Impact
- ~80% of concurrent model method calls failed during contention periods
- Entire workflow runs failed because any single step couldn't acquire the lock
- No automatic recovery — the contention only resolved when the lock-holding process naturally completed its cycle
Expected Behavior
- Lock acquisition should have a fair queue (FIFO) rather than everyone contending and timing out
- Workflow steps should share or release the lock between steps, not hold it across the entire workflow job
- A configurable timeout or retry mechanism would help
- Consider a read/write lock pattern for read-only methods
Shipped
Click a lifecycle step above to view its details.
webframp commented 6/1/2026, 6:44:42 PM
A second stress test (1 hour, 15 parallel subagents) confirms this is severe and reproducible.
Results
| Pattern | Subagents | Lock Waits | Lock Timeouts (60s) |
|---|---|---|---|
| Rapid dashboard (1-4s sleep between calls) | 3 | ~90 each | ~40 each |
| Mixed methods (get_current_user, list_all_projects, list_todos, 2-6s sleep) | 3 | 69 each | 19-32 each |
| daily-summary workflow only | 3 | ~63 each | 22-33 each |
| health-check + mr-triage workflows | 2 | ~59 each | ~32 each |
| Zero-sleep rapid fire (get_current_user, list_runners, list_todos in tight loop) | 1 | 110 | 45 |
All 9 model-method subagents hit lock waits and timeouts within the first few minutes. The zero-sleep agent was worst at 110 waits and 45 timeouts in 1 hour.
Key observation
Data operations (swamp data list, data query, data search) ran concurrently with zero lock contention — they use a different lock resource. This suggests the bottleneck is specifically the per-model datastore lock, not a global lock.
Impact quantification
- ~30-40% of rapid-fire model method calls failed with LockTimeoutError during contention periods
- 7-16 workflow runs failed entirely per subagent because individual steps could not acquire the lock
- Workflow lock failures cascade: one step timeout fails the entire workflow
Additional note
The lock holder is tracked by PID, but there is no evidence of stale locks (process died while holding). The contention is purely from >1 process queued behind a slow method call like dashboard (~3.5s execution).
webframp commented 6/1/2026, 9:02:31 PM
Retested on version 20260601.163824.0-sha.c2872a24 (15 subagents, 1 hour). Results are essentially unchanged:
| Metric | First test | Retest |
|---|---|---|
| Lock waits (rapid-fire, 3 agents) | ~90 each | ~80 each |
| Lock timeouts (rapid-fire, 3 agents) | ~40 each | ~35 each |
| Lock waits (zero-sleep, 1 agent) | 110 | 99 |
| Lock timeouts (zero-sleep, 1 agent) | 45 | 38 |
The per-model lock is still a hard bottleneck. Data operations (swamp data list/query/search) confirmed zero contention — they use a different lock domain.
Swamp version 163824.0 does not appear to contain a fix for this issue.
webframp commented 6/3/2026, 7:42:31 PM
Confirmed still reproducible on 20260603.160634.0-sha.89593ed4. Ran 13 concurrent callers against a single model instance: 5 LockTimeoutErrors observed within 90s, all at ~60,120ms — same cascade pattern.
webframp commented 6/24/2026, 7:25:56 PM
Baseline burn-in results (15 min, 15 concurrent subagents, single model)
Ran the new lock-contention burn-in framework against burnin-system with default 60s timeout.
| Metric | Pre-fix (Lab #520) | Post-fix (now) |
|---|---|---|
| Error rate | ~80% | 5.4% |
| Lock timeouts | ~45/hr | 43/15min |
| Lock retries | N/A (no backoff) | 530 |
| Lock waits | ~110/hr | 569 |
Backoff distribution: 173×1-retry, 94×2, 81×3, 49×4, 24×5, 37×6, 23×7, 22×8, 19×9, 8×10. No cascading failures observed. Exponential backoff + per-step locking eliminated the thundering herd pattern. More scenarios (extended timeout, fan-out, cross-account inventory) queued up next.
webframp commented 6/25/2026, 12:54:48 AM
Final burn-in results — all 5 scenarios complete
Summary
| Scenario | Duration | Calls | Errors | Error Rate | Lock Timeouts | Retries | Waits |
|---|---|---|---|---|---|---|---|
| Baseline (60s default) | 15m | 893 | 48 | 5.4% | 43 | 530 | 569 |
| Extended (300s timeout) | 60m | 3,240 | 0 | 0.0% | 0 | 2,108 | 2,105 |
| Tight (10s timeout) | 90m | 9,058 | 3,964 | 43.8% | 3,469 | 2,870 | 6,333 |
| Fanout (1 model/agent) | 90m | 28,649 | 0 | 0.0% | 0 | 0 | 0 |
| Inventory (60s, real AWS) | 60m | 3,700 | 314 | 8.5% | 279 | 1,459 | 1,733 |
| Total | 5h15m | 45,540 | 4,326 | 9.5% | 3,791 | 6,967 | 10,740 |
Conclusions
- Baseline 5.4% vs pre-fix ~80% — 15x improvement from exponential backoff + per-step locking
- Extended timeout (300s) eliminated all errors — configurable timeout works correctly
- Tight timeout (10s) degraded gracefully — 43.8% failures but no cascading, no crashes
- Fan-out pattern (one model per caller) is the optimal pattern — 28,649 calls, zero contention
- Real AWS calls under contention (inventory) shows 8.5% timeouts at 60s default — slightly higher than local methods due to longer AWS API latency holding the lock longer
- No cascading failures observed in any scenario — the thundering herd pattern is eliminated
The fix is verified and effective. We consider this issue resolved from the testing side.
Sign in to post a ripple.