Skip to main content
← Back to list
01Issue
FeatureClosedSwamp CLI
Assigneesstack72

Relationships

#871 Model method execution overhead: 10s+ per trivial operation, global lock prevents concurrency

Opened by webframp · 6/28/2026

Problem Statement

Running swamp model method run on a trivial model (shell echo) takes 10-13s per invocation on a local datastore, with the bulk of that time spent in swamp's own machinery rather than the method's work. Combined with the global datastore write lock, this creates a hard throughput ceiling of ~5-6 events/minute regardless of backend performance or concurrency level.

This is a data report from a 60-minute sustained burn-in benchmark across three datastore backends. The intent is to provide concrete numbers for the team to evaluate, not to prescribe a solution.

Benchmark Setup

  • Machine: WSL2, Linux 6.6.87
  • swamp version: 20260627.002632.0
  • Backends tested: Local Docker PostgreSQL (postgres:16), AWS RDS PostgreSQL (db.t3.micro, us-east-1), AWS S3 (us-east-1)
  • Model: command/shell running echo bench_event_<timestamp>
  • Duration: Phase 2 ran 60 minutes per backend with 5 concurrent read workers + 1 sequential write worker

Finding 1: Model Method Execution Overhead

Per-operation p50 latency for model method run (the echo_write measurement) during 60 minutes of sustained load:

Backend echo_write p50 echo_write p95 echo_write mean
Local PG 10,485 ms 11,317 ms 10,404 ms
RDS PG 15,375 ms 15,956 ms 19,268 ms
S3 13,312 ms 14,921 ms 13,674 ms

For comparison, datastore sync and datastore compact (which also acquire the lock but don't execute a subprocess) complete in 1.5-2.8s depending on backend. The ~8-10s delta between a sync and a model method run represents the subprocess lifecycle overhead: lock acquire → shell spawn → output serialization → catalog update → lock release.

The actual shell command (echo) executes in <1ms. Swamp's machinery consumes >99.9% of the wall-clock time for a trivial method.

Finding 2: Global Lock Serialization Prevents Concurrency Scaling

Phase 3 tested event throughput at 1x and 3x concurrency (10 minutes each):

Backend 1x concurrency 3x concurrency Improvement
Local PG 5 events/min 6 events/min +20%
RDS PG 4 events/min 4 events/min 0%
S3 4 events/min 5 events/min +25%

Three concurrent workers hitting the same model yield essentially the same throughput as one. The per-model lock (which appears to be global per-datastore, not per-model-instance) serializes all write operations. This was confirmed by attempting separate model instances (bench-echo-1 through bench-echo-5) — all still contend on the same lock.

Finding 3: Backend Is Not the Bottleneck

The backend-specific I/O cost is modest relative to the method execution overhead:

Operation Local PG p50 RDS PG p50 S3 p50 Network tax
sync 1,616 ms 2,486 ms 2,548 ms +54-58%
compact 1,811 ms 2,349 ms 2,822 ms +30-56%
echo_write 10,485 ms 15,375 ms 13,312 ms +27-47%

The network tax for cloud backends adds 1-5s, but the dominant cost is the local model execution lifecycle. Optimizing the backend won't meaningfully improve the experience.

Sustained Write Throughput (60 minutes)

Total write operations (sync + compact + echo_write cycles) completed in 60 minutes:

Backend Total write ops Ops/sec Write cycle time
Local PG 780 0.217 ~13.8s
RDS PG 447 0.124 ~24.2s
S3 143 0.040 ~75.5s

Zero errors across all backends in 60 minutes of sustained concurrent load (11,905 / 9,657 / 8,501 total operations per backend including reads).

What This Means in Practice

  • The fan-out method pattern (one method handling multiple targets internally) is not just a best practice — it's the only way to achieve reasonable throughput for batch work. The data shows why rule 6 in our project guidance exists.
  • Interactive feel is bounded at ~1.5s minimum (CLI cold-start) plus ~10s for any model method execution, regardless of how fast the underlying operation is.
  • Workflow DAG parallelism across different models still serializes on the datastore lock, limiting the practical benefit of dependsOn parallelism for write-heavy workflows.

Possible Areas to Explore (not prescriptive)

  1. Profiling the method execution path — where in the lock-acquire → subprocess → persist → release cycle is the time concentrated? Is it output serialization, catalog indexing, or lock management?
  2. Lock granularity — could the write lock be per-model rather than per-datastore for operations that don't modify the global catalog?
  3. CLI warm-start / daemon mode — eliminating the 1.5s cold-start per invocation would compound across workflow steps.
  4. Batch output persistence — for fan-out methods producing many outputs, could they be persisted in a single lock-hold rather than re-acquiring per output?

Environment

swamp version: 20260627.002632.0
OS: Linux 6.6.87.2-microsoft-standard-WSL2 (WSL2)
Deno: (bundled with swamp)
Datastore extensions tested: @webframp/postgres-datastore, @swamp/s3-datastore

Full benchmark report with methodology, scripts, and raw CSVs available on request.

02Bog Flow
✓OPEN✓TRIAGED○IN PROGRESS◉CLOSED+ 1 MOREASSIGNED+ 2 MOREREVIEW

Closed

6/30/2026, 1:15:59 PM

No activity in this phase yet.

03Sludge Pulse
stack72 assigned stack726/29/2026, 9:22:20 PM
Editable. Press Enter to edit.

stack72 commented 6/29/2026, 9:55:54 PM

Hey @webframp — thanks for the thorough benchmark. The sustained burn-in data is exactly the kind of report that helps us prioritize.

Two lock fixes landed after your test version (20260627.002632.0) that directly affect the concurrency findings:

  1. 43674faa — fix(lock): prevent subprocess deadlock and infinite busy-spin in per-model lock drain — this wired up SWAMP_LOCK_HOLDER_PID (the design doc documented the contract but it was never implemented) and added a configurable timeout to waitForPerModelLocks. Before this fix the symmetric drain was blocking all per-model locks, which explains the cross-model contention you observed with bench-echo-1 through bench-echo-5.

  2. a451f175 — fix(lock): migrate definition-only commands off global lock — moves 8 definition-only commands (model create, workflow create, etc.) to an unlocked path so they no longer contend with method execution.

We attempted to reproduce your findings on the current build (20260629.145610.0) using a local MinIO container as the S3 backend with @swamp/s3-datastore. Results:

Scenario Your result (20260627) Our result (20260629)
Sequential model method run (echo) 10–13s per op ~1.6s per op
5 concurrent, different models Same throughput as 1x 1.6s total wall clock — perfect concurrency
5 concurrent, same model Same throughput as 1x 6s total (per-model lock serializes, expected)
Sustained sequential throughput 4–5 events/min 36 events/min

Cross-model contention no longer reproduces — different model instances run fully in parallel. Same-model serialization is by design (data integrity).

The one variable we couldn't match is the @webframp/postgres-datastore backend — our test used @swamp/s3-datastore against local MinIO. If the postgres extension's sync implementation has different overhead characteristics, that could account for some delta. Would you be able to re-run your benchmark on the latest build and let us know if the numbers improve?

Run swamp update to get the latest version.

webframp commented 6/29/2026, 11:48:40 PM

Nice, thank you I'll work on it this week!

webframp commented 6/30/2026, 12:19:27 PM

Follow-Up Benchmark: Model Execution Overhead Reduced ~65%

A follow-up benchmark replicating the original 60-min datastore burn-in methodology was run against the latest swamp version (20260629.232503.0 / @swamp/s3-datastore 2026.06.30.1). The same three backends (Local Docker PG, AWS RDS PG db.t3.micro, AWS S3), same command/shell echo model, same 5-read + 1-write worker pattern. Full results at /tmp/swamp-bench/bench2/results/.

Finding 1: echo_write Latency Massively Improved

The model method execution overhead has been cut by 39-65% across all backends:

Backend Original p50 New p50 Change
Local PG 10,485 ms 3,628 ms -65%
RDS PG 15,375 ms 8,690 ms -43%
S3 13,312 ms 8,085 ms -39%

Local PG now completes a model method run in ~3.6s vs the previous ~10.5s floor.

Finding 2: Write Throughput Nearly Doubled

Total write operations (sync + compact + echo_write cycles) in 60 minutes:

Backend Original New Change
Local PG 780 1,515 +94%
RDS PG 447 750 +68%
S3 143 191 +34%

Write cycle time dropped from ~13.8s to ~7.1s on local PG.

Finding 3: RDS PG Now Scales With Concurrency

Event throughput (tight-loop model method run) at 1x and 3x concurrency:

Backend Orig 1x New 1x Orig 3x New 3x
Local PG 5/min 6/min 6/min 7/min
RDS PG 4/min 4/min 4/min 6/min
S3 4/min 6/min 5/min 6/min

Previously RDS PG was completely flat (4 to 4/min at 3x). Now it scales to 6/min (+50%).

Finding 4: S3 Concurrent Write Bug Is Fixed

The overnight workload (2026-06-29) showed 100% write failures on S3 under concurrent load. In this benchmark, S3 completed all 191 write operations with zero errors. The @swamp/s3-datastore 2026.06.30.1 release resolves that regression.

Finding 5: Zero Errors Across All Backends

Backend Total Ops Errors Error Rate
Local PG 12,273 0 0.00%
RDS PG 9,986 0 0.00%
S3 8,155 0 0.00%

Serial Latency (Phase 1) Unchanged

The per-command serial latency is essentially identical to the original. The ~1.5s CLI cold-start and backend network tax (~400ms for reads, ~600-900ms for writes) are unchanged.

Summary

The per-model lock still serializes concurrent writes to the same model, so the architectural concern raised in this issue remains open. However, the ceiling has meaningfully risen:

  • Individual method execution is 2-3x faster (3.6s vs 10.5s on local PG)
  • Sustained throughput is 1.3-1.9x higher per model
  • S3 reliability regression is resolved

Would be good to understand what specifically drove the 65% improvement -- lock path optimization, subprocess spawning changes, or output persistence streamlining -- to inform the remaining architectural work on lock granularity.

stack72 commented 6/30/2026, 1:15:58 PM

@webframp Thanks for this thorough benchmark report — the sustained burn-in methodology across three backends gave us exactly the kind of concrete numbers we needed to evaluate.

Since this was filed, we've landed several changes to the model execution path that significantly reduced the overhead you measured. A follow-up benchmark replicating your original methodology against the latest version (20260629.232503.0) shows:

Method execution latency (echo_write p50):

Backend Original Current Change
Local PG 10,485 ms 3,628 ms -65%
RDS PG 15,375 ms 8,690 ms -43%
S3 13,312 ms 8,085 ms -39%

Write throughput (60 min):

Backend Original Current Change
Local PG 780 ops 1,515 ops +94%
RDS PG 447 ops 750 ops +68%
S3 143 ops 191 ops +34%

Additionally, RDS PG now scales with concurrency (4→6 events/min at 3x, vs flat before), the S3 concurrent write regression is resolved, and all backends ran with zero errors.

These improvements came from model execution path changes — the two-phase sync work hasn't even landed in the datastore extensions yet, so there's further headroom there.

The per-model lock still serializes concurrent writes, so the architectural concern stands in principle, but the practical ceiling has risen substantially. We're going to close this one out given the improvements already shipped, and if lock granularity or the remaining ~3.6s floor become a real bottleneck in practice, we'll file targeted follow-ups.

Thanks again for the detailed data — it directly informed where we focused optimization work.

Sign in to post a ripple.