Skip to main content
← Back to list
01Issue
FeatureOpenSwamp CLI
AssigneesNone

Relationships

#812 Same-namespace writers fully serialize on the per-namespace lock — could maintenance/append writes avoid holding it?

Opened by mgreten · 6/25/2026

Following up on the question I left at the end of #666 about where to track the remaining intra-namespace piece — filing it here so the repro has a clean home. Happy to fold this back into #666 if you'd rather reopen that instead; #666's lock-scoping work genuinely shipped, so a fresh issue felt cleaner than reopening a closed one. This is also the post-per-namespace successor to #520 (same 60s-LockTimeout cascade symptom, now on the S3 datastore's .locks/<namespace>.lock rather than the old per-model/filesystem lock).

What happens

After the per-namespace lock split shipped (@swamp/s3-datastore@2026.06.24.1, thank you), cross-namespace writes no longer contend — but writers within one namespace still serialize completely, with no queuing or fairness. The loser hard-fails at the 60s maxWaitMs rather than waiting its turn.

Repro — 4 concurrent datastore sync against the same namespace, creds healthy (~20ms), lock clean, nothing else running:

  • run A: Sync complete (1 pulled) — held the lock ~50s
  • run B: LockTimeoutError after 60500ms (same holder)
  • run C: LockTimeoutError after 60746ms (same holder)
  • run D: LockTimeoutError after 60266ms (same holder)

So 1 winner + N-1 timeouts. The winner's ~50s hold is dominated by pulling/pushing the whole ~110 MB .datastore-index.json under the lock. Two ordinary same-namespace calls at once (e.g. a workflow step + an unrelated model method in the same repo) reliably produce a 60s timeout on the loser — not an edge case. A data gc that expired just 7 entries held the lock ~5.5 min for the same reason (full-index rewrite under lock).

The thing I keep wondering

Much of what contends here isn't read-modify-write on shared state — it's append / maintenance writes: auto-generated per-run report artifacts (#811), data gc, high-frequency pollers each writing their own data names. They don't conflict with each other's data; they only collide because every write rewrites the one shared manifest under the lock. So the lock is protecting the manifest, not their data.

That suggests the contention might be avoidable rather than just shortenable. A few directions, non-exclusive — wholly deferring to what fits the architecture:

  1. Decouple the write from the manifest rewrite (a commit queue / append log). Writers append their index delta to a cheap per-namespace log and return; a single background compactor coalesces deltas into the manifest periodically. The pollers/gc/artifact writes never block on the big rewrite — one owner pays the merge cost. This is the one I'm most curious about: it would let these writes proceed without waiting on the lock at all.
  2. Incremental / sharded index (the structural fix noted in #666 and #811): a write touches only its own shard, so lock-hold is proportional to the write, not the whole 110 MB. Shrinks the window rather than removing the lock.
  3. Lock fairness / FIFO queue: even if a hold stays long, a queued caller waits its turn instead of hard-erroring at 60s. The floor-level mitigation — turns a hard failure into latency.

(1) and (2) compose; (3) is independently worth having for the failure mode. Our real-world trigger is several launchd pollers writing one namespace, which the per-namespace split correctly doesn't separate — so the within-namespace path is where it bites. Glad to share more profiling or repro detail.

Environment

02Bog Flow
◉OPEN○TRIAGED○IN PROGRESS○SHIPPED

Open

6/25/2026, 4:13:58 AM

No activity in this phase yet.

03Sludge Pulse
Editable. Press Enter to edit.

mgreten commented 6/25/2026, 8:23:15 PM

Adding a consumer-side data point that reinforces the "append/maintenance writes shouldn't contend" framing — observed repeatedly during real multi-process use, not a synthetic repro.

Symptom: a write-on-every-call audit pattern silently degrades under namespace-lock contention

I run an autonomous build/QA pipeline as a long workflow run. Several scheduled pollers and a couple of interactive sessions operate in the same namespace concurrently. Whenever the workflow holds the namespace lock, other model-method calls from the concurrent processes hit the 60s LockTimeoutError you describe — 1 winner + N-1 timeouts, exactly as in your repro.

The part worth flagging: a large share of the contending calls are provider-resolution / decision calls that are read-mostly. The method computes a pure result from its inputs (flags, frontmatter, config maps passed in as args) and then does a single writeResource to persist an audit record of the decision. The decision needs no shared state; the write is append-only observability. But because that write rewrites the shared manifest under the lock, every such call contends with the workflow's lock hold.

Downstream effect: the caller has a local fallback for "swamp unavailable," so when the call times out (~60–90s) it silently falls back to a local computation and proceeds. Net result during any concurrent pipeline run:

  1. Every decision call pays a ~60–90s timeout before falling back, and
  2. The swamp-side decision/routing silently doesn't take effect — the local fallback wins by default. The behavior looks fine (no error surfaced to the user) but the swamp model's logic is effectively bypassed for the whole run.

So this isn't only a latency/fairness problem — for write-on-every-call audit patterns it can quietly nullify the model's intended behavior whenever another writer in the namespace holds the lock.

Why it strengthens the append-log direction

These audit/decision writes are the textbook case for option (1) in your post: they don't read-modify-write shared state, they only collide on the manifest. A commit-queue / append-log path (writer appends its delta and returns; a compactor coalesces) would let high-frequency decision/audit writes stop blocking — and stop being blocked into silent fallback — without weakening correctness for true RMW writers. A cheaper interim mitigation that would also help this class: let a write opt into "append-only, no manifest rewrite under lock" so audit artifacts don't serialize against unrelated work.

Happy to provide more concrete timing traces if useful. Filing as a ripple rather than a new issue per your note that the repro should live here.

Sign in to post a ripple.