Relationships
#829 Intra-namespace write concurrency: whole-index sync under the lock serializes fan-out workloads (split from shipped #666)
Opened by mgreten · 6/26/2026· Shipped 6/30/2026
Summary
On a single high-churn namespace, datastore write throughput is gated by something downstream of the per-namespace lock: every write pulls and pushes the whole index under the lock, so lock-hold scales with total index size rather than with the partition a write touches. For a workload that fans many concurrent writers into one namespace, this serializes everything and produces 60s lock-acquire timeouts.
Per-namespace locking (#666, shipped) solved contention between namespaces and is great. This is the remaining intra-namespace concurrency problem, split out from #666 since that issue is marked shipped. Filing it on its own so it can be tracked, with the full workload context that I don't think was ever on the record.
The workload (the part that wasn't communicated before)
I run an automated dev-workflow system. The shape that matters for the datastore:
- Many pipeline runs execute concurrently, each in its own git worktree (isolated filesystem, branch, ports).
- Each run is event-driven: it historically emitted a datastore write on every phase transition and (for some subscribers) every agent call — so a single run produced dozens of writes.
- All of those writes funnel into one namespace. The worktrees isolate the filesystem, but every run points its datastore calls at the same control-plane repo → same namespace → same lock. Worktree isolation is filesystem isolation, not datastore isolation.
- Separately, ~13 scheduled jobs on one machine plus a second machine write that same namespace continuously.
So per-namespace locking doesn't help this workload: the entire concurrent fan-out lives inside one namespace and serializes on its lock.
Measurements
Sampling the lock object every 2s and attributing each hold to its command, on aligned latest versions:
- Lock-hold p50 ≈ 94–102s, max 268s, across a ~20-minute window.
- The longest holds were spread across completely different operations — a notification send, a small provider lookup, a poller refresh — all converging on the same ~268s ceiling. That uniformity is the tell: the duration tracks the shared whole-index sync, not what the operation does.
- Index was ~93 MB; a single high-cardinality data stream was ~40% of it, inflating the hold for every writer.
Concrete impact: a batch of back-to-back runs that should complete in roughly the low single-digit hours instead ran overnight — the wall-clock was dominated by writers waiting on the lock, not by the actual work.
What I changed on my side (so this is scoped to what's genuinely the datastore's)
I've already removed the avoidable share of this:
- Collapsed a per-invocation unique-named artifact (
resolve-<phase>-<timestamp>) to a stable per-phase name — it was thousands of write-only index entries nothing read back. - Aligned the write unit with the run unit: telemetry-class writes are now buffered in order during a run and replayed as a single batch at completion, so one run holds the lock once instead of dozens of times.
Those cut the number of lock acquisitions dramatically. But the per-acquisition cost (whole-index sync) is unchanged, so concurrent runs and the scheduled writers still serialize on the single lock — that part is the datastore's to solve, which is why I'm filing it.
Two directions that would each independently unblock fan-out workloads
Thinking out loud, not prescribing:
Incremental / scoped index sync — if a write synced only the partition it touched rather than the whole index, same-namespace concurrent writers would stop blocking each other for tens of seconds. This is the more general fix and helps every at-scale user, not just fan-out ones. (The partitioned
_index/shards already exist; this would be making the sync under the lock honor that partitioning.)A lightweight per-run / ephemeral namespace primitive — let a short-lived job cheaply get its own lock scope and fold its data into a parent namespace afterward, without standing up a separate repo checkout and recreating model instances by hand. Today the only way to get a second lock is a second checkout, which is too heavy for a per-run pattern.
What I'm considering if neither lands
If intra-namespace concurrency stays serialized, I'll likely have to break my workload apart at the repo level — a separate checkout (hence separate namespace + lock) per writer-class (pollers vs. pipeline vs. the second machine), and possibly an ephemeral per-run namespace that I provision and tear down myself, harvesting each run's data into a central analytics namespace afterward. That works (my analytics already reassembles from an on-disk source of truth, so runs can live in any namespace), but it's a lot of self-managed namespace plumbing to work around the lock — exactly the kind of thing a primitive like (2), or simply (1), would make unnecessary.
Happy to share the lock-sampling script, the per-run write trace, or profiling data if any of it would help. Thanks again for all the recent datastore work — #788 and the per-namespace locks have both been real improvements even as I work through this.
Environment
@swamp/s3-datastore@2026.06.24.1- swamp
20260625.225837.0 - MinIO backend, two writer machines sharing one bucket
Shipped
Click a lifecycle step above to view its details.
mgreten commented 6/27/2026, 3:29:55 PM
Following up with a concrete workload that makes this issue's impact tangible, in case it helps prioritize.
I'm exploring "sandbox" isolation for the fan-out described above: spinning up N parallel agentic-tooling work-streams, each ideally with its own datastore lock so they stop serializing on the one namespace lock. While prototyping I confirmed the current levers and where they fall short — sharing in case the framing is useful:
What works today (and is genuinely great):
- Per-namespace locking (#666) cleanly isolates between namespaces — verified
.locks/<ns>.lockper namespace. datastore catalog pull --namespaces <ns>+ cross-namespace CEL (#516) give a solid read/merge-back path: I pulled a foreign namespace's full catalog (5,283 rows) into another repo's view in one call. This means a "harvest cold" merge model is buildable in userland.- A runtime
prefixviaSWAMP_DATASTOREgives a fully isolated datastore root + its own lock with zero.swamp.yamledits — nice for throwaway sandboxes.
Where the workaround gets fiddly (the gap that keeps pointing back at #829):
- The cheap isolation lever (env-var
prefix) and the blessed merge-back lever (catalog pull, which reads<namespace>/.catalog-export.jsonat the top level) don't line up — a prefix-isolated sandbox nests as<prefix>/<namespace>/…, socatalog pullcan't reach it. To get native retrieval I have to make each sandbox a real namespace, which means a.swamp.yamlper sandbox. - And even with namespace-per-sandbox, this is all working around the actual problem: the hot fan-out still can't share one namespace because of the whole-index-sync-under-lock behavior this issue describes. The sandbox scheme is essentially me partitioning the workload by hand to dodge that.
No pressure on timeline at all — I know there's a lot in flight. Mostly wanted to put the real use-case on the record: if intra-namespace writes could avoid holding the lock for the whole index (the append/partition idea in #812), the entire sandbox workaround becomes unnecessary. Happy to share lock-hold measurements or test a build against this workload whenever it's useful.
mgreten commented 7/6/2026, 11:02:15 PM
Following up with fresh measurements on @swamp/s3-datastore 2026.07.02.1 (the shard-first release), because I'm still seeing the whole-index-sync cost this issue describes — and this time it's not contention, which surprised me.
The measurement (fully quiesced, single writer)
I stopped every scheduled writer on the namespace, killed all in-flight swamp-bin processes, confirmed the lock was free, then timed two back-to-back writes of a method whose actual work is sub-second (a pr-test-planner planForPr that early-outs on a backend-only PR — no LLM call, tiny artifact):
run 1: WALL=1159.1s method=719ms status=succeeded
run 2: WALL=1251.6s method=767ms status=succeededSo ~19–21 minutes wall per write, of which <1s is the method. The remaining ~1150–1250s is pure datastore sync, paid by a single uncontended writer. Run 2 wasn't faster than run 1, so it isn't cold-cache/first-touch — it's a fixed per-write cost that tracks total namespace size, not the shard the write touches.
Namespace at the time: 123,797 catalog rows / ~95 MB, dominated by two streams — adw-analytics sessions (62,222 rows, lifetime: infinite) and an orphaned pr-feed stream (44,389 rows left behind by a model that migrated to another namespace). The write above touches only pr-test-planner (166 rows). It should not be paying for the other 123k.
Why this looks like the shard-first fast path not engaging
The 2026.07.02.1 description advertises "shard-first indexing with per-model/per-workflow partition shards under _index/ as the source of truth" and a "commitSeq-based zero-diff fast path." But:
- Two identical consecutive writes both pay the full ~20 min — a commitSeq zero-diff path should make the second nearly free.
- A write touching one 166-row model still pays for a 62k-row sibling stream — per-shard sync should skip untouched shards.
One thing I noticed while poking (offered as a lead, not a conclusion): the local cache under .swamp/data/ is a single monolithic _catalog.db — there's no local _index/ shard directory. If the partition shards are the source of truth in S3 but the local side still reconciles a monolithic catalog on every write, that would explain why the per-shard layout doesn't reduce the per-write cost. I could be misreading the layout, though.
What I've already done on my side
Following the guidance in this issue, I've kept my own write amplification down (batched telemetry, stable artifact names, cut a per-PR snapshot retention from 100×30d to 5×7d). Those help the write count. But the per-acquisition cost above is what still dominates, and it's what I think is squarely the datastore's.
Happy to share the lock-sampling script, the two-write repro harness, or run a build against this namespace whenever it's useful — this one has a nicely pathological 95 MB index to test against. And thank you again for the shard-first work; #666 and the two-phase sync have both been real improvements even as I chase this last piece.
Environment
@swamp/s3-datastore 2026.07.02.1swamp 20260706.015952.0-sha.a3f4a37c- MinIO backend, single namespace, single writer during the measurement
mgreten commented 7/7/2026, 2:18:00 AM
Splitting my two follow-ups above into standalone issues so they can be triaged on their own — they're really separate from this issue's concurrency/fan-out scope:
- #1008 —
datastore migrate-indexaborts on a hardcoded 30s S3 requestTimeout for large shards (the intended monolithic→shard-first fix can't complete at scale). - #1009 — orphaned data (from a migrated-away/deleted model) can't be GC'd or deleted.
What remains squarely on this issue: my quiesced-write measurement (two back-to-back single-writer writes, ~1159s and ~1252s wall for <1s of method work, on a 95MB/630k-entry namespace whose local index is still monolithic). That's the whole-index-sync cost this issue names, still present on 2026.07.02.1 — offered as evidence in case it's useful for deciding whether this stays closed. No pressure either way; thanks again.
mgreten commented 7/7/2026, 9:41:34 PM
One more facet from the same root cause, on the interactive-UX side — flagging because it's what actually blocked a live agent, and it's arguably separable from the throughput fix.
When the whole-index sync is slow (the ~20min-per-write cost this issue describes, on a ~95MB namespace), an interactive swamp model method run … gives the caller no backpressure: it sits on "Syncing model from datastore…" for many minutes with no progress output, no ETA, and no fail-fast. A live agent (in another repo, calling a swamp model to file a ticket) hung 7+ minutes on that line and was ultimately SIGKILLed by its own 2-minute tool timeout — so from the caller's side it's indistinguishable from a hang, and it leaves them retrying into the same wall.
Even before the throughput fix lands, two things would make this far more livable for interactive callers:
- Progress / heartbeat on the sync line (e.g. "syncing index: N/M shards" or bytes transferred) so a caller can tell it's working vs. wedged.
- A fail-fast option — a client-visible sync deadline that returns a clear "datastore busy, try later / lock contended" error instead of an open-ended hang. Today
SWAMP_DATASTORE_SYNC_TIMEOUT_MSexists but defaults high and, when it does fire, surfaces as a lock_timeout rather than actionable backpressure.
Not asking to reprioritize the core fix (#1008 migrate-index + shrinking the index is the real cure) — just noting that the hang-with-no-signal behavior is a distinct pain that a cheaper progress/deadline change could relieve in the meantime. Happy to share the caller-side transcript if useful.
mgreten commented 7/8/2026, 5:17:32 PM
Wanted to close the loop with good news and a thank-you: we migrated to shard-first v2 today (@swamp/s3-datastore 2026.07.02.1) and the scoped pull is excellent — pullIndex ~423ms on a namespace where reads used to take minutes. The remaining cost we see is on the push side (the v2 path still re-uploads the full monolithic back-compat index + catalog export each commit); we've written that up with traces and measurements in #1034 rather than piling onto this one. Thanks for landing this — it made a real difference for us.
mgreten commented 8/30/2026, 7:25:55 PM
Thank you again for the shard-first and fast-path work that shipped from this issue and its follow-ups. I am adding current-version measurements because there appears to be a remaining changed-push / global-index and remote-lock behavior under one logical repo/namespace. Please treat this as evidence and questions, not a confident root-cause claim; I may be conflating expected serialization, endpoint latency, cold-cache behavior, and a missed dirty/index fast path.
Important distinctions
1. Expected expensive cold rebuild / full verification
A separate homelab datastore rebuilt 191,438 entries across 262 shards and pushed 113,651 files. That operation needed an extended timeout. I consider that a legitimately expensive cold/full-rebuild case and am not using it as evidence that ordinary incremental pushes should be fast.
2. Zero-/no-index-change path improved
After upgrading @swamp/s3-datastore from 2026.07.09.2 to current/latest 2026.08.28.1, Moment Savor observed zero-/no-index-change operations around 1.7–2.4s. A later confirmed true no-index-change operation was about 33s. The spread may reflect endpoint or cold-cache effects, but this is materially better than the older slow path and should be considered separately from changed pushes.
3. Remaining changed-push / index and lock observations
On one logical repo/namespace, observed changed pushes include:
- 451 files: roughly 9–12 minutes
- 26 files: 7m51s
- 20 index files: 8m34s
- 9 files: 12m12s
- Importing one evidence record: roughly 8 minutes with sustained high CPU
Additional concurrency/locking observations:
- Before a consumer-side local locking repair, two collectors could both prepare the same 237 dirty files.
- One holder logged:
Lock .datastore.lock was taken over during hold; skipping delete to avoid orphaning the successor's work. - A 30s remote conditional-lock timeout occurred after a long push.
We found and repaired local host-concurrency gaps separately (consumer-local tracker #323; not Lab #323), so I do not want to attribute those overlapping collectors solely to the extension. Ordinary same-repo serialization and endpoint/cold-cache latency are also confounders.
Topology clarification: this is one repoId shared across checkouts, not a Giga-Swamp topology where each checkout has its own namespace. The measurements therefore represent same-repo/same-namespace clients.
Current read-only diagnostics
Run with current CLI 20260830.034252.0-sha.43b19693 and @swamp/s3-datastore@2026.08.28.1 (registry reports it current/latest):
swamp doctor datastores --json: overallpass; remote datastore accessible; catalog completeness check passed.swamp datastore status --json: healthy; point-in-time endpoint health latency about 34ms.swamp datastore lock status --json: no lock held at inspection time.swamp doctor extensions --json: completed; 108/112 sources healthy, with unrelated orphan-bundle findings.swamp run history --active --json: no currently active process runs.- Workflow history still contains many older records marked
runningplus one suspended record; that may be separate stale metadata and was not repaired or mutated for this report. swamp update --check: CLI is up to date.
So the current doctors/status can pass while the reported changed-push latency is still observed.
Questions for maintainers
Could you please clarify:
- What changed-push complexity and timings are expected on shard-first v2 as the namespace and shard count grow? Should a small changed set still traverse or merge a broad/global index?
- Which supported diagnostics distinguish an expected cold rebuild/full verification from a missed dirty-path, commitSeq, or index fast path?
- Do same-repo clients in separate local processes require an officially supported cross-process host lock, or is remote locking intended to be sufficient by itself?
- Is lock lease renewal/takeover during CPU-heavy index work expected, including the quoted “taken over during hold” condition?
- Should a per-record write be capable of traversing/committing hundreds of index files, or would that indicate the changed set was broadened unexpectedly?
- Which metrics or traces would be most useful next (phase timings, shard counts read/written, dirty-path cardinality before/after preparation, lock acquire/renew/hold durations, CPU profiles, request counts/bytes)?
I have not attached raw logs because they may contain private endpoints, host/user identifiers, bucket names, credentials, tokens, email, or sensitive record payloads. I can provide a narrowly sanitized trace format if you identify the preferred flags/fields.
Sign in to post a ripple.