Skip to main content
← Back to list
01Issue
BugShippedSwamp CLIPublic
Assigneesstack72

Relationships

#1932 Catalog export blocks event loop and global lock during serve push

Opened by stack72 · 9/1/2026· Shipped 9/1/2026

Problem

writeCatalogExportIfNeeded runs on every model method and workflow step flush in serve. It has two blocking dimensions that degrade datastore access for all concurrent users:

1. Synchronous SQLite reads block the Deno event loop

writeCatalogExport (repository_factory.ts:155-167) calls [...catalogStore.iterateNamespace(namespace)] which uses DatabaseSync (node:sqlite synchronous API) to page through every catalog row for the namespace, then JSON.stringifys the full array. While this runs, the entire event loop is frozen — no HTTP handlers, WebSocket messages, or workflow steps can make progress.

The impact scales linearly with catalog size. A namespace with thousands of data items blocks the event loop for potentially hundreds of milliseconds per flush.

2. Single-phase push holds the global distributed lock during export

In flushSinglePhasePush (repo_context.ts:1410-1485), the sequence is:

  1. Acquire global lock
  2. writeCatalogExportIfNeeded — blocking read + serialize + file write
  3. pushChanged — network upload
  4. Release global lock

Any other model completing concurrently blocks waiting for the global lock during its own flush. Two-phase push already moves the export outside the lock (repo_context.ts:1517), but single-phase does not.

User impact

  • Data queries (data.get, data.query, data.list) stall during export
  • Workflow step latency compounds across multi-step workflows
  • WebSocket responsiveness drops
  • Concurrent model runs serialize behind the global lock (single-phase)
  • Gets progressively worse as users accumulate more data

Proposed fixes

  1. Move export before global lock in single-phase push — match what two-phase already does
  2. Move SQLite read + JSON serialization to a worker thread — unblock the event loop
  3. Stream JSON instead of materializing all rows then JSON.stringify
  4. Incremental/delta exports instead of full namespace dump on every push
02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 2 MORETRIAGE+ 7 MOREREVIEW+ 8 MOREPR_MERGED+ 2 MORESESSION_SUMMARIZED

Shipped

9/1/2026, 6:21:20 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack729/1/2026, 4:56:13 PM

Sign in to post a ripple.