← Back to list
01Issue
FeatureOpenSwamp CLIPublic
AssigneesNone
Relationships
#4032 Datastore v2 on macOS: order data-version writes with F_BARRIERFSYNC instead of full syncs
Opened by zack · 10/27/2026
Description
Split out of swamp-club#4017 (durable v2 data-version writes). #4017 makes every v2 data-version write sync its files and directory entries before the latest marker and the catalog publish it. It uses full syncs everywhere. On macOS, Deno's FsFile.sync() is F_FULLFSYNC, a drive-cache flush, so each v2 save gets noticeably slower there. This issue brings the cost back down for the common case with the macOS ordering barrier (F_BARRIERFSYNC), as swamp-club#3737 did for v3 chunks.
Measurements (cube: M1 Max, internal APFS SSD, macOS 26.6.2)
A Deno microbenchmark of a v2 save's file sequence plus one WAL catalog commit, 200 saves per mode, p50 / p90 ms per save:
| Mode | Idle | Busy disk |
|---|---|---|
| unsynced (today) | 0.9-1.0 / 2.1-2.2 | 1.5 / 3.5 |
| full syncs, 3 file + 4 dir | 32-33 / 39 | 69 / 84 |
| full syncs, reduced 3 file + 3 dir | 24 / 30-34 | 66 / 113 |
| reduced + F_BARRIERFSYNC | 5.6-6.3 / 7.6-8.0 | 11 / 62 |
Proposed design (from #4017's plan review rounds 2-5)
- A v2 sync policy, src/infrastructure/persistence/ordered_sync.ts, with two constructors. fullSync is FsFile.sync() plus syncDirectory. orderedSync uses barrierSync (persistence/barrier_sync.ts) on macOS and fullSync elsewhere. A volume that refuses the barrier is remembered and falls back to a full sync, never to no sync.
- Selection is decided once in createRepositoryContext, from the datastore resolver and never from absent hooks. Use orderedSync only for the built-in filesystem datastore type, v2 engine, whose resolved data directory is the repo's own .swamp/data (localPath of data), or when there is no resolver. Only there do the catalog (catalogDbPath) and the latest marker share the content's tree and device, with nothing replicating it.
- Everything else uses fullSync: custom-path and namespaced filesystem datastores (their catalog stays in .swamp/data, possibly on another drive), S3/GCS/extension caches (a push could ship a version before the barrier puts it on disk, as v3ChunksOrderedBefore guards), and v3.
- The repository exposes its policy as a read-only writeSync. It is threaded beside fetchForeignContent from the three WorkflowExecutionService construction sites (workflow_run.ts, workflow_resume.ts, serve/deps.ts), ideally through one tested helper. Inside execution_service.ts it reaches the service's own repository, the DefaultStepExecutor it constructs, fromRepoDir, resolveDeps, buildDeps and the child service, typed through the already-pinned unified_data_repository import (no new PINNED_DOMAIN_INFRA_EDGES entry). The libswamp direct construction sites keep the fullSync default.
- Optionally, a process-wide memo of directories already synced, to cut redundant directory syncs. It must store in-flight promises keyed by policy strength. A creating writer must register its parent sync before its mkdir, so a concurrent writer that finds the directory already present waits on it. A rejected sync is forgotten. It must also be able to hold parents above baseDir (findings ADV-22, 28, 29, 35, 36 on #4017).
- The barrier's directory-entry ordering rests on the v3 precedent (fs_datastore_storage.ts, design/enablers/datastores.md, swamp-club#3737).
Acceptance
- Unit tests for policy selection (default datastore, no resolver, custom path, namespace, eager or lazy custom cache built read-only, v3), barrier fallback, and writeSync reaching every repository the workflow service builds.
- A save loop timed on macOS with the compiled binary, before and after.
- design/enablers/datastores.md documents the v2 policy beside the v3 barrier section.
02Bog Flow
Open
No activity in this phase yet.
03Sludge Pulse
Sign in to post a ripple.