Skip to main content
← Back to list
01Issue
FeatureOpenSwamp CLIPublic
AssigneesNone

Relationships

#4032 Datastore v2 on macOS: order data-version writes with F_BARRIERFSYNC instead of full syncs

Opened by zack · 10/27/2026

Description

Split out of swamp-club#4017 (durable v2 data-version writes). #4017 makes every v2 data-version write sync its files and directory entries before the latest marker and the catalog publish it. It uses full syncs everywhere. On macOS, Deno's FsFile.sync() is F_FULLFSYNC, a drive-cache flush, so each v2 save gets noticeably slower there. This issue brings the cost back down for the common case with the macOS ordering barrier (F_BARRIERFSYNC), as swamp-club#3737 did for v3 chunks.

Measurements (cube: M1 Max, internal APFS SSD, macOS 26.6.2)

A Deno microbenchmark of a v2 save's file sequence plus one WAL catalog commit, 200 saves per mode, p50 / p90 ms per save:

Mode Idle Busy disk
unsynced (today) 0.9-1.0 / 2.1-2.2 1.5 / 3.5
full syncs, 3 file + 4 dir 32-33 / 39 69 / 84
full syncs, reduced 3 file + 3 dir 24 / 30-34 66 / 113
reduced + F_BARRIERFSYNC 5.6-6.3 / 7.6-8.0 11 / 62

Proposed design (from #4017's plan review rounds 2-5)

  • A v2 sync policy, src/infrastructure/persistence/ordered_sync.ts, with two constructors. fullSync is FsFile.sync() plus syncDirectory. orderedSync uses barrierSync (persistence/barrier_sync.ts) on macOS and fullSync elsewhere. A volume that refuses the barrier is remembered and falls back to a full sync, never to no sync.
  • Selection is decided once in createRepositoryContext, from the datastore resolver and never from absent hooks. Use orderedSync only for the built-in filesystem datastore type, v2 engine, whose resolved data directory is the repo's own .swamp/data (localPath of data), or when there is no resolver. Only there do the catalog (catalogDbPath) and the latest marker share the content's tree and device, with nothing replicating it.
  • Everything else uses fullSync: custom-path and namespaced filesystem datastores (their catalog stays in .swamp/data, possibly on another drive), S3/GCS/extension caches (a push could ship a version before the barrier puts it on disk, as v3ChunksOrderedBefore guards), and v3.
  • The repository exposes its policy as a read-only writeSync. It is threaded beside fetchForeignContent from the three WorkflowExecutionService construction sites (workflow_run.ts, workflow_resume.ts, serve/deps.ts), ideally through one tested helper. Inside execution_service.ts it reaches the service's own repository, the DefaultStepExecutor it constructs, fromRepoDir, resolveDeps, buildDeps and the child service, typed through the already-pinned unified_data_repository import (no new PINNED_DOMAIN_INFRA_EDGES entry). The libswamp direct construction sites keep the fullSync default.
  • Optionally, a process-wide memo of directories already synced, to cut redundant directory syncs. It must store in-flight promises keyed by policy strength. A creating writer must register its parent sync before its mkdir, so a concurrent writer that finds the directory already present waits on it. A rejected sync is forgotten. It must also be able to hold parents above baseDir (findings ADV-22, 28, 29, 35, 36 on #4017).
  • The barrier's directory-entry ordering rests on the v3 precedent (fs_datastore_storage.ts, design/enablers/datastores.md, swamp-club#3737).

Acceptance

  • Unit tests for policy selection (default datastore, no resolver, custom path, namespace, eager or lazy custom cache built read-only, v3), barrier fallback, and writeSync reaching every repository the workflow service builds.
  • A save loop timed on macOS with the compiled binary, before and after.
  • design/enablers/datastores.md documents the v2 policy beside the v3 barrier section.
02Bog Flow
◉OPEN○TRIAGED○IN PROGRESS○SHIPPED

Open

10/27/2026, 6:02:26 PM

No activity in this phase yet.

03Sludge Pulse

Sign in to post a ripple.