Skip to main content
← Back to list
01Issue
BugShippedSwamp CLIPublicTeam
Assigneesstack72

Relationships

#4767 serve: its own access reload is read as a foreign change, the whole data catalog is invalidated, and the rebuild blocks the event loop

Opened by nckrtl · 11/15/2026· Shipped 11/16/2026

Engine: commit db3f6373 (swamp 20261103.000636.0-sha.db3f6373), v2 filesystem datastore (the default .swamp/ of the repository), swamp serve --auth-mode token --hot-reload.

What happens

  1. An operator changes a file under grants/ and runs swamp access reload --server ws://…. Serve reconciles the file grants itself, in handleAccessReload (src/serve/handlers/access_handlers.ts:888-904). It writes the grant records with createFileGrantStore(...).writeGrant → createResourceWriter (src/domain/access/grant_file_reconciler.ts:409-425). No ModelUpdated event is published for these writes.
  2. Within the next poll (30 s), FilesystemAccessDataChangeSource.detect() sees the changed grant records. It counts a change as its own only when a ModelUpdated event named the versions (src/serve/filesystem_access_data_change_source.ts:99-113, isOwnChange). Only method runs publish that event (src/libswamp/models/run.ts:1521). So serve's own reload counts as a peer change, and serve logs: Access data poller: 2 record(s) changed by another process, reloading policy snapshot
  3. The poller then calls catalogInvalidate() (src/serve/access_data_poller.ts:156-163). That drops the populated flag of the WHOLE data catalog (catalog_store.ts invalidate), although only grant records changed.
  4. The next data.query (any CEL data.query(...) in a scheduled workflow, or the hourly retention pass, which calls ensurePopulated) runs backfillAsync (src/domain/data/data_query_service.ts:1762). It walks every data version on disk with awaits, which is fine. Then it calls catalogStore.bulkUpsert(rows) and enforceUniqueLatest() synchronously over every row (data_query_service.ts:1840-1841, catalog_store.ts:580-645, :1690). During that call, serve answers no WebSocket ping, no scheduler tick and no request.

Impact we measured

  • Production serve, catalog 317,408 rows (426 MB): every release that changed one grant was followed, 4-5 minutes later, by 30-60 s of silence. 4 of 4 sampled silences start at a poller line and end when the waiting data.query dispatches. During them, Operation starts that need serve wait (a 15 s create took 70 s twice), and release health checks fail.
  • Reproduction below, 300,000 synthetic versions, 2 CPUs, NVMe: bulkUpsert 7.8 s plus enforceUniqueLatest 0.8 s, both synchronous. The event loop was blocked for 8.6 s in one piece. The rebuild took 81 s in all.

Asks

  1. Attribute serve's own access writes. Writes made by access.reload, the boot reconcile and the grants auto-reload are this process's writes. Publish ModelUpdated for them (as a method run does), or re-prime the change source's baseline after the reload. Then the poller does not report them as "changed by another process".
  2. Do not invalidate the data catalog for access records. Grant and group records are written through the catalog. A peer's grant change needs the policy snapshot reloaded, not every data row re-read. At most, refresh the changed access records' rows.
  3. Commit the backfill in chunks with a yield. Write the backfill rows in chunks (for example ITERATE_PAGE_SIZE = 1000) with await between chunks, and run enforceUniqueLatest the same way. Then no rebuild can block the event loop for seconds, whatever its cause (this one, config_poller at config_poller.ts:189, or boot_reconciliation.ts:138).

Reproduction (synthetic data only)

repro.ts (at the end of this report) uses Swamp's own modules, wired as src/cli/commands/serve.ts wires them. It is the same pattern as integration/serve_filesystem_access_data_poller_test.ts. It:

  1. initializes a repository in a temp folder under the working directory, and writes N synthetic command/shell data versions (20 per data name) in the on-disk layout;
  2. populates the catalog with one data.query;
  3. builds the policy loader and createAccessDataPoller (1 s interval) on the serve repository context;
  4. writes grants/repro.yaml, then reconciles it exactly as handleAccessReload does (readGrantFiles, createFileGrantStore, runInRootUnitOfWork + reconcileAllFileGrants). It then changes the key pattern in the file and reconciles again;
  5. runs one data.query, times the two synchronous catalog calls, and measures the longest event-loop stall with a 50 ms ticker.

Run: SWAMP_SRC=<swamp checkout> deno run -A --config "$SWAMP_SRC/deno.json" repro.ts 300000 (about 3 minutes, about 900k small files, 600 MB RSS).

Output on db3f6373:

generated 300000 synthetic data versions in 44961 ms
first data query populated the catalog in 55359 ms
access reload (by serve itself): 1 created, 0 revoked
[INF] swamp·serve·access-data-poller: Access data poller: 1 record(s) changed by another process, reloading policy snapshot
access reload (by serve itself): 1 created, 1 revoked
[INF] swamp·serve·access-data-poller: Access data poller: 2 record(s) changed by another process, reloading policy snapshot
after a changed grant file: invalidations=1
catalog bulkUpsert over 300003 rows: 7765 ms, synchronous
catalog enforceUniqueLatest: 840 ms, synchronous
data query after the invalidation: 81104 ms in total; longest event-loop stall 8618 ms (rows 300000)

No other process writes anything in this run. Expected: no "changed by another process" line, no invalidation, and no stall.

Data retention (garbageCollection.data) is the only way to bound the catalog. It exempts only the control-plane types, and it removes about 25 versions per second in batches of 100 (measured on the same machine; the longest stall was 124 ms, so it does not starve serve). A per-type or per-record exemption, or honouring lifetime: infinite in the age-based delete, would let a repository keep long-lived state records while it prunes per-run output.

Checked against main

The reproduction ran on db3f6373 only. On main at 537e03d18f (14 November) the same two pieces are still there by reading: the access data poller calls catalogInvalidate() after "changed by another process" (src/serve/access_data_poller.ts:212-219), and backfillAsync calls bulkUpsert and enforceUniqueLatest synchronously (src/domain/data/data_query_service.ts:1828-1829).

repro.ts

// Reproduction for the upstream issue: serve's own `access reload` is seen as "changed by another process",
// the access data poller invalidates the whole data catalog, and the next data query rebuilds it with one long
// synchronous commit that blocks the event loop.
//
// Synthetic data only. Runs against Swamp's own modules, wired as src/cli/commands/serve.ts and
// src/serve/handlers/access_handlers.ts wire them (the same pattern as
// integration/serve_filesystem_access_data_poller_test.ts).
//
// Usage (from any directory; SWAMP_SRC is a Swamp checkout):
//   deno run -A --config "$SWAMP_SRC/deno.json" repro.ts <rows> [--keep]
// Example: SWAMP_SRC=~/swamp deno run -A --config ~/swamp/deno.json repro.ts 300000
//
// Run against Swamp commit db3f6373.
const SRC = Deno.env.get("SWAMP_SRC");
if (!SRC) throw new Error("Set SWAMP_SRC to a Swamp checkout");
const rows = Number(Deno.args[0] ?? "300000");
const keep = Deno.args.includes("--keep");

const { join } = await import("jsr:@std/path@1");
await import(`${SRC}/src/domain/models/models.ts`);
const { initializeLogging } = await import(`${SRC}/src/infrastructure/logging/logger.ts`);
const { RepoService } = await import(`${SRC}/src/domain/repo/repo_service.ts`);
const { RepoPath } = await import(`${SRC}/src/domain/repo/repo_path.ts`);
const { VERSION } = await import(`${SRC}/src/cli/commands/version.ts`);
const { requireInitializedRepoUnlocked } = await import(`${SRC}/src/cli/repo_context.ts`);
const { createAccessDataPoller } = await import(`${SRC}/src/serve/access_data_poller.ts`);
const { resolverEngine } = await import(`${SRC}/src/domain/datastore/datastore_path_resolver.ts`);
const { PolicySnapshotLoader } = await import(`${SRC}/src/domain/access/policy_snapshot_loader.ts`);
const { readGrantFiles } = await import(`${SRC}/src/domain/access/grant_file.ts`);
const { validateGrantCondition } = await import(`${SRC}/src/infrastructure/cel/grant_condition_environment.ts`);
const { createFileGrantStore, reconcileAllFileGrants } = await import(
  `${SRC}/src/domain/access/grant_file_reconciler.ts`
);
const { YamlDefinitionRepository } = await import(`${SRC}/src/infrastructure/persistence/yaml_definition_repository.ts`);
const { runInRootUnitOfWork } = await import(`${SRC}/src/infrastructure/persistence/repo_unit_of_work.ts`);
const { DataQueryService } = await import(`${SRC}/src/domain/data/data_query_service.ts`);

await initializeLogging({});
const log = (line: string) => console.log(`${new Date().toISOString()} ${line}`);

// In the working directory: 300k versions are about 900k files, too many for a small /tmp.
const dir = await Deno.makeTempDir({ dir: Deno.cwd(), prefix: "swamp-catalog-stall-" });
const repoDir = join(dir, "repo");
await Deno.mkdir(repoDir, { recursive: true });
const homeDir = join(dir, "home");
await new RepoService(VERSION, { homeDir, configDir: join(homeDir, ".config", "swamp") })
  .init(RepoPath.create(repoDir), { tools: [] });

// 1. Synthetic model data on disk: `rows` versions, 20 per data name, in the on-disk layout serve writes.
const t0 = performance.now();
const enc = new TextEncoder();
const perName = 20;
const created = Date.parse("2026-11-01T00:00:00Z");
for (let n = 0; n < rows / perName; n++) {
  const modelId = crypto.randomUUID();
  const nameDir = join(repoDir, ".swamp", "data", "command", "shell", modelId, "result");
  for (let v = 1; v <= perName; v++) {
    const vDir = join(nameDir, String(v));
    await Deno.mkdir(vDir, { recursive: true });
    const raw = enc.encode(JSON.stringify({ stdout: `synthetic ${n}/${v}`, exitCode: 0 }));
    const sum = Array.from(new Uint8Array(await crypto.subtle.digest("SHA-256", raw)))
      .map((b) => b.toString(16).padStart(2, "0")).join("");
    await Deno.writeFile(join(vDir, "raw"), raw);
    await Deno.writeTextFile(
      join(vDir, "metadata.yaml"),
      `id: ${crypto.randomUUID()}\nname: result\nversion: ${v}\ncontentType: application/json\nlifetime: infinite\n` +
        `garbageCollection: 50\nstreaming: false\ntags:\n  type: resource\n  modelName: synthetic-${n}\n` +
        `ownerDefinition:\n  ownerType: model-method\n  ownerRef: command/shell:${modelId}\n` +
        `createdAt: '${new Date(created + n * perName + v).toISOString()}'\nsize: ${raw.length}\nchecksum: ${sum}\n`,
    );
  }
  await Deno.writeTextFile(join(nameDir, "latest"), String(perName));
}
log(`generated ${rows} synthetic data versions in ${Math.round(performance.now() - t0)} ms`);

// 2. Serve's side, as serve.ts builds it: a populated catalog, the policy loader, the access data poller.
const serve = await requireInitializedRepoUnlocked({ repoDir, outputMode: "json" });
const query = new DataQueryService(serve.repoContext.catalogStore, serve.repoContext.unifiedDataRepo);
let t = performance.now();
await query.query('modelName == "synthetic-0"');
log(`first data query populated the catalog in ${Math.round(performance.now() - t)} ms`);

let invalidations = 0;
const loader = new PolicySnapshotLoader(
  serve.repoContext.unifiedDataRepo,
  serve.repoContext.eventBus,
  "manual",
  { runImpliesApprove: true, runImpliesSignal: true },
);
const poller = await createAccessDataPoller({
  syncService: serve.syncService,
  pollIntervalMs: 1000,
  policySnapshotLoader: loader,
  catalogInvalidate: () => {
    invalidations++;
    serve.repoContext.catalogStore.invalidate();
  },
  datastoreConfig: serve.datastoreConfig,
  datastoreEngine: resolverEngine(serve.datastoreResolver),
  datastoreResolver: serve.datastoreResolver,
  eventBus: serve.repoContext.eventBus,
});
if (!poller) throw new Error("expected a filesystem access data poller");
await loader.load();
poller.start();

// 3. `swamp access reload --server`: serve itself reconciles the grants directory, exactly as
//    handleAccessReload does (src/serve/handlers/access_handlers.ts: readGrantFiles, createFileGrantStore,
//    runInRootUnitOfWork + reconcileAllFileGrants). No other process writes anything.
async function reloadAsServe(): Promise<void> {
  const files = await readGrantFiles(join(repoDir, "grants"), validateGrantCondition);
  const entries = new Map();
  for (const [filename, result] of files) {
    if (result.errors.length) throw new Error(JSON.stringify(result.errors));
    entries.set(filename, result.entries);
  }
  const autoDefRepo = new YamlDefinitionRepository(
    repoDir,
    serve.repoContext.eventBus,
    serve.repoContext.autoDefinitionsDir,
    false,
    serve.repoContext.markDirty,
  );
  const store = createFileGrantStore(
    serve.repoContext.definitionRepo,
    autoDefRepo,
    serve.repoContext.unifiedDataRepo,
  );
  const result = await runInRootUnitOfWork(
    serve.repoContext,
    { flush: undefined },
    () => reconcileAllFileGrants(entries, store),
  );
  await loader.load();
  log(`access reload (by serve itself): ${result.totalCreated} created, ${result.totalRevoked} revoked`);
}

const grantFile = (pattern: string) =>
  `grants:\n  - subject: "user:repro-gateway"\n    effect: allow\n    actions: [read]\n` +
  `    resource: "vault:repro-inputs"\n    condition: 'key.matches("${pattern}")'\n`;
await Deno.writeTextFile(join(repoDir, "grants", "repro.yaml"), grantFile("^a-"));
await reloadAsServe();
await new Promise((r) => setTimeout(r, 3000));
log(`after the first grant file: invalidations=${invalidations}`);
invalidations = 0;
// The release case: the grant file changes (a wider key pattern), and serve reloads it.
await Deno.writeTextFile(join(repoDir, "grants", "repro.yaml"), grantFile("^(a|b)-"));
await reloadAsServe();
await new Promise((r) => setTimeout(r, 3000));
log(`after a changed grant file: invalidations=${invalidations} (any value above 0 is the defect: serve wrote it)`);
await poller.stop();

// 4. The next data query rebuilds the catalog. Measure how long the event loop is blocked, and time the two
//    synchronous catalog calls that end the rebuild (data_query_service.ts backfillAsync).
const store = serve.repoContext.catalogStore as unknown as Record<string, (...a: unknown[]) => unknown>;
for (const method of ["bulkUpsert", "enforceUniqueLatest"]) {
  const original = store[method].bind(store);
  store[method] = (...args: unknown[]) => {
    const started = performance.now();
    try {
      return original(...args);
    } finally {
      const rowsArg = Array.isArray(args[0]) ? ` over ${args[0].length} rows` : "";
      log(`catalog ${method}${rowsArg}: ${Math.round(performance.now() - started)} ms, synchronous`);
    }
  };
}
let maxGap = 0;
let last = performance.now();
const ticker = setInterval(() => {
  const now = performance.now();
  maxGap = Math.max(maxGap, now - last - 50);
  last = now;
}, 50);
t = performance.now();
await query.query('modelName == "synthetic-0"');
const total = performance.now() - t;
// Let the ticker's late callback run: the stall shows as the gap it reports.
await new Promise((r) => setTimeout(r, 200));
clearInterval(ticker);
log(
  `data query after the invalidation: ${Math.round(total)} ms in total; longest event-loop stall ${
    Math.round(maxGap)
  } ms (rows ${rows}); RSS ${Math.round(Deno.memoryUsage().rss / 1e6)} MB`,
);

await loader.dispose();
serve.repoContext.catalogStore.close();
if (!keep) await Deno.remove(dir, { recursive: true });
else log(`kept ${dir}`);
02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 5 MOREREVIEW+ 12 MOREPR_LINKED+ 2 MORESESSION_SUMMARIZED

Shipped

11/16/2026, 5:57:29 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack7211/16/2026, 1:51:30 PM
Editable. Press Enter to edit.

nckrtl commented 11/15/2026, 7:53:01 AM

Reproduced on main as well: 537e03d18f (2026-11-14), same machine, same script. One line of repro.ts must change for main, because VERSION moved: import it from src/infrastructure/runtime/build_info.ts instead of src/cli/commands/version.ts.

Output on 537e03d18f:

generated 300000 synthetic data versions in 74296 ms
first data query populated the catalog in 58110 ms
access reload (by serve itself): 1 created, 0 revoked
[INF] swamp·serve·access-data-poller: Access data poller: 1 record(s) changed by another process, reloading the "policy snapshot"
access reload (by serve itself): 1 created, 1 revoked
[INF] swamp·serve·access-data-poller: Access data poller: 2 record(s) changed by another process, reloading the "policy snapshot"
after a changed grant file: invalidations=1
catalog bulkUpsert over 300003 rows: 10540 ms, synchronous
catalog enforceUniqueLatest: 1871 ms, synchronous
data query after the invalidation: 50701 ms in total; longest event-loop stall 12423 ms (rows 300000); RSS 577 MB

(The machine was busier than in the first run, so the absolute times are higher.)

stack72 commented 11/16/2026, 5:57:31 PM

Thanks @nckrtl for reporting this! We shipped: Stop serve's own grant and delegation writes from reading as peer changes, refresh only the access-family catalog rows on a real peer change instead of invalidating the whole data catalog, and commit the full catalog backfill in name-aligned chunks with yields, single-flight per catalog store, so no rebuild blocks the event loop for seconds or runs more than once at a time.. The fix has been merged and a release is on its way. We appreciate your contribution to swamp.

Sign in to post a ripple.