Relationships
#4767 serve: its own access reload is read as a foreign change, the whole data catalog is invalidated, and the rebuild blocks the event loop
Opened by nckrtl · 11/15/2026· Shipped 11/16/2026
Engine: commit db3f6373 (swamp 20261103.000636.0-sha.db3f6373), v2 filesystem datastore (the default .swamp/ of the repository), swamp serve --auth-mode token --hot-reload.
What happens
- An operator changes a file under
grants/and runsswamp access reload --server ws://…. Serve reconciles the file grants itself, inhandleAccessReload(src/serve/handlers/access_handlers.ts:888-904). It writes the grant records withcreateFileGrantStore(...).writeGrant→createResourceWriter(src/domain/access/grant_file_reconciler.ts:409-425). NoModelUpdatedevent is published for these writes. - Within the next poll (30 s),
FilesystemAccessDataChangeSource.detect()sees the changed grant records. It counts a change as its own only when aModelUpdatedevent named the versions (src/serve/filesystem_access_data_change_source.ts:99-113,isOwnChange). Only method runs publish that event (src/libswamp/models/run.ts:1521). So serve's own reload counts as a peer change, and serve logs:Access data poller: 2 record(s) changed by another process, reloading policy snapshot - The poller then calls
catalogInvalidate()(src/serve/access_data_poller.ts:156-163). That drops thepopulatedflag of the WHOLE data catalog (catalog_store.tsinvalidate), although only grant records changed. - The next
data.query(any CELdata.query(...)in a scheduled workflow, or the hourly retention pass, which callsensurePopulated) runsbackfillAsync(src/domain/data/data_query_service.ts:1762). It walks every data version on disk with awaits, which is fine. Then it callscatalogStore.bulkUpsert(rows)andenforceUniqueLatest()synchronously over every row (data_query_service.ts:1840-1841,catalog_store.ts:580-645,:1690). During that call, serve answers no WebSocket ping, no scheduler tick and no request.
Impact we measured
- Production serve, catalog 317,408 rows (426 MB): every release that changed one grant was followed, 4-5 minutes later, by 30-60 s of silence. 4 of 4 sampled silences start at a poller line and end when the waiting
data.querydispatches. During them, Operation starts that need serve wait (a 15 s create took 70 s twice), and release health checks fail. - Reproduction below, 300,000 synthetic versions, 2 CPUs, NVMe:
bulkUpsert7.8 s plusenforceUniqueLatest0.8 s, both synchronous. The event loop was blocked for 8.6 s in one piece. The rebuild took 81 s in all.
Asks
- Attribute serve's own access writes. Writes made by
access.reload, the boot reconcile and the grants auto-reload are this process's writes. PublishModelUpdatedfor them (as a method run does), or re-prime the change source's baseline after the reload. Then the poller does not report them as "changed by another process". - Do not invalidate the data catalog for access records. Grant and group records are written through the catalog. A peer's grant change needs the policy snapshot reloaded, not every data row re-read. At most, refresh the changed access records' rows.
- Commit the backfill in chunks with a yield. Write the backfill rows in chunks (for example
ITERATE_PAGE_SIZE= 1000) withawaitbetween chunks, and runenforceUniqueLatestthe same way. Then no rebuild can block the event loop for seconds, whatever its cause (this one,config_polleratconfig_poller.ts:189, orboot_reconciliation.ts:138).
Reproduction (synthetic data only)
repro.ts (at the end of this report) uses Swamp's own modules, wired as src/cli/commands/serve.ts wires them. It is the same pattern as integration/serve_filesystem_access_data_poller_test.ts. It:
- initializes a repository in a temp folder under the working directory, and writes N synthetic
command/shelldata versions (20 per data name) in the on-disk layout; - populates the catalog with one
data.query; - builds the policy loader and
createAccessDataPoller(1 s interval) on the serve repository context; - writes
grants/repro.yaml, then reconciles it exactly ashandleAccessReloaddoes (readGrantFiles,createFileGrantStore,runInRootUnitOfWork+reconcileAllFileGrants). It then changes the key pattern in the file and reconciles again; - runs one
data.query, times the two synchronous catalog calls, and measures the longest event-loop stall with a 50 ms ticker.
Run: SWAMP_SRC=<swamp checkout> deno run -A --config "$SWAMP_SRC/deno.json" repro.ts 300000 (about 3 minutes, about 900k small files, 600 MB RSS).
Output on db3f6373:
generated 300000 synthetic data versions in 44961 ms
first data query populated the catalog in 55359 ms
access reload (by serve itself): 1 created, 0 revoked
[INF] swamp·serve·access-data-poller: Access data poller: 1 record(s) changed by another process, reloading policy snapshot
access reload (by serve itself): 1 created, 1 revoked
[INF] swamp·serve·access-data-poller: Access data poller: 2 record(s) changed by another process, reloading policy snapshot
after a changed grant file: invalidations=1
catalog bulkUpsert over 300003 rows: 7765 ms, synchronous
catalog enforceUniqueLatest: 840 ms, synchronous
data query after the invalidation: 81104 ms in total; longest event-loop stall 8618 ms (rows 300000)No other process writes anything in this run. Expected: no "changed by another process" line, no invalidation, and no stall.
Related, for context
Data retention (garbageCollection.data) is the only way to bound the catalog. It exempts only the control-plane types, and it removes about 25 versions per second in batches of 100 (measured on the same machine; the longest stall was 124 ms, so it does not starve serve). A per-type or per-record exemption, or honouring lifetime: infinite in the age-based delete, would let a repository keep long-lived state records while it prunes per-run output.
Checked against main
The reproduction ran on db3f6373 only. On main at 537e03d18f (14 November) the same two pieces are still there by reading: the access data poller calls catalogInvalidate() after "changed by another process" (src/serve/access_data_poller.ts:212-219), and backfillAsync calls bulkUpsert and enforceUniqueLatest synchronously (src/domain/data/data_query_service.ts:1828-1829).
repro.ts
// Reproduction for the upstream issue: serve's own `access reload` is seen as "changed by another process",
// the access data poller invalidates the whole data catalog, and the next data query rebuilds it with one long
// synchronous commit that blocks the event loop.
//
// Synthetic data only. Runs against Swamp's own modules, wired as src/cli/commands/serve.ts and
// src/serve/handlers/access_handlers.ts wire them (the same pattern as
// integration/serve_filesystem_access_data_poller_test.ts).
//
// Usage (from any directory; SWAMP_SRC is a Swamp checkout):
// deno run -A --config "$SWAMP_SRC/deno.json" repro.ts <rows> [--keep]
// Example: SWAMP_SRC=~/swamp deno run -A --config ~/swamp/deno.json repro.ts 300000
//
// Run against Swamp commit db3f6373.
const SRC = Deno.env.get("SWAMP_SRC");
if (!SRC) throw new Error("Set SWAMP_SRC to a Swamp checkout");
const rows = Number(Deno.args[0] ?? "300000");
const keep = Deno.args.includes("--keep");
const { join } = await import("jsr:@std/path@1");
await import(`${SRC}/src/domain/models/models.ts`);
const { initializeLogging } = await import(`${SRC}/src/infrastructure/logging/logger.ts`);
const { RepoService } = await import(`${SRC}/src/domain/repo/repo_service.ts`);
const { RepoPath } = await import(`${SRC}/src/domain/repo/repo_path.ts`);
const { VERSION } = await import(`${SRC}/src/cli/commands/version.ts`);
const { requireInitializedRepoUnlocked } = await import(`${SRC}/src/cli/repo_context.ts`);
const { createAccessDataPoller } = await import(`${SRC}/src/serve/access_data_poller.ts`);
const { resolverEngine } = await import(`${SRC}/src/domain/datastore/datastore_path_resolver.ts`);
const { PolicySnapshotLoader } = await import(`${SRC}/src/domain/access/policy_snapshot_loader.ts`);
const { readGrantFiles } = await import(`${SRC}/src/domain/access/grant_file.ts`);
const { validateGrantCondition } = await import(`${SRC}/src/infrastructure/cel/grant_condition_environment.ts`);
const { createFileGrantStore, reconcileAllFileGrants } = await import(
`${SRC}/src/domain/access/grant_file_reconciler.ts`
);
const { YamlDefinitionRepository } = await import(`${SRC}/src/infrastructure/persistence/yaml_definition_repository.ts`);
const { runInRootUnitOfWork } = await import(`${SRC}/src/infrastructure/persistence/repo_unit_of_work.ts`);
const { DataQueryService } = await import(`${SRC}/src/domain/data/data_query_service.ts`);
await initializeLogging({});
const log = (line: string) => console.log(`${new Date().toISOString()} ${line}`);
// In the working directory: 300k versions are about 900k files, too many for a small /tmp.
const dir = await Deno.makeTempDir({ dir: Deno.cwd(), prefix: "swamp-catalog-stall-" });
const repoDir = join(dir, "repo");
await Deno.mkdir(repoDir, { recursive: true });
const homeDir = join(dir, "home");
await new RepoService(VERSION, { homeDir, configDir: join(homeDir, ".config", "swamp") })
.init(RepoPath.create(repoDir), { tools: [] });
// 1. Synthetic model data on disk: `rows` versions, 20 per data name, in the on-disk layout serve writes.
const t0 = performance.now();
const enc = new TextEncoder();
const perName = 20;
const created = Date.parse("2026-11-01T00:00:00Z");
for (let n = 0; n < rows / perName; n++) {
const modelId = crypto.randomUUID();
const nameDir = join(repoDir, ".swamp", "data", "command", "shell", modelId, "result");
for (let v = 1; v <= perName; v++) {
const vDir = join(nameDir, String(v));
await Deno.mkdir(vDir, { recursive: true });
const raw = enc.encode(JSON.stringify({ stdout: `synthetic ${n}/${v}`, exitCode: 0 }));
const sum = Array.from(new Uint8Array(await crypto.subtle.digest("SHA-256", raw)))
.map((b) => b.toString(16).padStart(2, "0")).join("");
await Deno.writeFile(join(vDir, "raw"), raw);
await Deno.writeTextFile(
join(vDir, "metadata.yaml"),
`id: ${crypto.randomUUID()}\nname: result\nversion: ${v}\ncontentType: application/json\nlifetime: infinite\n` +
`garbageCollection: 50\nstreaming: false\ntags:\n type: resource\n modelName: synthetic-${n}\n` +
`ownerDefinition:\n ownerType: model-method\n ownerRef: command/shell:${modelId}\n` +
`createdAt: '${new Date(created + n * perName + v).toISOString()}'\nsize: ${raw.length}\nchecksum: ${sum}\n`,
);
}
await Deno.writeTextFile(join(nameDir, "latest"), String(perName));
}
log(`generated ${rows} synthetic data versions in ${Math.round(performance.now() - t0)} ms`);
// 2. Serve's side, as serve.ts builds it: a populated catalog, the policy loader, the access data poller.
const serve = await requireInitializedRepoUnlocked({ repoDir, outputMode: "json" });
const query = new DataQueryService(serve.repoContext.catalogStore, serve.repoContext.unifiedDataRepo);
let t = performance.now();
await query.query('modelName == "synthetic-0"');
log(`first data query populated the catalog in ${Math.round(performance.now() - t)} ms`);
let invalidations = 0;
const loader = new PolicySnapshotLoader(
serve.repoContext.unifiedDataRepo,
serve.repoContext.eventBus,
"manual",
{ runImpliesApprove: true, runImpliesSignal: true },
);
const poller = await createAccessDataPoller({
syncService: serve.syncService,
pollIntervalMs: 1000,
policySnapshotLoader: loader,
catalogInvalidate: () => {
invalidations++;
serve.repoContext.catalogStore.invalidate();
},
datastoreConfig: serve.datastoreConfig,
datastoreEngine: resolverEngine(serve.datastoreResolver),
datastoreResolver: serve.datastoreResolver,
eventBus: serve.repoContext.eventBus,
});
if (!poller) throw new Error("expected a filesystem access data poller");
await loader.load();
poller.start();
// 3. `swamp access reload --server`: serve itself reconciles the grants directory, exactly as
// handleAccessReload does (src/serve/handlers/access_handlers.ts: readGrantFiles, createFileGrantStore,
// runInRootUnitOfWork + reconcileAllFileGrants). No other process writes anything.
async function reloadAsServe(): Promise<void> {
const files = await readGrantFiles(join(repoDir, "grants"), validateGrantCondition);
const entries = new Map();
for (const [filename, result] of files) {
if (result.errors.length) throw new Error(JSON.stringify(result.errors));
entries.set(filename, result.entries);
}
const autoDefRepo = new YamlDefinitionRepository(
repoDir,
serve.repoContext.eventBus,
serve.repoContext.autoDefinitionsDir,
false,
serve.repoContext.markDirty,
);
const store = createFileGrantStore(
serve.repoContext.definitionRepo,
autoDefRepo,
serve.repoContext.unifiedDataRepo,
);
const result = await runInRootUnitOfWork(
serve.repoContext,
{ flush: undefined },
() => reconcileAllFileGrants(entries, store),
);
await loader.load();
log(`access reload (by serve itself): ${result.totalCreated} created, ${result.totalRevoked} revoked`);
}
const grantFile = (pattern: string) =>
`grants:\n - subject: "user:repro-gateway"\n effect: allow\n actions: [read]\n` +
` resource: "vault:repro-inputs"\n condition: 'key.matches("${pattern}")'\n`;
await Deno.writeTextFile(join(repoDir, "grants", "repro.yaml"), grantFile("^a-"));
await reloadAsServe();
await new Promise((r) => setTimeout(r, 3000));
log(`after the first grant file: invalidations=${invalidations}`);
invalidations = 0;
// The release case: the grant file changes (a wider key pattern), and serve reloads it.
await Deno.writeTextFile(join(repoDir, "grants", "repro.yaml"), grantFile("^(a|b)-"));
await reloadAsServe();
await new Promise((r) => setTimeout(r, 3000));
log(`after a changed grant file: invalidations=${invalidations} (any value above 0 is the defect: serve wrote it)`);
await poller.stop();
// 4. The next data query rebuilds the catalog. Measure how long the event loop is blocked, and time the two
// synchronous catalog calls that end the rebuild (data_query_service.ts backfillAsync).
const store = serve.repoContext.catalogStore as unknown as Record<string, (...a: unknown[]) => unknown>;
for (const method of ["bulkUpsert", "enforceUniqueLatest"]) {
const original = store[method].bind(store);
store[method] = (...args: unknown[]) => {
const started = performance.now();
try {
return original(...args);
} finally {
const rowsArg = Array.isArray(args[0]) ? ` over ${args[0].length} rows` : "";
log(`catalog ${method}${rowsArg}: ${Math.round(performance.now() - started)} ms, synchronous`);
}
};
}
let maxGap = 0;
let last = performance.now();
const ticker = setInterval(() => {
const now = performance.now();
maxGap = Math.max(maxGap, now - last - 50);
last = now;
}, 50);
t = performance.now();
await query.query('modelName == "synthetic-0"');
const total = performance.now() - t;
// Let the ticker's late callback run: the stall shows as the gap it reports.
await new Promise((r) => setTimeout(r, 200));
clearInterval(ticker);
log(
`data query after the invalidation: ${Math.round(total)} ms in total; longest event-loop stall ${
Math.round(maxGap)
} ms (rows ${rows}); RSS ${Math.round(Deno.memoryUsage().rss / 1e6)} MB`,
);
await loader.dispose();
serve.repoContext.catalogStore.close();
if (!keep) await Deno.remove(dir, { recursive: true });
else log(`kept ${dir}`);Shipped
Click a lifecycle step above to view its details.
nckrtl commented 11/15/2026, 7:53:01 AM
Reproduced on main as well: 537e03d18f (2026-11-14), same machine, same script. One line of repro.ts must change for main, because VERSION moved: import it from src/infrastructure/runtime/build_info.ts instead of src/cli/commands/version.ts.
Output on 537e03d18f:
generated 300000 synthetic data versions in 74296 ms
first data query populated the catalog in 58110 ms
access reload (by serve itself): 1 created, 0 revoked
[INF] swamp·serve·access-data-poller: Access data poller: 1 record(s) changed by another process, reloading the "policy snapshot"
access reload (by serve itself): 1 created, 1 revoked
[INF] swamp·serve·access-data-poller: Access data poller: 2 record(s) changed by another process, reloading the "policy snapshot"
after a changed grant file: invalidations=1
catalog bulkUpsert over 300003 rows: 10540 ms, synchronous
catalog enforceUniqueLatest: 1871 ms, synchronous
data query after the invalidation: 50701 ms in total; longest event-loop stall 12423 ms (rows 300000); RSS 577 MB(The machine was busier than in the first run, so the absolute times are higher.)
stack72 commented 11/16/2026, 5:57:31 PM
Thanks @nckrtl for reporting this! We shipped: Stop serve's own grant and delegation writes from reading as peer changes, refresh only the access-family catalog rows on a real peer change instead of invalidating the whole data catalog, and commit the full catalog backfill in name-aligned chunks with yields, single-flight per catalog store, so no rebuild blocks the event loop for seconds or runs more than once at a time.. The fix has been merged and a release is on its way. We appreciate your contribution to swamp.
Sign in to post a ripple.