Relationships
#4162 Filesystem control-plane put races empty-parent cleanup (cron pending-run ENOENT)
Opened by nckrtl · 10/30/2026
Filesystem control-plane put races delete's empty-parent cleanup, causing cron dual-write ENOENT.
Observed with 20261028.205235.0-sha.1602a617 on Linux/systemd, local filesystem datastore. Serve intermittently logs Control-plane dual-write failed for cron pending run ... No such file or directory: writefile .swamp/_control/pending-runs/.<uuid>.tmp while scheduling and claiming concurrently.
FileSystemControlPlaneStore.put() does mkdir(dirname(path), {recursive:true}) then atomicWriteFile(). A concurrent delete() of the directory's last record calls cleanupEmptyParentDirs() between those two operations. The atomic temporary-file write then fails with NotFound. This reproduces without retention or GC. Recreating the directory is temporary because normal record deletion removes it again. putIfAbsent() already handles a similar NotFound window when staging, but put() does not.
Deterministic minimal test, run in the Swamp source checkout:
import { assertRejects } from "@std/assert";
import { FileSystemControlPlaneStore } from "./src/infrastructure/persistence/fs_control_plane_store.ts";
Deno.test("control put races cleanup of an empty parent", async () => {
const root = await Deno.makeTempDir();
const store = new FileSystemControlPlaneStore(root);
const mkdir = Deno.mkdir;
try {
await store.put("pending-runs/old", new TextEncoder().encode("old"));
let once = true;
Deno.mkdir = async (...args: Parameters<typeof Deno.mkdir>) => {
await mkdir(...args);
if (once && String(args[0]).endsWith("/pending-runs")) {
once = false;
await store.delete("pending-runs/old");
}
};
// Current behavior: NotFound. Expected after a fix: put succeeds and
// get("pending-runs/new") returns the complete new bytes.
await assertRejects(
() => store.put("pending-runs/new", new TextEncoder().encode("new")),
Deno.errors.NotFound,
);
} finally {
Deno.mkdir = mkdir;
await Deno.remove(root, { recursive: true });
}
});Please make atomic control-plane puts safe against concurrent empty-parent cleanup, and add this interleaving as a regression. No production records or credentials are needed for the reproduction.
Open
No activity in this phase yet.
Sign in to post a ripple.