Relationships
#4723 v3 targeted recovery leaves a different data/<model>/<name>/latest than a full rebuild after a rename onto a deleted name with an interrupted writer (deterministic property-test seed)
Opened by stack72 · 11/13/2026· Shipped 11/13/2026
Related: #4712 (wall-clock flakes in the same recovery tests; a different failure), #4702, #4677, #4581 (the targeted-recovery work this test covers).
What failed
- Test:
integration/datastore_v3_targeted_recovery_property_test.ts, the case "targeted recovery: leaves the catalog, renames, files and live records a full rebuild leaves, whatever the history and whatever an interrupted writer or dropped work left (swamp-club#4581)". - Where it was seen: a verify-build run on branch
4693(swamp-club#4693, serve auto-continuation), commit0549ab2f, 2026-11-13 15:15 UTC. - fast-check output:
Property failed after 146 tests,{ seed: 206505560, path: "145:2:2:1:3:6:9:9:8:8:8:8:8:8:8:10:10:10:10:10:10:11:10:10:14:14", endOnFailure: true }.
It is not caused by the 4693 branch
That branch touches nothing under src/infrastructure/datastore_v3/, src/domain/datastore/ or this test. With the seed pinned, the test fails the same way on three clean checkouts:
| Checkout | Result |
|---|---|
529118c0a, the 4693 merge base (none of its commits) |
Property failed after 1 tests |
0549ab2f, the 4693 head |
Property failed after 1 tests |
origin/main at 150772096, which includes #4677 3fc69ff93 |
Property failed after 1 tests |
It fails in about 100 ms on an idle machine, so load and timing are not involved.
Why it has not shown up elsewhere
The test runs 150 random histories per run, with a fresh seed each time. Most seeds never generate this history; this build happened to draw one that does. A plain rerun draws a new seed and will very likely pass, but the bug stays.
Repro
In integration/datastore_v3_targeted_recovery_property_test.ts, change the fc.assert options ({ numRuns: 150 }, near line 768) to:
{ numRuns: 150, seed: 206505560, path: "145:2:2:1:3:6:9:9:8:8:8:8:8:8:8:10:10:10:10:10:10:11:10:10:14:14", endOnFailure: true }Then run deno test -A integration/datastore_v3_targeted_recovery_property_test.ts --filter "whatever the history".
Counterexample (shrunk)
[[{"model":0,"kind":"rename","name":"c","to":"a","pick":0},
{"model":0,"kind":"delete","name":"c","to":"a","pick":0},
{"model":0,"kind":"append","name":"c","to":"a","pick":0},
{"model":0,"kind":"save","name":"a","to":"a","pick":0},
{"model":0,"kind":"save","name":"a","to":"a","pick":0},
{"model":0,"kind":"rename","name":"c","to":"a","pick":0}],
0,
[{"model":0,"kind":"repo-save","name":"a","pick":0}],
{"pending":[{"model":0,"kind":"repo-save","name":"a","pick":0},
{"model":0,"kind":"repo-delete","name":"c","pick":1}],"lands":false},
"interrupted_writer",
undefined]The history in plain words: rename c→a, delete c, append to c, save a twice, then rename c→a again. Then a dirty repo-save a. Then an interrupted writer whose pending repo-save a and repo-delete c do not land.
The difference
The assertion is assertEquals(targeted, full) (around line 746), so Actual is the targeted recovery and Expected is the full rebuild. The catalog rows and the renames row are identical on both sides. The only difference is one file in the observed tree:
- data/@swamp/aws-ec2/6f1c2a3b-4d5e-4f60-8a7b-9c0d1e2f3a4b/c/latest: d4735e3a… (sha256 of "2") <- targeted
+ data/@swamp/aws-ec2/6f1c2a3b-4d5e-4f60-8a7b-9c0d1e2f3a4b/c/latest: 6b86b273… (sha256 of "1") <- full rebuildTargeted recovery leaves c/latest pointing at version 2; the full rebuild points it at version 1. In the same tree, c/2/raw exists and no c/1/* files are listed for that model. Which side is right, or whether both are wrong, has not been worked out.
Not established
- The root cause.
- Whether the fault is in the targeted path, in the full rebuild, or in how
latestis derived after a rename onto a deleted name. - Whether #4677's rewrite changed the mechanism. It still fails after #4677, but that is the only thing checked.
Expected
Targeted recovery leaves the same tree as a full rebuild for this history, and the test passes with this seed pinned. Add the seed as a fixed regression case so it doesn't depend on a random draw. swamp-club#4693 is held until this test is green.
Shipped
Click a lifecycle step above to view its details.
Sign in to post a ripple.