Skip to main content
← Back to list
01Issue
BugShippedSwamp CLIPublic
Assigneesstack72

Relationships

#4723 v3 targeted recovery leaves a different data/<model>/<name>/latest than a full rebuild after a rename onto a deleted name with an interrupted writer (deterministic property-test seed)

Opened by stack72 · 11/13/2026· Shipped 11/13/2026

Related: #4712 (wall-clock flakes in the same recovery tests; a different failure), #4702, #4677, #4581 (the targeted-recovery work this test covers).

What failed

  • Test: integration/datastore_v3_targeted_recovery_property_test.ts, the case "targeted recovery: leaves the catalog, renames, files and live records a full rebuild leaves, whatever the history and whatever an interrupted writer or dropped work left (swamp-club#4581)".
  • Where it was seen: a verify-build run on branch 4693 (swamp-club#4693, serve auto-continuation), commit 0549ab2f, 2026-11-13 15:15 UTC.
  • fast-check output: Property failed after 146 tests, { seed: 206505560, path: "145:2:2:1:3:6:9:9:8:8:8:8:8:8:8:10:10:10:10:10:10:11:10:10:14:14", endOnFailure: true }.

It is not caused by the 4693 branch

That branch touches nothing under src/infrastructure/datastore_v3/, src/domain/datastore/ or this test. With the seed pinned, the test fails the same way on three clean checkouts:

Checkout Result
529118c0a, the 4693 merge base (none of its commits) Property failed after 1 tests
0549ab2f, the 4693 head Property failed after 1 tests
origin/main at 150772096, which includes #4677 3fc69ff93 Property failed after 1 tests

It fails in about 100 ms on an idle machine, so load and timing are not involved.

Why it has not shown up elsewhere

The test runs 150 random histories per run, with a fresh seed each time. Most seeds never generate this history; this build happened to draw one that does. A plain rerun draws a new seed and will very likely pass, but the bug stays.

Repro

In integration/datastore_v3_targeted_recovery_property_test.ts, change the fc.assert options ({ numRuns: 150 }, near line 768) to:

{ numRuns: 150, seed: 206505560, path: "145:2:2:1:3:6:9:9:8:8:8:8:8:8:8:10:10:10:10:10:10:11:10:10:14:14", endOnFailure: true }

Then run deno test -A integration/datastore_v3_targeted_recovery_property_test.ts --filter "whatever the history".

Counterexample (shrunk)

[[{"model":0,"kind":"rename","name":"c","to":"a","pick":0},
  {"model":0,"kind":"delete","name":"c","to":"a","pick":0},
  {"model":0,"kind":"append","name":"c","to":"a","pick":0},
  {"model":0,"kind":"save","name":"a","to":"a","pick":0},
  {"model":0,"kind":"save","name":"a","to":"a","pick":0},
  {"model":0,"kind":"rename","name":"c","to":"a","pick":0}],
 0,
 [{"model":0,"kind":"repo-save","name":"a","pick":0}],
 {"pending":[{"model":0,"kind":"repo-save","name":"a","pick":0},
             {"model":0,"kind":"repo-delete","name":"c","pick":1}],"lands":false},
 "interrupted_writer",
 undefined]

The history in plain words: rename c→a, delete c, append to c, save a twice, then rename c→a again. Then a dirty repo-save a. Then an interrupted writer whose pending repo-save a and repo-delete c do not land.

The difference

The assertion is assertEquals(targeted, full) (around line 746), so Actual is the targeted recovery and Expected is the full rebuild. The catalog rows and the renames row are identical on both sides. The only difference is one file in the observed tree:

- data/@swamp/aws-ec2/6f1c2a3b-4d5e-4f60-8a7b-9c0d1e2f3a4b/c/latest: d4735e3a…  (sha256 of "2")  <- targeted
+ data/@swamp/aws-ec2/6f1c2a3b-4d5e-4f60-8a7b-9c0d1e2f3a4b/c/latest: 6b86b273…  (sha256 of "1")  <- full rebuild

Targeted recovery leaves c/latest pointing at version 2; the full rebuild points it at version 1. In the same tree, c/2/raw exists and no c/1/* files are listed for that model. Which side is right, or whether both are wrong, has not been worked out.

Not established

  • The root cause.
  • Whether the fault is in the targeted path, in the full rebuild, or in how latest is derived after a rename onto a deleted name.
  • Whether #4677's rewrite changed the mechanism. It still fails after #4677, but that is the only thing checked.

Expected

Targeted recovery leaves the same tree as a full rebuild for this history, and the test passes with this seed pinned. Add the seed as a fixed regression case so it doesn't depend on a random draw. swamp-club#4693 is held until this test is green.

02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 2 MOREREVIEW+ 6 MOREPR_LINKED+ 2 MORESESSION_SUMMARIZED

Shipped

11/13/2026, 8:13:30 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack7211/13/2026, 6:22:48 PM

Sign in to post a ripple.