Relationships
#4527 fix(datastore): the run-record judge freezes a fan-out job's snapshot with its plan, so a forwarded record cannot rewrite the selector or worker scopes a resume trusts
Opened by stack72 · 11/8/2026· Shipped 11/8/2026
Problem
#4516 (PR #3671, f4852124e) added fanOutSnapshot to the job run, beside the frozen fan-out plan. JobRun.recordFanOutPlan writes it in the same call as the plan. It holds:
- the job's worker selector (labels and platform) as authored when the run began
- each lane worker's
self.workerscope (name,labels,platform,arch)
Resume trusts both. The selector decides the "selects different workers than when the run began" refusal (fanOutStructure in src/domain/workflows/resume_reset.ts). A stored scope becomes self.worker for a resumed lane whose worker has left the pool (execution_service.ts).
judgeRunRecord freezes the plan: fanOutPlanRefusal in src/domain/datastore_v3/content_bindings.ts refuses a record that adds a plan after the job's steps began, or changes or drops one. Nothing in the judge looks at fanOutSnapshot. So a forwarded run record (team mode) can add, change or drop the snapshot after the plan is written, and resume then:
- skips the "labels or platform changed" refusal, or
- gives a resumed lane a
self.workerscope that does not describe its worker.
It cannot add hosts or move a lane to another worker, because the plan itself stays frozen. It is still a gap in what the judge exists to guarantee.
Not a regression. Both the snapshot and fan-out resume are new in #4516. Before it, every resume of a fan-out run was refused.
Proposed fix
Treat fanOutSnapshot exactly as fanOutPlanRefusal treats fanOutPlan:
- It may only be written in the same record that first writes the job's plan.
- Once written, it is never changed or dropped.
- A record that carries a snapshot without a plan is refused.
Extend fanOutPlanRefusal, or add a sibling check called beside it, with messages in the same form ("it changes the fan-out snapshot of job '…'").
Runs from the release that shipped #4506 have a plan and no snapshot. They stay as they are: resume already refuses them for having no recorded selector, and the judge must not start refusing records that keep them unchanged.
Done when
Judge unit tests (content_bindings tests), each red on main:
- A record that writes the plan and the snapshot together is accepted.
- A later record that changes the snapshot's selector, or one worker scope, is refused.
- A later record that drops the snapshot is refused.
- A later record that adds a snapshot to a job whose plan is already written is refused.
- A record with a snapshot and no plan is refused.
- A record from a #4506-era run (a plan and no snapshot), saved again unchanged, is accepted.
Also:
- The existing plan-freeze tests and the fan-out resume suites (
integration/fleet_fan_out_resume_test.ts) pass unedited. - If the run-record judge has a model-based property test, its forgery set gains "forged snapshot".
Context
- Found reviewing PR #3671 before it merged. It was left out of that PR on purpose, as not a regression, to be fixed next.
- Fan-out design: https://claude.ai/artifact/WbnKGvQcuJdo6jQBbezDxz
- Work plan (Phase 2, "Freeze the fan-out snapshot"): https://claude.ai/artifact/CTfkUbRmw6JxccPyyVhY9H#w-snapshot. This card holds the Phase 3 gate.
Shipped
Click a lifecycle step above to view its details.
Sign in to post a ripple.