#4096 Datastore rework Phase 5 (team mode): serve leader election on the v3 writer lease, followers, and serve's control plane in the bucket

Background

Datastore rework Phase 5 (team mode). The client side has shipped:

• serve's datastore API (#3915);
• the CLI's serve upstream (#3985), and a copy that refuses a serve not serving its log (#4049);
• the compacted base (#4006), and resets on a member's access change (#4089);
• datastore claim (#4063).

What's left is serve itself. A v3 serve runs single-instance only, unlike v2.

Most of the writer side was built in Phase 4:

• serve takes the fenced v3 writer lease, and lower fences are refused (WriterFencedError);
• each fence starts its own Litestream lineage;
• a paused writer that resumes after a takeover lands nothing (integration tests).

What's missing:

1. A second serve node can't follow. It waits out the lock timeout and fails with V3WriterBusyError (v3_writer_lock.ts).
2. Serve's control plane lives in the checkout, not the bucket: instance heartbeats (instance_heartbeat.ts), active and pending runs (active_run_tracker.ts), continuation claims and boot reconciliation (boot_reconciliation.ts). Other nodes can't see it. #3962 is the result: a run left running by a stopped serve is never settled, neither by another node nor by a quick restart of the same checkout.
3. Serve moves its lease to a new fence on every probe while a replacement replica keeps failing (PR #3371's review). v3_repo_context.ts sets advancedFor back to undefined on any failure, so serve fences its own commits once per probe for as long as the outage lasts.

Design: design/enablers/datastore-commit-log.md (single-leader team mode and its failover), and the "Team mode" sections of design/enablers/datastores.md.

What to build

1. Leader election on the writer lease.
   • A serve node that finds the lease held follows instead of failing: it serves reads from its own copy, kept current from the replica or the leader, and watches the lease.
   • When the lease expires, a follower takes it: a new fence, a new lineage, a restore to the latest replicated state first.
   • Writes reaching a follower are refused for now with a stable, retryable code that names the leader's address. Forwarding writes to the leader is the next item (p5-feed), so keep the refusal where p5-feed will replace it.
2. The control plane moves under the bucket. Heartbeats, active and pending runs, continuation claims, and boot reconciliation's view of other instances become datastore state every node reads.
   • Recommend: records under a control-plane namespace in commits.db, written by the leader under its fence, so they replicate and fail over with the log.
   • A node judges another instance's runs by that instance's heartbeat in the shared state. That closes #3962: a stopped serve's running run is settled by the next leader once its heartbeat lapses.
3. One fence advance per outage (from PR #3371): a replacement replica that keeps failing moves the lease at most once until a replacement succeeds. Test with a red check.
4. Keep #4063's ownership rule. The owner of a claimed datastore is serve's address, not a node: every node behind that address writes as the owner, and an owner serve needs --address. A takeover must not look like a change of owner, and a node started with another --address stays refused.

Out of scope (next items)

• Forwarding follower writes to the leader, and commits.subscribe (p5-feed).
• Model leases with fencing tokens (p5-leases).
• The failover drills on real buckets (p5-failover). This issue still needs an in-process takeover test.

Tests

Use Workers as machines, as the v3 multi-machine tests do:

• Two nodes: one leads; the other follows, serves reads, and refuses a write with the retryable code naming the leader.
• Takeover: stop or pause the leader; the follower takes the lease after expiry and writes under a higher fence; the old leader, resumed, lands nothing (WriterFencedError).
• Shared control plane:
  • a run left running by the stopped leader is settled interrupted by the new leader once the old heartbeat lapses;
  • continuation claims survive the switch;
  • #3962's two scenarios (second checkout, and a quick restart of the same checkout) both settle.
• Claims: a claimed datastore keeps its owner across a takeover; a node with another --address is refused.
• Fence churn: a replacement replica failing N times advances the fence once.
• Filesystem and bucket (emulator) datastores.

Where

src/serve/ (instance_heartbeat.ts, active_run_tracker.ts, boot_reconciliation.ts, startup), src/infrastructure/datastore_v3/v3_writer_lock.ts, v3_datastore_binding.ts, v3_repo_context.ts, and the design docs.

Includes swamp-club#3962: use link_issue so it ships with this.