#3564 v3: writer liveness is judged wrong: a same-hostname container takes over a live writer, a stalled writer's file is reverted then committed, and a fenced retention pass rewinds the new writer (bug hunt group 1)

Background

swamp is reworking its datastore layer (tracking: swamp-club#2865). On a v3 datastore one process at a time holds the fenced writer lease (swamp-club#3359): records under _control/writer/ with a fence that only increases. A holder whose process is gone on this host can be taken over at once; any other holder only after its TTL plus skew (30 s + 10 s). The same "is the writer still alive?" judgement also decides:

• whether a read-only command may catch up in a writer's checkout (the writer.checkout.json sidecar);
• whether interrupted-writer recovery (swamp-club#3299) removes a writer's uncommitted files (the writer marker);
• whether outbox recovery (swamp-club#3490) lands a writer's prepared commit.

A v3 bug hunt (2026-10-16, verified on main d3b76490c) found that this judgement is wrong in several ways. Today it bites filesystem v3 (opt-in). Phase 4E (swamp-club#3534, PR #3180) puts v3 on S3 and GCS, where takeovers between machines are routine. This issue fixes the liveness rule once, for every place that uses it.

The bugs

1. Same hostname is treated as same host (V3-03, S2). judgeWriterLease (writer_lease.ts:174-179) returns process-gone when the hostnames match and the pid isn't found locally. v3_writer_lock.ts:526-535 records only Deno.hostname(). The same rule is in interrupted_writer_recovery.ts:183-189 (writerMarkerState), :289-296 (lockHeldByLiveProcess) and commit_outbox_recovery.ts:140-145 (defaultWriterAlive). processHostIdentity() (process.ts:35-45, hostname#pidns) exists for exactly this, but only the catalog uses it.

• Reproduced with two Docker containers (denoland/deno:2.9.7, both --hostname shared-host, the datastore bind-mounted into both). A container took over a live, renewing holder's lease in 5 ms (fence 1 → 2) and judged its writer marker interrupted. Same-namespace control: V3WriterBusyError, as it should.
• Consequences, confirmed by code: the taker's outbox recovery lands the live writer's prepared entry; a read-only command in the live writer's checkout runs recovery and deletes its uncommitted files (interrupted_writer_recovery.ts:230-256).
• Trigger: docker run --network host, podman host networking, Kubernetes hostNetwork pods sharing a bind-mounted repo or datastore.

2. A stalled same-host writer is judged gone by TTL alone (V3-02, S2). anotherProcessWritesCheckout (v3_repo_context.ts:585-614) returns false once acquiredAt + ttlMs < now, with no skew and no pid check. lockHeldByLiveProcess does prefer pid liveness.

• Reproduced: a lease held by a live pid, 35 s old, plus a sidecar naming the checkout. A read-only open in that checkout caught up and reverted the writer's in-place file from writer-uncommitted-v2 to committed-v1, while acquireV3WriterLock(maxWaitMs: 0) was still refused with V3WriterBusyError (the lease was still held).
• Consequence, confirmed by code: a stream-sourced put (> 8 KiB) reopens the file at commit (unified_data_repository.ts:327-331), while its metadata was captured at staging (:307). The writer then commits the reverted bytes under the new metadata (size mismatch) and reports success. Every checkout replicates it.
• Trigger: synchronous method or extension code (outputSync on a slow tool, CPU-bound work) or a suspend over ~30 s, plus a read command in the same checkout.

3. A fenced process's retention pass rejoins its lost hold (V3-07, S2). CLI teardown runs flushDatastoreSync → runRetentionAtCommandEnd → closeV3DatastoresAfterSuccess (cli/mod.ts:2838-2845). Retention opens requireInitializedRepo (retention_teardown.ts:123) → holdV3Writer. acquireV3WriterLock joins the existing hold even when it's lost, because the lost check only runs under retakeLost (v3_writer_lock.ts:182-217). holdV3Writer rewrites the sidecar with this pid on every join (v3_repo_context.ts:411-424), and removeOwnWriterCheckout (:561-570) deletes it on a pid + host match.

• Reproduced in process: sidecar repoB/pid P2 → after the join repoA/pid A → after close <missing>. A reader in P2's checkout then caught up and rewound P2's file, so P2's acknowledged commit landed stale content.
• Trigger: datastore lock release --force (which the busy error suggests), or a CLI paused > 40 s after its last commit, with garbageCollection.automatic + retention, while another writer works and someone reads in that writer's checkout.
• Note: PR #3180 (4E) adds confirmV3CommandReplicated() at the start of runRetentionAtCommandEnd; it doesn't change this.

4. The liveness probe resumes a stopped process (V3-16, S3). v3_writer_lock.ts:534 and :586 (refuseLiveV3LegacyWriter) probe with isProcessDead, which sends SIGCONT (process.ts:58-70). isProcessGone's own doc (process.ts:74-88) says persisted pids must be probed without disturbing them. Reproduced: a SIGSTOPped holder went from state T to S after v3WriterLockHandle().inspect() (that's lock status). Ctrl-Z on a long write, then lock status or any read, resumes it.

5. Pid reuse makes a dead writer look alive (V3-37, S3, design weakness). Outbox entries and writer markers carry only host + pid. Reproduced: with an unrelated live pid in the entry, the first recovery skips it, the floor rises anyway (commit_outbox_recovery.ts:228-232), and the next drops it as overtaken (commit_outbox.ts:297-298): 0 commits land, reported as "had lost the lease". The marker reads live and keeps the uncommitted file. Trigger: container restarts with a stable hostname and low deterministic pids on a persistent volume.

6. A zombie holder looks alive (V3-47, S4). Both probes call an unreaped (state Z) process alive (process.ts:58-100), so takeover waits TTL + skew (~38 s measured, 0.5 s for a reaped control). Trigger: a container whose PID 1 doesn't reap.

7. Lease contention exits 1, not 75 (V3-15, S3). exitCodeForError (src/presentation/output/error_output.ts:127-140) maps 75 only for lock_timeout, temporary auth blocks and replication-unconfirmed. V3WriterBusyError (datastore_v3_writer_busy) wraps a LockTimeoutError as cause but exits 1. model_method_run.ts:245 promises "75 = lock contention (temporary — retry with backoff)"; v2 contention exits 75.

What to do

1. One liveness rule, used everywhere (judgeWriterLease, anotherProcessWritesCheckout, writerMarkerState, lockHeldByLiveProcess, defaultWriterAlive):
   • record a process identity in the fence record, the outbox's CommitWriter and the writer marker: processHostIdentity() (host + pid namespace), plus boot id and process start time where the platform gives them (Linux /proc/sys/kernel/random/boot_id and /proc/<pid>/stat field 22; macOS kern.boottime and proc_pidinfo; Windows process creation time);
   • same identity, pid gone or start time differs or state Z/X → gone (immediate takeover, recovery runs);
   • same identity, pid alive with matching start time → alive, whatever the TTL says. This fixes (2): a same-host reader never catches up in a live writer's checkout;
   • identity differs or missing (other host, other namespace, a record from an older swamp) → TTL + skew and the fence only. Never judge such a holder dead. On a bucket datastore this is the normal case.
   • probe with isProcessGone everywhere, never SIGCONT (4). Consider the same change in FileLock and run_tracker_store, which also send SIGCONT; note it if you don't.
   • adding a field to a persisted record: check whether commits.db/outbox schema needs a no-op bump so older swamps say "upgrade" rather than "corrupt" (as #3532 did); fence and marker records are JSON files, so older readers should ignore the field. Say which.
2. A lost hold is never joined (3): acquireV3WriterLock refuses to join a lost hold even without retakeLost; holdV3Writer writes the sidecar only on first acquisition while the hold is current; removeOwnWriterCheckout compares the fence too; the retention pass is skipped when assertV3WriterCurrent throws. A hold.lost check alone isn't enough, since renewal may not have noticed the loss yet: check the fence record.
3. Don't raise the outbox floor past skipped entries (5): a skipped entry (writer judged alive) stays landable once its writer is judged gone.
4. Belt and braces for (2): at commit, check a stream-sourced put's size and sha256 against the staged metadata, and refuse on a mismatch.
5. Exit 75 for datastore_v3_writer_busy (7), and list it in the help text. Keep datastore_v3_writer_fenced at its current code (a fenced command isn't safe to blindly retry) and say so.
6. Plan against PR #3180 (4E) as well as main. It changes v3_writer_lock.ts (lease records via v3ControlPlaneAt, no Phase 3 legacy lock on buckets) and retention_teardown.ts. It will very likely merge first; expect to rebase over those two hunks. On a bucket, the "identity differs" branch is the common one: test it there.
7. Update design/datastores.md's lease and recovery sections with the identity rule, in this PR.

Tests

Each must fail on main first.

• Other pid namespace, same hostname: a holder whose recorded identity has a different pid namespace and a pid absent locally is refused (V3WriterBusyError), not taken over; a read-only open in its checkout leaves its files alone. In-process: plant the fence record and marker with a foreign hostname#pidns. A live two-container check (as in the hunt) is worth running once by hand; don't add it to the suite.
• Stalled same-host writer: plant a lease held by a live pid, older than its TTL, plus a sidecar; a read-only open in that checkout does not catch up.
• Retention after a lost hold: hold, force-release, plant fence 2 for another live pid and its sidecar; the first process's retention pass and close leave that sidecar unchanged.
• No SIGCONT: a SIGSTOPped holder stays in state T after inspect(), acquire and a read-only open (POSIX only; skip on Windows).
• Pid reuse: an outbox entry and a marker naming a live unrelated pid with a different start time are judged gone; the entry lands.
• Zombie: an unreaped holder is taken over at once (Linux; skip elsewhere if the platform can't report state Z).
• Exit code: exitCodeForError(new V3WriterBusyError(...)) is 75.
• Older records: a fence record and marker without the identity field are judged by TTL + fence only.
• swamp-uat passes unchanged on filesystem-v3 (and on s3-v3/gcs-v3 once H lands).

Severity: three S2s (a live writer taken over, a stalled writer's file reverted then committed, an acknowledged commit rewound); the rest S3/S4. All reachable on filesystem v3 today; more so once 4E ships.

Blocked by: nothing. Can start before PR #3180 merges.