#3801 v3 identity on buckets: S3 checkouts with different endpoint spellings fork the datastore silently, and a hostname change blocks every write for ~5 minutes (found by #3756)

Background

swamp's datastore rework (tracking: swamp-club#2865) has finished solo mode: v3 datastores on filesystem, S3 and GCS, with one writer at a time and several checkouts taking turns. On a bucket, each checkout's commit log (commits.db) is replicated by a Litestream sidecar into a lineage in the bucket, and a fenced writer lease says which process may write.

The Phase 4 dogfood (swamp-club#3756) found two ways v3 decides "is this the same thing?" by the wrong property. Both are in the replication and lease code, so they are one fix. Both were reproduced on main at bc388c720 against ministack and fake-gcs.

1. S3 checkouts that spell the endpoint differently fork the datastore silently

Symptom. A checkout whose S3 endpoint is spelled differently from the writer's, against the same bucket and prefix:

• sees an empty datastore, with no warning;
• starts its own lineage with its first command, even a read-only one (workflow approvals);
• writes into that lineage, where no other checkout ever sees the writes, and it never sees theirs.

The bucket ends up holding two lineage chains sharing one _control/ and _blobs/. The empty view also happens on the released binary.

Repro (~/dogfood-3756/repro_endpoint.sh):

1. The writer uses endpoint: http://localhost:47441 (a proxy in front of ministack) and writes sh1 result v1 to v3.
2. A clone with the same spelling sees v3.
3. A clone with http://localhost:47411 (ministack directly, same bucket and prefix) runs data list sh1: [].
4. That clone's workflow approvals creates lineage 000000000005.
5. That clone's write lands only in lineage 5, where result is at v1.
6. The writer's next write starts lineage 7 and removes lineage 1. The bucket now holds lineages 5 and 7 side by side.

Cause. replicaRootOf (src/infrastructure/datastore_v3/litestream/replica_setting.ts) builds the S3 identity as s3:${endpoint ?? ""}:${bucket}:${path}, and currentLineage (src/domain/datastore_v3/published_lineage.ts) trusts only lineages started under the same root string. GCS's root is gcs:${bucket}:${path}, so GCS is not affected.

When it bites: an endpoint from the environment (a VPC endpoint in CI, the public one on laptops); the default AWS endpoint set explicitly in one checkout and left implicit in another; or the same endpoint with a trailing slash, a different scheme or a different case.

Fix:

• Match lineages on bucket and path only. The endpoint is how a checkout reaches the bucket, not which replica it is, just as forcePathStyle already isn't. Lineages are stored inside the bucket itself, so a truly different S3 service never sees another service's lineages; the endpoint adds no protection.
• Keep writing the root string exactly as today. Older binaries with the same spelling then still match a lineage the new binary starts, so no format change and no schema bump are needed. Older binaries with a different spelling keep today's behaviour until they upgrade.
• A command that only reads must never start a lineage, or publish any lineage record. With nothing to restore, a read-only command reads an empty datastore and leaves the bucket untouched.
• Never fork silently: if a lineage is found whose root differs from this checkout's in anything other than the endpoint, refuse with a stable code naming both roots, rather than starting a new lineage beside it.

2. A hostname change blocks every v3 bucket write for about 5 minutes

Symptom. From 18:22 to 18:27, 58 benchmark writes failed on every v3 bucket repo at once with datastore_v3_replication_unavailable: "swamp cannot replicate this v3 datastore: another user or host runs this datastore's Litestream sidecar. Nothing was changed." Nothing else touched those repos. At 18:22:14 macOS configd had set the hostname to Mac.localdomain during a DHCP renew (it had flipped once already that day). No person changed it.

Repro, no sudo (~/dogfood-3756/repro_hostname.sh): start a live sidecar, rewrite the hostname in its presence.json, then write every 15 s. Writes are refused from +0 s to +298 s; the first success is at +315 s, when the idle sidecar exits.

Cause. The replicator judges a sidecar's owner by record.hostname !== Deno.hostname() || record.uid !== Deno.uid() (litestream_replicator.ts, around lines 456, 676 and 703). After a hostname change this machine's own live sidecar counts as foreign, so it is neither reused nor replaced. Refused writes don't touch use.lock, so the sidecar idles out 5 minutes after the last successful write.

The writer lease has the same exposure. A process recognises its own lease and checkout records by pid plus hostname (src/infrastructure/datastore_v3/v3_repo_context.ts, removeOwnWriterCheckout, isOwnWriterCheckout, mayBeAnotherWriter, legacyWriterWritesCheckout), and refuseLiveV3LegacyWriter in v3_writer_lock.ts treats another hostname as alive. A long-running process such as serve that sees a hostname change mid-run stops recognising its own records. This was not seen failing, and not tested.

Fix:

• Judge "same host" and "my own record" by process identity, never by hostname. Use the identity swamp-club#3564 introduced for the lease's liveness rule (boot id, pid and process start time), plus isSupervisorFor's check that a live pid's command line names this database. The hostname stays in records for display only.
• Apply it to every hostname comparison under src/infrastructure/datastore_v3/: the replicator, the writer-checkout records and the legacy lock.
• A record from before this change, which has no process identity, keeps today's rule.

Done when

• Endpoint: two checkouts with different spellings of one S3 endpoint (with and without a trailing slash, explicit vs implicit default, a proxy vs direct) see and write one datastore, with one lineage chain in the bucket. A read-only command on an empty or unrestored checkout publishes nothing to the bucket. A lineage under a genuinely different root is refused with a stable code, never forked.
• Hostname: writes keep working across a hostname change, with the sidecar reused, in the CLI and in a running serve. A sidecar or lease record of another live process, including one in a container that shares the host's boot id, is still treated as another owner.
• Tests red on main first for each of these: unit tests for root matching and process-identity ownership, and an integration test for each scenario. Test the hostname change through an injectable hostname, not by changing the machine's.
• The swamp-uat datastore suites pass unchanged.

Severity: high. The endpoint fork splits one datastore in two without a word; the hostname change blocks every write on a laptop for up to 5 minutes, with a message blaming another host. Blocks announcing v3 on buckets (swamp-club#3756).

Blocked by: nothing. Runs in parallel with the outage-errors and clone-performance issues filed alongside it.