#3985 Datastore rework Phase 5 (team mode) 2/3: the CLI's serve upstream — with serverAddress, a v3 checkout commits, reads chunks and catches up through serve's datastore API, with no storage credentials

Background

swamp's datastore rework (tracking: swamp-club#2865) is in Phase 5, team mode: a team writes a v3 datastore through swamp serve, which holds the writer lease, and laptops and CI hold a serve token instead of storage credentials. v3 on buckets is announced only once team mode works.

Team mode's client work is three issues:

1. #3915 (shipped, PR #3368): serve's datastore API. Under /api/v1/datastore/{namespace}/:

   • POST chunks/has, PUT and GET chunks/{sha256};
   • POST partitions/{p}/commits, framed with the chunks it carries;
   • GET partitions/{p}/commits?after= (catch-up);
   • GET changes?cursor=, filtered to what the caller may read.

   Every op is authorised on its intent, serve stamps the caller as principal, and conflicts are 409s. #3938 hardened it.

2. This issue: the CLI's serve upstream. The CLI side, so a checkout configured for serve reads and writes the datastore through that API.

3. Later: swamp datastore claim and release, so direct writers refuse once serve owns a datastore.

Today, on main, serverAddress (.swamp.yaml) and SWAMP_SERVE_URL are read only by resolveServeUrl (src/cli/remote_run.ts), to route --server runs and admin commands. No datastore code reads them. A v3 checkout always writes its storage directly: it takes the bucket's writer lease, commits to its local commits.db through the outbox (outbox_commit_sink.ts), stores chunks through DatastoreStorage, and Litestream replicates commits.db.

What to build

When a v3 repo has a serve address (serverAddress in .swamp.yaml, or SWAMP_SERVE_URL) and the user has a serve token (servers.json, swamp auth server-login), its embedded datastore service uses serve as its upstream instead of storage. Everything above the service (commands, output, the projection with real file paths, the catalog, the outbox, catch-up) stays as it is in solo mode.

1. Commits go to serve. The outbox's sink forwards each commit to POST partitions/{p}/commits with its expectedSeq, txnId, fence and ops, carrying the chunks serve doesn't hold. Ask chunks/has first; resume large chunks with PUT chunks/{sha256}. The outbox keeps its durability rules: an entry settles only on serve's {seq} (or a 409 for a duplicate txnId, which is landed).
2. Chunks are read from serve. A DatastoreStorage adapter over the chunk endpoints (has, get with Range, put). It must pass the existing storage conformance suite (packages/testing/storage_conformance.ts) against a port-0 serve, for what serve's API supports. Name any capability it can't offer, and why.
3. Catch-up comes from serve. The checkout's commits.db becomes a local copy of serve's commit log, filled from GET changes?cursor= (and partitions/{p}/commits?after= per partition). It's never replicated, and the client never takes the bucket's writer lease or starts a Litestream sidecar.
4. Conflicts read as #3915 words them. A stale expectedSeq (409 with the current seq) catches up and retries, as solo mode does. A failed precondition shows serve's "changed by … at …" message. A fenced, lease-lost or unreachable-storage reply keeps its existing code.
5. When serve is unreachable, writes refuse; they never fall back to storage. A write fails with a stable code (exit 75, nothing changed) when serve can't be reached. Reads use the last applied local state and say it may be out of date. Queueing offline writes is a later item (p5-offline); until then, refuse.
6. Auth: the user's serve token, as --server runs use it. A missing or expired token fails with the existing auth error and its "server-login" hint. The datastore config's storage credentials are never read in this mode.
7. Design: add a "Team mode: the CLI's serve upstream" section to design/enablers/datastores.md, next to #3915's section.

Today every content-bearing write through the API needs admin (definitions, evaluated files, vault configs, and PUT chunks), until swamp-club#3922 ships content-level parity. #3922 is in flight. So build and test this with an admin token now; non-admins benefit when #3922 lands. Don't widen any authorisation here.

Out of scope (later items)

• swamp datastore claim and release (client 3/3).
• The offline queue and rebase (p5-offline).
• Model leases (p5-leases) and per-file config edits (p5-config).
• commits.subscribe and the live feed (p5-feed).
• Follower serve nodes (p5-leader).
• The swamp-uat team-mode backend (p5-uat), which follows this issue and claim.

Done when

• Reads and writes through serve: a CLI with a serve address and only a serve token (no storage credentials anywhere in its environment) creates, runs, reads and deletes through serve on a v3 datastore. The result matches a solo checkout of the same commands, in integration tests against a port-0 serve on filesystem v3 and on S3 and GCS emulators.

• Conformance: the serve-backed chunk storage passes storage_conformance.ts.

• Errors:

  • an unreachable serve refuses writes with exit 75 and never touches storage;
  • a stale seq catches up and retries;
  • a precondition conflict shows who changed what;
  • an expired token shows the login hint.

  Each has a test that fails without its rule.

• No change for solo checkouts: no serve address means exactly today's behaviour, and the swamp-uat datastore suites pass unchanged.

• The design section is written.

Blocked by: nothing. #3915 has shipped, and it runs in parallel with #3922.