#3237 Datastore rework Phase 3: the chunk store and the storage/1 contract, dormant (no behaviour change)

Background (read this first)

swamp is reworking its datastore layer (tracking: swamp-club#2865). In the target design, v3:

• Commits live in a single-writer SQLite commits.db, replicated by Litestream.
• Content is stored as immutable, content-addressed chunks: SHA-256 of the bytes, with identical content stored once per namespace.
• Small content (8 KiB or less) goes inside the commit instead (MAX_INLINE_BYTES).
• Larger content becomes one or more chunks, referenced from a put op by a BlobRef ({ sha256, size }, src/domain/datastore_v3/blob_ref.ts).
• Order of writes. Chunks are uploaded before the commit that names them, so a commit never refers to a missing chunk.

Phase 3 builds this engine dormant, reachable only when a user opts a new, empty datastore into v3, which is a later issue.

What's on main already (1b5a2958):

• The v3 domain model (swamp-club#3233, src/domain/datastore_v3/, "v3 domain model" in design/enablers/datastores.md). It has BlobRef, isSha256Hex and MAX_INLINE_BYTES, and integration/datastore_v3_rules_test.ts keeps the model dormant.
• The Litestream decision (swamp-club#3234, design/enablers/datastore-litestream.md). Litestream replicates commits.db with its own S3, GCS and file clients. It does not use swamp's storage contract.
• Today's extension contract. Extensions implement DatastoreProvider (src/domain/datastore/datastore_provider.ts, mirrored in packages/testing/datastore_types.ts), with a sync service (sync/2), locks, and the optional datastoreControlPlaneStore() added in #3191. @swamp-club/swamp-testing ships conformance suites for these (e.g. assertControlPlaneStoreConformance, assertDatastoreControlPlaneStoreConformance).

Why the contract can be smaller than the old draft. An earlier design draft had a seven-method storage/1 byte store (putIfAbsent, put, get with range, head, list with startAfter, delete, capabilities), because it kept partition logs, snapshots and the feed in object storage. Commits now live in SQLite and Litestream ships the database, so v3 object storage holds chunks only. The draft's ChunkStore interface was:

interface ChunkStore {
  maxChunkBytes(): number;                                   // core sizes chunks to min(16 MiB, this)
  has(namespace: string, hashes: string[], signal: AbortSignal): Promise<Set<string>>;
  put(namespace: string, sha256: string, body: ReadableStream<Uint8Array>, signal: AbortSignal): Promise<void>; // idempotent by hash
  get(namespace: string, sha256: string, signal: AbortSignal, range?: { start: number; end: number }): Promise<ReadableStream<Uint8Array> | null>;
  touch(namespace: string, hashes: string[], signal: AbortSignal): Promise<void>;  // refresh for the GC grace period
  listOlderThan(namespace: string, before: Date, signal: AbortSignal): AsyncIterable<string>;
  delete(namespace: string, sha256: string, signal: AbortSignal): Promise<void>;
}

Goal

Define the storage/1 contract that datastore extensions implement for v3 content. Build the core chunk store on top of it, and a filesystem reference implementation and conformance suite, so the S3 and GCS extensions can implement it next. It is dormant: nothing in production uses it yet.

Design

1. Settle the contract's shape and record the reasons in design/enablers/datastores.md:

   • Choose between a chunk-shaped contract (the ChunkStore above, possibly trimmed) and a minimal byte store that core's chunk store builds on. Prefer the smallest surface that serves chunks, GC (a later issue) and probing. Remember that extension authors must implement every method.
   • Name it with a contract string ("storage/1") and declared capabilities. Each capability needs a fallback that stays correct. For example, without range reads core buffers up to a cap; without touch, core re-uploads before the commit.
   • Add it as an optional DatastoreProvider member, e.g. storage?(): DatastoreStorage, in both src/domain/datastore/ and packages/testing/, mirroring how datastoreControlPlaneStore was added in #3191. Released extensions must keep working unchanged.
   • Key layout under the datastore prefix, e.g. <prefix>/_blobs/<namespace>/sha256/<aa>/<hash>. It sits beside _control/ and _litestream/ and is namespaced, because dedup is per namespace.

2. The core chunk store (src/domain/datastore_v3/ port plus an infrastructure implementation):

   • Chunking. Content above MAX_INLINE_BYTES is split into chunks of min(16 MiB, maxChunkBytes). Decide how a multi-chunk item is referenced: a manifest chunk listing the parts, or a list in the BlobRef. Record the choice, and extend BlobRef only if needed, keeping its JSON round trip.
   • Upload checks has first and skips chunks already present. Uploads are idempotent by hash.
   • Reads verify each chunk's SHA-256 before handing bytes on. A mismatch is an error, never silently returned.
   • Every call takes an AbortSignal.

3. The filesystem reference implementation of the contract, small and readable. It is the scaffold extension authors start from. Writes are atomic (temp file then rename), and it uses @std/path throughout, so Windows works.

4. The conformance suite in @swamp-club/swamp-testing, e.g. assertStorageConformance. It covers:

   • put then get round trips, has on present and missing hashes, idempotent re-put, and delete;
   • namespace isolation;
   • range reads, if declared;
   • touch and listOlderThan behaving as documented;
   • concurrent puts of one hash leaving one correct object;
   • declared capabilities matching behaviour;
   • abort honoured.

   Run it against the filesystem implementation and the in-memory remote (packages/testing/in_memory_remote.ts; add a minimal storage there behind an option, so existing behaviour is unchanged).

5. Docs. Add the contract and suite to the bundled skill's datastore extension references (.claude/skills/swamp/references/extension/references/datastore/api.md and testing.md), as #3191 did. Run deno fmt, and expect verify-skills to review it.

Tests

• Unit tests for chunking (boundaries at MAX_INLINE_BYTES, exactly one chunk, many chunks, empty content) and digest verification.
• A property test (fast-check): random content split, stored and read back gives identical bytes and stable BlobRefs, and identical content gives identical references.
• The conformance suite passing on the filesystem implementation and the in-memory remote.
• An integration test of the chunk store over the filesystem implementation on a real temp directory.
• Dormancy: no production code outside the v3 directories imports the chunk store, enforced by extending datastore_v3_rules_test.ts. The new optional DatastoreProvider member is the only change to existing types.

Out of scope

• Chunk garbage collection. Deciding which chunks no commit references needs the commit store or read model, so it's a later issue after both exist. Only the primitives GC will need, such as touch and listOlderThan, belong here.
• The S3 and GCS implementations (a later swamp-extensions change, filed as a swamp-club Lab issue because swamp-extensions has issues disabled).
• The commit store, the read model, the outbox, Litestream and the opt-in.

Done when

• The contract, chunk store, filesystem implementation and conformance suite are on main, dormant.
• The contract decision is in datastores.md, and the skill references are updated.
• All existing suites pass unchanged.

Blocked by: nothing (#3233 has merged). It runs in parallel with the commits.db commit-store issue.