#3249 Datastore rework Phase 3: chunk garbage collection for unreferenced chunks, dormant (no behaviour change)

Background (read this first)

swamp is reworking its datastore layer (tracking: swamp-club#2865). In the target design, v3:

• Commits. Every change is a commit in a single-writer SQLite commits.db, replicated by Litestream.
• Chunks. Content over 8 KiB is stored as immutable, content-addressed chunks in object storage through the storage/1 contract. A put op references them by a BlobRef: sha256 is the whole content, and parts lists the chunks when there are several.
• Write order. Chunks are stored before the commit that names them.

Phase 3 builds the v3 engine dormant: nothing reaches it until a user opts a new, empty datastore into v3.

What's on main (8453c3cc). It's all dormant and documented in design/enablers/datastores.md under "v3 commit store", "v3 chunk storage", "v3 read model" and the commit-log unit sections:

• storage/1 (src/domain/datastore/datastore_storage.ts): has, put (atomic, idempotent, refreshes lastModified), get, optional touch (returns the hashes it did not find), list(namespace) yielding { sha256, lastModified }, and delete (idempotent). Chunk keys are <prefix>/_blobs/<ns>/sha256/<aa>/<sha256>. ns is the namespace slug, or _default for an unnamespaced datastore. The filesystem reference implementation is src/infrastructure/persistence/fs_datastore_storage.ts.
• ChunkStore (src/domain/datastore_v3/chunk_store.ts): stores chunks before any commit names them. A chunk already present is refreshed (touch, or put again), so every chunk a commit is about to name is present and fresh. Writes trust has, so a chunk damaged in place is never repaired by a later write; reads fail with ChunkIntegrityError.
• The commit store (src/infrastructure/datastore_v3/sqlite_commit_store.ts): one commits.db per datastore, a namespace column, changesSince(cursor) and readPartition. Its schema is versioned in PRAGMA user_version, with append-only migrations.
• The commit-log unit (#3240): stores chunks and then appends. An abandoned unit, a refused commit or a crash between the two leaves chunks that no commit names.
• The read model (#3239): applies commits in order to a checkout's local files. It has chunksInUse(namespace), the chunks behind that one checkout's present records.

The constraint that sets this issue's scope. The read model applies each commit in turn and reads every put's chunk as it goes. A fresh checkout catching up from cursor 0 replays the whole history, so it needs every chunk that any commit ever named. That includes content later overwritten, and data versions later tombstoned by data gc or prune. Deleting such a chunk would make a fresh catch-up fail with ChunkMissingError on a commit it can never get past.

chunksInUse also can't be the reference set. It describes one checkout's present records at that checkout's cursor, which may be behind the store.

Goal

Reclaim chunks that no commit references: those left by abandoned units, refused or failed commits, and crashes between storing chunks and committing. Find damaged chunks, and clean up orphaned temp files from filesystem puts. It stays dormant.

Reclaiming superseded content (chunks only historical commits reference) needs history compaction first, so it's out of scope. The plan tracks it separately.

Design

1. The reference set comes from commits.db, not a read model.
   • Add a referenced-chunk index to the commit store, for example a chunk_refs(namespace, sha256) table written in the append's own transaction.
   • Each put's BlobRef adds its whole-content sha256 when it has no parts, or every part when it has.
   • Ship it as a schema migration, version 1 to 2. It backfills from existing rows and is tested on a version-1 database.
   • A refused or rolled-back append writes no refs.
2. The sweep, per storage namespace:
   1. take now;
   2. read the referenced set;
   3. list the stored chunks;
   4. delete a chunk only when it is not referenced and its lastModified is older than now - grace.
   • The grace period protects a unit's chunks stored before its commit lands. Set the default well above a unit's longest life (say 24 h) and make it configurable for tests. Record why in datastores.md.
   • Order matters. Read the referenced set after now is fixed. A chunk referenced by a commit that landed during the sweep was stored or refreshed after now - grace, so it is never a candidate.
   • Races with a concurrent write:
     • a write that refreshes a candidate before its delete keeps it, because touch and put move lastModified;
     • a write that runs has just before a delete and then commits loses the race. Look for a sound fix, for example re-checking lastModified immediately before each delete, or a two-phase mark then delete across sweeps. Document which case remains and why the grace period makes it unreachable in practice;
     • add a test for each.
3. Damaged chunks. Optionally, within a size or time budget, verify stored chunks by reading them and checking their SHA-256. A damaged chunk is deleted, so the next write of that content stores it again (writes trust has). Report what was found. If a damaged chunk is referenced, reads already fail; say that in the report, but still delete it so a re-put can repair it.
4. Filesystem temp files. Remove orphaned temp files that FileSystemStorage.put left under _blobs/ once they're older than the grace period. list already skips them.
5. Port and placement. A ChunkCollector in src/infrastructure/datastore_v3/ (or a domain service plus an infrastructure adapter), taking the commit store, DatastoreStorage, the namespace, the grace period and an abort signal. It returns a report: scanned, referenced, deleted, kept because too new, damaged and temp files. Nothing in production calls it yet. A future datastore gc for v3 wires it at the opt-in.
6. Not here:
   • superseded-content reclamation (see Background);
   • splitting large data gc/prune tombstone sets across commits. That belongs to the data and output staging issue, because a unit refuses more than MAX_COMMIT_OPS changes for one partition, so a checkpoint must come first.

Tests

• The referenced-chunk index:
  • one-chunk and multi-chunk refs;
  • inline puts add none;
  • a refused append adds none;
  • the migration backfill matches a fresh build;
  • a property test that the index equals the set computed by scanning every stored commit.
• The sweep on the filesystem storage/1 and a real temp commits.db:
  • unreferenced and old chunks are deleted;
  • referenced chunks are never deleted, whatever their age;
  • unreferenced but fresh chunks are kept;
  • a chunk refreshed or committed mid-sweep survives, using injected clocks and hooks, never real sleeps;
  • namespaces are isolated, including _default.
• Damage and temp files: a damaged chunk is reported and deleted, then a later ChunkStore.write of the same content stores it again; old temp files are removed and new ones kept.
• The history guarantee: after a sweep, a fresh read model catching up from cursor 0 succeeds over a history with overwrites, tombstones and data gc.
• Dormancy: extend integration/datastore_v3_rules_test.ts.

Out of scope

• Superseded-content reclamation and history compaction (tracked in the plan).
• Wiring a v3 datastore gc command (at the opt-in).
• The S3 and GCS storage/1 implementations (on hold).
• Splitting data gc/prune across commits (data staging).

Done when

• The index, the migration, the sweep, damage detection and temp-file cleanup are on main, dormant, with these tests.
• The grace-period and race reasoning is recorded in datastores.md.
• All existing suites pass unchanged.

Blocked by: nothing (#3236, #3237, #3239 and #3240 have merged). It runs in parallel with the data and output staging issue.