#3278 Datastore rework Phase 3: a v3 home for audit, telemetry, logs and files, dormant (no behaviour change)

Background (read this first)

swamp is reworking its datastore layer (tracking: swamp-club#2865). In the target design, v3:

• Every change is a commit to a partition in a single-writer SQLite commits.db, made through a commit-log unit of work from typed changes.
• Content over 8 KiB is stored as chunks through storage/1.
• Each checkout's local files are rebuilt from commits by a read model.

All of that is on main (a7855165) and dormant, documented in design/enablers/datastores.md under the "v3 …" sections (swamp-club#3233, #3236, #3237, #3239, #3240, #3249, #3250, #3256, #3257, #3258). The next step is the v3 opt-in, filed in parallel with this issue. It makes v3 usable for new, empty filesystem datastores with a single writer. S3 and GCS v3 come after Phase 4 (Litestream).

The gap this issue closes. DEFAULT_DATASTORE_SUBDIRS (src/domain/datastore/datastore_config.ts) includes audit, telemetry, logs and files. Today they sync like every other datastore file. They aren't aggregate storage, though. They're high-volume, append-mostly streams: audit records, telemetry, run and method logs, and the files model methods write. They have no v3 home:

• no partition kind;
• no intents;
• nothing stages them.

So on a v3 datastore they would silently stop syncing. #3233 recorded this as an open requirement: "the opt-in issue must give them one before a v3 datastore can be enabled".

Goal

Decide where these four directories live on v3, implement that for filesystem datastores, and record a design that also works for S3 and GCS in Phase 4. It stays dormant until the opt-in wires it in.

Questions to answer first (record them in datastores.md)

1. What each directory actually holds, and who reads it. Trace every writer and reader in src/:

   • audit (serve's audit log and its local audit-wal);
   • telemetry;
   • logs (run and method logs; swamp workflow history logs, model output logs, serve log streams);
   • files (method file outputs, and whether data records or CEL .path reference them).

   For each, note whether readers on other checkouts or serve nodes need it, and how fresh it must be.

2. Which model fits each one. For example:

   • (a) Commit-log records. A partition per run or per log stream, written as puts of whole segments, with chunk content. That gives ordering, GC and read-model materialisation for free, but a commit per append may be too many.
   • (b) A separate append-only stream area. For example _streams/<ns>/<kind>/<id>/<segment>, written through storage/1 as immutable, content-addressed segments, with a small index. The commit log never sees them.
   • (c) Local only. It's never synced on v3, if no other checkout needs it. That would be a behaviour change from v2, so it needs a clear reason.

   files/ especially: if data or outputs reference these files by path, they probably need to be content-addressed puts in the owning model's partition, like data content, so a record never points at a missing file.

3. Ordering and atomicity with commits. A run's log must never claim lines from a run the commit log refused. An output's files must exist before the commit that names them. That's the same rule as chunks before commits.

4. GC and compaction. How each kind is reclaimed, and how it interacts with #3249's chunk GC and #3258's compaction.

5. Filesystem now, S3 and GCS later. On a filesystem datastore, the simplest correct answer may be writing to the datastore path directly. Make sure the design carries to storage/1 without a format change.

Implementation (filesystem, dormant)

• Implement the chosen model for each of the four kinds, behind the v3 rules (integration/datastore_v3_rules_test.ts), with no production importers until the opt-in.
• Provide the seam the opt-in will call. For example: a v3 writer and reader per kind, and a way for repositories and services to route these writes on a v3 datastore.
• v2 doesn't change. If any existing writer gains a hook point, the v2 gate applies: suites unedited, and the whole-suite route recorder identical. #3240 and #3250 describe the approach.

Tests

• Per kind, on a filesystem v3 datastore, two checkouts: what one writes, the other reads, with the freshness the decision promises.
• Ordering: an output's files exist before the commit naming them, and a refused run's logs aren't presented as that run's.
• GC and compaction leave every referenced stream segment or file readable.
• Property or round-trip tests for any new value objects (the ddd skill's table).

Out of scope

• Wiring into CLI and serve: the opt-in issue calls this issue's seam.
• S3 and GCS implementations (Phase 4, and the extension work on hold).

Done when

• The design for all four kinds is recorded, for every backend.
• The filesystem implementation and its tests are on main, dormant.
• All existing suites pass unchanged.

Blocked by: nothing. It runs in parallel with the v3 opt-in issue, which must not merge before this one: without it, a v3 datastore would stop syncing these directories.