Relationships
#3299 Datastore rework: on v3, an interrupted method run's outputs never reach other clones (SIGTERM should commit; SIGKILL must be visible)
Opened by stack72 · 10/12/2026· Shipped 10/12/2026
Background (read this first)
swamp is reworking its datastore layer (tracking: swamp-club#2865). The new format, v3, records every change as a commit in a single-writer SQLite commits.db. Content is stored as chunks, and each checkout's local files and _catalog_v3.db are rebuilt from the commits by a read model that catches up when a context opens.
swamp-club#3279 made v3 usable for new, empty filesystem datastores with a single writer, through swamp datastore setup filesystem --path <dir> --format 3. It's released from 20261012.143252.0-sha.cc63f3a8. The design is in design/enablers/datastores.md, under the "v3 …" sections and the opt-in section.
Facts about v3 that this issue depends on:
- Repositories still write local files, and also stage typed changes into a commit-log unit. A commit lands when the unit ends.
withUnitOfWork(src/libswamp/unit_of_work.ts) ends a v3 unit by decision 5:- it commits on
completed,suspended,cancelledand a reportederror; - it abandons on a throw or an early stop.
- it commits on
- A second checkout joins as a git clone does: the same
.swamp.yamland repo id, and a fresh.swamp/. Its first command catches up. - v2 must not change. Any shared code path needs the v2 gate that #3240, #3250 and #3279 used: v2 suites unedited, and a whole-suite route recorder that's identical before and after.
Found by swamp-uat#619's filesystem-v3 backend (PR swamp-uat#623). That suite fails on this bug today, on purpose.
The bug
An interrupted method run's outputs reach the writer's checkout but never the commit log, so clones disagree permanently.
swamp-uat#623's failing test is "swamp model method run killed with SIGTERM|SIGKILL after writing its outputs leaves them complete or absent on every clone [filesystem-v3]":
- on v3, clone A shows the outputs and clone B never does;
- on v2, a SIGTERMed run syncs its outputs before it exits. The marks persist, and the root's push runs on the way out.
Why. The repository writes the output files locally while the run is in progress. Their typed changes are staged into the run's unit, which only commits when the unit ends. When the process is killed, the unit never ends, so nothing is committed, but the local files stay. The writer's checkout now holds files no commit names. Nothing reconciles that, and no other checkout will ever see them.
Goal
- SIGTERM and SIGINT: a run interrupted by a catchable signal ends its units the way decision 5 says for a cancelled run. The record shows
cancelled, or whatever outcome the run reports, and commits before the process exits, so every clone converges. - SIGKILL, or a crash, can't commit anything. Make that state visible and recoverable, not silent.
Design
Signal handling. Trace how a CLI
model method runandworkflow runhandle SIGTERM and SIGINT today:- the abort signal;
- the executor's cancellation;
closeV3Datastoresat teardown (src/cli/mod.ts).
Make the v3 path end the use case and root as cancelled, so the commit-log units commit, before teardown releases the writer lock. Do the same for serve's run cancellation and shutdown drain (
interruptShutdownRuns). Reuse #3257's run-outcome rules: no new commit policy.The SIGKILL residue. Choose one, record why in
datastores.md, and test it:- (a) Detect: a writer's checkout that holds datastore-tier files no commit names is reported by
swamp datastore statusordoctor datastoreson v3, so the user knows the clones disagree; - (b) Reconcile: on the next writing command in that checkout, files the read model doesn't know are either committed as recovered outputs or removed. Removing is safer: the run never completed.
The Phase 4 outbox will make interrupted commits durable. Note that this issue's choice is the Phase 3 answer.
- (a) Detect: a writer's checkout that holds datastore-tier files no commit names is reported by
v2 doesn't change. The SIGTERM path on v2 must behave exactly as today. Run the v2 gate.
Tests
- In-process, on a v3 filesystem datastore (no CLI subprocesses in this repo):
- abort a running method run, and a workflow run, the way SIGTERM does (through the abort signal);
- the record commits as cancelled;
- a second checkout's read model catches up to the same outputs, or the same absence;
- the runs end in the same state they would on v2.
- The SIGKILL residue: a unit that never ends, with its local files written, is detected or reconciled per your choice.
- The v2 gate, as described in Background.
- When this ships, swamp-uat#623's test above must pass against the release.
Out of scope
- The Phase 4 outbox.
- S3 and GCS v3.
Blocked by: nothing. It runs in parallel with the v3 managedConfig and v3 namespace issues.
Shipped
Click a lifecycle step above to view its details.
Sign in to post a ripple.