Relationships
#2723 Install extensions by stage and swap, with a journal and crash recovery
Opened by hammz · 9/29/2026· Shipped 9/30/2026
Summary
applyInstall (swamp-club#2708) writes a new version over the live one with a merge copy: copyDir per kind dir into <pulledRoot>/<name>/ and into each bundle namespace (src/libswamp/extensions/pull.ts ~1098-1195), then orphan pruning. If it fails or the process dies partway, the extension is left with a mix of two versions' files and no record of which is which. There is also no way to undo it: the old files have already been overwritten or pruned.
Replace the merge copy with stage and swap. The new version is fully built in a staging folder, the live roots are moved aside, the new roots are moved in, and the old roots are kept until the install commits. A journal written before anything moves makes a crash at any point recoverable.
This is the first half of unit U4 of the swamp-club#2612 split (see the planning ripples there). It affects every repo and ships on its own. #2612's convergence and swamp-club#2495's adoption both build on it. Depends on swamp-club#2709 (recovery runs under that lock).
Scope
- Roots that are swapped: the extension root
<pulledRoot>/<name>/and each non-empty bundle namespace dir (.swamp/{bundles,vault-bundles,datastore-bundles,report-bundles,webhook-bundles}/<ns>). Skills are not swapped: a skill dir can be shared with the user or another extension, so skills keep today's merge copy andcreatedPaths. - Staging: build the new roots under
<pulledRoot>/.swamp-staging/<uuid>/. Bundle roots stage in a sibling.swamp-staging-<uuid>next to their namespace dir, at least two levels deep, so every rename stays on one filesystem. Every loader, the digest reader,doctor extensions,extension list, and the datastore on-disk scan must ignore.swamp-staging*entries; add a test for each. - Journal first: write
<pulledRoot>/.swamp-staging/<uuid>/journal.jsonbefore creating any staging folder, including bundle staging (ADV-30). It holds: an owner id, the phase, the lockfile path this install writes, the new checksum, the old and new manifest digests, and every root's live, old and new paths. Zod-validated. Containment: every path must be under the pulled root or a bundle dir. The lockfile path must exactly equal one of the resolved lockfile paths, including the non-managedextensions/models/upstream_extensions.json(ADV-32). - Two-phase swap:
- Phase 1 moves each live root to
old/<i>. - Phase 2 moves each new root into place, with
manifest.yamllast. The phase then becomesswapped. - Any failure undoes every completed step in reverse order.
- The lock holder re-checks ownership before each phase. Document that a shared volume mounted by two hosts is not protected (ADV-31).
- On Windows, never rename over an existing dir. The two phases already avoid that; add a Windows-specific test.
- Phase 1 moves each live root to
- Nested entry roots: a scoped name can nest another (
@a/bcontains@a/b/c). Moving@a/b's root must carry@a/b/c's root across unchanged. Keep@a/b/cin place, or copy it into the new root before phase 2, and record the choice in the journal. filesChecksumexcludes nested entry roots. TodayreadInstalledExtensionDigestwalks the whole root (src/infrastructure/persistence/installed_extension_digest_reader.ts:52). Installing@a/b/ctherefore changes@a/b's digest, and@a/blooks locally edited. The digest is unchanged for roots with no nested entries. Note in the PR that entries whose stored digest included a child will reportdriftonce.- Commit handle:
applyInstallreturns with the swap done andold/kept.commit()deletes the staging folder, includingold/. For this issue,installExtensioncommits at the end of apply. swamp-club#2724 moves that to after the catalog save. - Crash recovery:
- Runs under the #2709 lock at the start of every apply, at the start of
rm, and from arecoverPulledExtensionStaging(repoDir)entry point that #2612's convergence will call. - A journal whose owner id is not in this process's active set is orphaned. Recovery uses no pid-liveness checks, because containers reuse pids.
- Roll forward when the journal reached
swappedand the recorded lockfile's entry matches the new checksum. Otherwise roll back. - Decide each root from what is actually in staging (whether
old/<i>and the new root are present), not from the extension root's manifest (ADV-33). Never delete the only copy of a root. - An invalid or out-of-containment journal is left alone, with one warning naming it.
- Stale staging folders with no journal are swept only past an age threshold. Set the folder's mtime explicitly in tests.
- Runs under the #2709 lock at the start of every apply, at the start of
- Behavior change: the swap drops files that are not in the archive. Today's merge copy keeps a user's extra file in
<pulledRoot>/<name>/. The swap removes it. This can only happen with--force: without it, the #126 local-edits guard already refuses when the digest does not match, and any extra file changes the digest. Document this indesign/primitives/extensions.md, and test it. - Orphan pruning of the extension root and bundle roots is replaced by the swap. Keep
prunedin the output, computed from the old and new file sets, and keep pruning for skills.
Tests
- Unit: the swap for first install, upgrade and same-version reinstall produces the same tree and lockfile entry as today's merge copy, when there are no extra files.
- Unit: inject a failure at each rename step, including on the second rename. Each rolls back to the exact prior tree.
- Unit: recovery for a journal at each phase, both roll-forward and roll-back. Also: a root that was never moved (ADV-33), an invalid journal, a journal with an out-of-containment path, a journal owned by an active install (left alone), and an old staging folder with no journal (swept).
- Unit: a nested root survives an upgrade of its parent. The parent's digest ignores the child.
- Unit: an extra file is dropped with
--forceand refused without it. - Unit: loaders, the digest, doctor, list and the datastore scan ignore
.swamp-staging*. - Property (fast-check): for random old/new file sets and a random failure point, the tree after rollback or recovery equals either the old tree or the new tree, never a mix.
- The Windows-specific test from scope item 4.
Docs
design/primitives/extensions.md: the install transaction, the journal, recovery, the extra-file behavior, and the nested-root rule. Update the crash-state recovery section.
Shipped
Click a lifecycle step above to view its details.
Sign in to post a ripple.