#2612 managedConfig: store pulled extension archives in the config tier and converge each checkout to the shared lockfile

Triage and planning record (1 of 4): overview and proposed split

Triage. Classified as a feature (high confidence). The #2429 prerequisites are merged (PR #2676): ConfigPoller reloads on the tier lockfile hash, and the CLI extension writers push by path through pushManagedConfigPathsDeferred. The code references in the issue body were checked and are accurate.

Found beyond the issue text:

• The channel is dropped on restore in two places, not one: createExtensionInstallDeps (src/cli/create_extension_install_deps.ts:92-103) and the auto-resolver context (src/cli/auto_resolver_adapters.ts:289-302).
• installExtension spreads the parent context into dependency installs (src/libswamp/extensions/pull.ts:1421), so the parent's expectedChecksum leaks into any dependency missing from the lockfile.
• extractTarGz / listTarGzEntries have no decompressed-size cap, and there is no archive byte-limit constant anywhere (the nearest is the safety analyzer's 10 MB source limit).
• SCOPED_NAME_PATTERN is duplicated in 7 places and is module-private in extension_manifest.ts.
• The intro of "Managed Config Deployment Architecture" in design/enablers/datastores.md already claims sources live in the tier, contradicting the rest of the section.
• LockfileRepository serves reads from a construction-time snapshot, so any lock-based design needs an explicit refresh.

Planning. An 18-step plan went through four adversarial review rounds (v1 to v4; 36 findings, all blocking ones resolved). It is too large for one change, so it is being split. Ripples 2 and 3 record the v4 design; ripple 4 records the open findings and the decisions the reviews forced.

Proposed split (each unit ships on its own and does not break existing users):

Each unit carries its own design-doc updates. The seven follow-ups listed in the issue body are filed once U5 to U7 land. U1 is being filed now as its own issue.

Triage and planning record (2 of 4): plan v4, groundwork (steps 1-8)

1. Archive identity and interfaces. Export SCOPED_NAME_PATTERN. Add an ExtensionArchiveKey value object (scoped name plus exactly 64 lowercase hex characters; create() rejects anything else). Add ports ExtensionArchiveSource.read(key, signal) and ExtensionArchiveSink.store/delete, each returning the paths touched. Add MAX_EXTENSION_ARCHIVE_BYTES and MAX_EXTENSION_ARCHIVE_DECOMPRESSED_BYTES; push checks the same compressed constant so they cannot drift.
2. Decompressed-size cap on both tar passes, as one byte-counting stream option on listTarGzEntries and extractTarGz. Also close the two file handles at pull.ts:929 and 944.
3. TierExtensionArchiveStore over <config base>/pulled-archives/<name>/<sha256>.tar.gz. Paths are built only from a validated key and contained with assertSafePath. Writes are atomic and idempotent. Reads refuse oversize files and re-hash the bytes; a mismatch is a tamper warning. A local miss hydrates the one archive, which needs a HydrateFileHook variant that forwards a signal, plus a Promise.race deadline. Archives live under config/, so config pulls bring them: a fresh pod downloads everything once, and later pulls fetch only new archives.
4. Per-checkout pulled-extensions lock: an in-process keyed mutex plus a cross-process FileLock on .swamp/pulled-extensions.lock, with a tryAcquire(). Reentrancy uses an AsyncLocalStorage lease cleared in finally, so an auto-resolve inside a locked section runs inline, but cached promises resolved later cannot bypass the lock. installExtension is split into an unlocked prepare (fetch, verify, extract, safety analysis) and a locked apply (refresh the lockfile snapshot, stage, swap, writeEntry, dependencies, phase 8, commit). The lock wraps InstallExtensionService.execute, the direct-caller wrapper and RemoveExtensionService.execute, so every entry point holds it. It is never held across the interactive conflict prompt. Lock order: this lock first, the lockfile advisory .lock innermost. LockfileRepository.refresh() is called at the start of apply.
5. installExtension archive source and record policy. InstallContext gains archiveSource, archiveSink, record (always / if-missing-checksum / never, default always) and mode: 'converge'. An archive hit skips the registry calls, and requires the manifest name to match and, for an exact pin, the version. The record policy gates writeEntry, the sink, dependency recursion and orphan pruning. Only archives whose registry checksum came back non-null and matched may be stored. Dependencies no longer inherit expectedChecksum or the overrides.
6. Stage, swap, commit. A staged swap under <pulledRoot>/.swamp-staging/<uuid>/ replaces the merge copy. Bundle folders get a sibling .swamp-staging-<uuid> at least two levels deep. A journal is written before any staging folder is created; it holds the owner id, phase, the lockfile path this install writes, the new checksum, and old/new manifest digests and paths. Phase 1 moves live roots to old/<i>; phase 2 moves new roots in, manifest.yaml last, and the phase becomes swapped. Any failure after the swap rolls back the install and its dependencies. A PendingInstall handle has commit() (delete old/, store the archive, delete the superseded archive) and rollback() (reverse the swap, and restore the prior entry with channel and pulledAt only if writeEntry ran). The service commits after saveAll, rolls back on DuplicateTypeError, and commits then rethrows on other phase-8 errors. filesChecksum excludes nested entry roots. The swap drops local files that are not in the archive; this is documented.
7. Crash recovery. The journal is Zod-validated with containment checks. Recovery runs under the exclusive lock and treats a journal whose owner id is not in this process's active set as orphaned; there are no pid heuristics. It rolls forward when the journal reached swapped and the recorded lockfile's entry matches the new checksum; otherwise it rolls back. It never discards the only copy of a root. It runs at the start of every apply, at the start of rm, and in the convergence gate.
8. Keep the channel on restore in createExtensionInstallDeps and the auto-resolver.

Triage and planning record (3 of 4): plan v4, archives and convergence (steps 9-18)

1. Archive wiring for every writer. One factory, tierArchiveStoreFor(repoDir, marker, lockfilePath), returns a store only when managedConfig is active, the base is resolved and the lockfile is the tier lockfile (the #445 exemption never stores archives). It is passed through the shared factories: extensionPull (pull, search install, doctor repair), libswamp createInstallContext (update), createExtensionInstallDeps (install, serve install, repo upgrade), and the auto-resolver. The auto-resolver also gets the archive source, and uses if-missing-checksum for pinned entries. Archive paths go to the publish call each caller already uses (pushManagedConfigPathsDeferred, pushManagedLockfileIfChangedDeferred with a new extraPaths argument, serve's markExtensionChanges). The pinned-writer rule test checks for this.
2. Superseded archives are deleted in commit() (for rm, after the lockfile write), only when the old and new checksums differ. Serve propagates the deletion. The CLI on S3/GCS does not, because its first push in a process is a full walk that skips deletion detection (#2273); the leftover is unreferenced and harmless. This is documented.
3. ConvergenceState in .swamp/pulled-extensions-state.json, Zod-validated: config base, lockfile hash, pending entries with backoff (1 minute to 1 hour), unrestorable entries, backfill failures, lastVerified, restoreGeneration, and a warn-once digest. Invalid state means a full pass. A failed write is one warning.
4. extension install restores from archives first, then the registry. It records always for legacy migration or missing checksums, otherwise if-missing-checksum. It backfills missing archives (exact version from the on-disk manifest for constraint pins, verified against the entry checksum and a non-null registry checksum). The output gains a per-entry source and a top-level backfilled list, both additive.
5. convergePulledExtensions reads only the tier lockfile, the archive source and the state, and never throws. The gate: recover orphaned journals; skip when the lockfile is absent or unparseable; full pass when the config base changed; skip when the hash matches and nothing is due. Converge mode uses record never, skillsDirs: [], and no registry functions. It checks only files under <pulledRoot>/<name>, excluding skills, bundles and nested roots, and never migrates. Restore rule: files missing, OR an exact-pin version skew, OR (checksum differs from lastVerified AND the digest differs from filesChecksum). A digest match records lastVerified. Same-version drift is a warning. Calls in one process share a single in-flight run created outside any lock lease; across processes it uses the try-lock and reports busy.
6. CLI wiring in requireInitializedRepoReadOnly, requireInitializedRepo (after the coordinator pull) and requireInitializedRepoUnlocked, plus a skipConvergence option. It also runs in configureStartupExtensions before reconcile and the missing-files check, except for thin-client commands and when local file checks are off. withLoaderContext is the fallback.
7. Serve wiring. Convergence runs in pullManagedConfigAtBoot and on the no-sync-service boot path, before the boot hash and registry loads. The poller converges every poll. A busy result delays the hash-change reload until a non-busy pass. It reloads when the hash changed or restoreGeneration moved. ServeReloadResponse is unchanged.
8. Tests. In-process integration tests with two checkouts using different marker tools on a shared filesystem datastore; serve on a filesystem datastore; the extension-backed bootstrap case; a runtime contract test that convergence never calls the registry or writeEntry.
9. Docs: datastores.md, serve.md, extensions.md (including the wrong sha256- example), architecture.md, remote-execution.md, and the "until #2612" notes in code and tests.
10. File the issue's seven follow-ups.

Triage and planning record (4 of 4): review decisions and open findings

Decisions forced by the adversarial reviews (carry these into the split issues):

• Lock at the install/remove service boundary and hold it through commit, not around whole commands. A command-level lock deadlocks against the auto-resolver, which resolveManagedLockfileForWrite can trigger, and would be held across the conflict prompt.
• Take the lock and the archive sink in the shared factories, so search install, doctor repair, repo upgrade and serve install are not missed.
• Converge mode must ignore skill paths: entry.files records the marker tool of whoever pulled, so checkouts on different tools would otherwise re-swap forever. It must also never run legacy migration, which deletes files.
• The if-missing-checksum record policy breaks legacy-layout migration (install.ts:254-262 relies on the rewritten files[]); use always there.
• Pending entries need backoff and an unrestorable bucket, or every CLI command pays a network round trip.
• Recovery must not rely on pid liveness: containers reuse pids. Use owner ids plus the exclusive lock.
• Restoring whenever the checksum differs from lastVerified overwrites same-version edits on first run; also require the digest to differ.
• The lockfile snapshot cache makes "re-read under the lock" a no-op without refresh().

Open findings on v4 (not yet folded in; the U3, U4 and U7 issues should address them):

• ADV-32: journal lockfile containment rejects the non-managed lockfile at extensions/models/upstream_extensions.json; accept exact matches of the resolved lockfile paths.
• ADV-33: decide each root's recovery from the staging layout (old/new presence), not from the extension root's manifest verdict, or a root that was never moved can be lost.
• ADV-34: the restoreGeneration reload trigger needs the same baseline and failure cap as the hash trigger.
• ADV-35: a busy converge should defer every reload, and the generation should bump only after commit, recovery or rollback.
• ADV-36: RemoveExtensionService.execute needs refresh() under the lock.
• ADV-29 (low): pass the poller's stop signal into convergence so shutdown is not held by a hydrate.
• ADV-30 (low): write the journal before creating bundle staging folders.
• ADV-31 (low): the lock holder should re-check ownership before each rename phase; document the shared-volume limitation.

U1 (restore-path correctness: channel dropped on restore, and the parent's expectedChecksum leaking into dependency installs) is filed as swamp-club#2639. It has no dependencies and will be triaged on its own.

Finer split (2026-09-29)

U2 and U3 are now filed as three sub-issues. Each ships on its own and affects every repo, not just managedConfig ones:

Corrections to the plan found while filing:

• The dependency-override leak is already covered: U1 (#2639) clears expectedChecksum for dependencies at pull.ts:1430, and the other overrides do not exist yet.
• FsFile.readable closes on stream end or cancel, so #2707 confirms whether the flagged handles actually leak before changing them.
• Lock order: datastore global lock → pulled-extensions lock → lockfile .lock. The auto-resolver runs inside commands that already hold the global lock, so swamp-club#2495 must take the global lock before the pulled-extensions lock, never inside apply.

Remaining units, to be filed later:

• U4 → D (stage-and-swap with journal and crash recovery) and E (commit/rollback handle across the catalog save).
• U5 → F (tier archive store plus every writer storing archives), G (delete superseded archives) and H (extension install restores from archives, plus backfill).
• U6 → I (CLI convergence).
• U7 → J (serve boot convergence) and K (per-poll convergence and reloads).

swamp-club#2495 comes next after #2709. It should be split into (a) hydrate-before-write and scoped publish, (b) the auto-resolver writing the shared lockfile, (c) adoption (after D), (d) legacy-lockfile retirement, and (e) the datastore setup extension overwrite. Its (a) and (b) should land before I.

U4 is filed as swamp-club#2723 (D: stage-and-swap install with a journal and crash recovery; covers ADV-30 to ADV-33, the staging sweep, nested roots, and the filesChecksum fix) and swamp-club#2724 (E: commit/rollback install handle across the catalog save). Order: #2708 → #2709 → #2723 → #2724. Found while filing: today a v1→v2 upgrade that hits DuplicateTypeError is rolled back to no version at all, because createdPaths includes the files that overwrote v1 and v1-only files were already pruned. Also, filesChecksum walks nested entry roots, so installing @a/b/c makes @a/b look locally edited.