Mongodb Datastore

2026.09.01.2

Merges upstream keeb 2026.08.19.2 into the @magistr fork, keeping two fork guards the upstream tree does not carry and fixing a third defect found by live testing.

Fixed: the incremental push was inert (pre-existing, also on 2026.09.01.1)

Core calls markDirty and hydrateFile with a path that ALREADY carries the namespace (<ns>/data/...), while tierRoot scopes only the LOCAL side — every remote _id, the local walk and isSecretsPath speak tier-relative paths. swamp-club#1554 records the same asymmetry for @swamp/s3-datastore's lazy-hydration hook.

One prefix broke three things: the dirty journal recorded paths no walk could match, so ordinary writes NEVER reached the remote and only a full walk persisted; isSecretsPath/isExcludedPath stopped matching, so the vault tier was no longer filtered out of the journal; and hydrateFile looked up an _id no remote doc carries.

toTierRelative() normalizes at both hooks, before the secrets check. It strips only when the remainder begins with a real tier directory, so a tier-relative path is never mangled. Measured live: a namespaced repo held 37 local files and 0 remote paths; after the fix, live paths went 70 -> 100 -> 144 over two runs.

Two fork guards kept, NOT inherited from upstream

• Path-traversal confinement. Upstream composes `${cachePath}/${relPath}` by raw concatenation. One database is shared by every repo and tenant (isolation is only a collection prefix), so a remote _id is untrusted: a doc named ../../../../.ssh/authorized_keys would be written through — or unlinked when carrying deletedAt. resolveWithinCache/isSafeRelPath restored and re-applied to all five merged write sites.
• Namespace tier-root scoping. Upstream roots reads/writes at the bare cache path. Core reads the tier one segment deeper, so that builds a second, invisible tier the reader never sees; push then refuses with "un-migrated data found at root level" and namespace migrate cannot recover (swamp-club#1458/#1554). Worse, pushChanged tombstones every remote path absent from its local walk — rooted wrongly, that walk finds nothing and one push tombstones the whole namespace. tierRoot() restored in config.ts and applied in createSyncService.

Both carry a comment naming them as fork guards so a later merge cannot quietly drop them again.

From upstream

Dirty tracking rides an append-only journal with ancestor coalescing (O(1) markDirty); push reconciles dirty roots in batches rather than one round-trip each. Sidecar becomes an interned getSidecar() plus reconcileWatermark. Blob docs gain createdAt for the orphan sweep's grace window. Adds @magistr/mongodb-datastore/maintenance and the datastore-maintenance workflow — inventory, sweep orphaned blobs and aged tombstones, compact — over every namespace in one execution. manifest.yaml gains models:/workflows:; quality.yaml's methods suite moves na -> present.

Tests

89 across seven suites. Master's four newer suites are KEPT, not replaced by upstream's — the branch predates them and would have deleted them. New tests pin the guards' WIRING, not just their helpers: every prior test exercised tierRoot() and resolveWithinCache() as functions, which is exactly how the upstream merge regressed both while all 17 stayed green. Each new test is mutation-verified.

2026.09.01.1

Re-release of 2026.08.31.1, whose publish never reached the registry — the extension-publish job only fires on a version bump, so a version that fails to publish stays stranded until the next one. Content is unchanged from 2026.08.31.1; its notes are carried forward below so the registry keeps a full record.

Fixed

• Sync tier is now rooted at the namespaced cache path. Swamp core hands createSyncService the bare cache path (the repoId-keyed default, since resolveCachePath returns undefined as it does for every remote datastore) but reads and writes the tier through DefaultDatastorePathResolver.datastorePath(), which prepends the namespace as the outermost segment — {cache}/{namespace}/data/.... The sync service walked and wrote the bare path, so every sync --pull built a second, invisible copy of the whole tier at the cache root. Artifacts stayed queryable through the catalog while swamp data get reported "Data not found"; sync --push then refused with "un-migrated data found at root level", and datastore namespace migrate cannot recover once both layouts exist.

  Same defect fixed in @swamp/s3-datastore for swamp-club#1458 and #1554. S3 was less exposed because its remote keys embed the namespace, so the segment round-trips; this extension partitions by collection prefix (t_<tenant>_r_<ns>_*) with tier-relative _ids, so nothing reintroduced it.

  tierRoot() scopes the local side only — remote _ids stay tier-relative, so existing MongoDB collections need no migration. The sidecar moves alongside the tier it describes; a cold-start sidecar has lastPulledAt: null, which already suppresses the reconciliation tombstone pass in fullWalkPush, so the first push after upgrading uploads without deleting. An empty namespace returns the bare cache path, byte-identical to a non-namespaced repo.

  Upgrading does not repair a cache that is already split. Merge the root tier into {cache}/{namespace}/ (rsync -a --ignore-existing, version directories are immutable so a merge is safe), remove the root tier directories, then push.

• Pull no longer writes or deletes outside the cache root. pullChanged took its local target straight from a remote path doc's _id and interpolated it into ${cachePath}/${_id} for the write, the hash pre-check, and the deletedAt unlink — with no containment check. One MongoDB database holds every repo's tier and isolation is only a collection prefix, so an _id is untrusted input: a doc named ../../../../.ssh/authorized_keys was written through, and with deletedAt set, unlinked, anywhere the process could reach. resolveWithinCache() now rejects absolute paths, ./.. segments, empty segments, backslashes and NUL at every join. Unsafe docs are skipped at admission (with a warning) rather than thrown on, so one planted document cannot wedge every future pull.

• deno task test now grants the permissions its own suite needs. The task was bare deno test, so all 17 sidecar tests failed with NotCapable: Requires write access to <TMP> — the suite could never pass in CI. Now runs with --allow-read --allow-write --allow-env.

Added

• config_test.ts covering tierRoot: namespaced paths, solo mode (empty namespace), whitespace-only namespace, trailing-slash normalisation, composition with a tier-relative path, and that the namespace used for collection prefixing is never mutated.
• sync_adversarial_test.ts — path traversal, absolute paths, backslash/NUL injection, secrets-tier boundaries, and regex-state reuse.
• sync_coverage_test.ts — regression cover for the load-bearing guards: the lastPulledAt !== null && !lazyPullActive gate that keeps a cold-start push from tombstoning the datastore, and the tier-root/sidecar scoping.
• sync_property_test.ts — fast-check properties (gated by FC_NUM_RUNS) for tier placement and the containment invariant: an accepted path can never resolve outside the cache root, for arbitrary remote input.
• quality.yaml, and this changelog.