#1554 s3-datastore: bulk pull writes to cache root instead of the namespace, wedging push and arming a full-datastore delete

Summary

pullFile in @swamp/s3-datastore writes to the bare cache-relative path while reading from the namespaced S3 key. Every bulk pull therefore hydrates the whole datastore into the ROOT of ~/.swamp/repos/<repoId>/, while pushes only ever walk <repoId>/<namespace>/. The cache ends up holding both layouts at once, and the repo can no longer push.

Location

datastores/_lib/s3_cache_sync.ts (~L1518-1548):

const localPath = assertSafePath(this.cachePath, relativePath);          // <cache>/data/...
({ data } = await ... this.s3.getObject(this.dataKey(relativePath), signal)); // <ns>/data/...

Index keys are bare, so pullChanged writes every object to the cache root. Push does the inverse (walks <cache>/<ns>/**, strips the prefix). The local path needs the same dataKey-symmetric namespace prefixing the read side already applies.

Lazy hydration is NOT affected — that hook passes a path already containing the namespace, misses dataKey(), and lands correctly via the bare fallback. Only the bulk path is broken.

Reproduce

On a repo with namespace configured:

1. swamp datastore sync --pull
2. ls ~/.swamp/repos/<repoId>/ -> data/, outputs/, workflow-runs/, ... at root, alongside <namespace>/
3. swamp datastore sync --push -> Cannot push: un-migrated data found at root level
4. swamp doctor datastores -> FAIL

Three separate commands each reproduce it in one invocation: bare swamp datastore sync, sync --pull, and swamp datastore setup extension (which calls a full pullChanged).

Impact: this can delete the entire remote datastore

Worse than blocked pushes. In the full-walk branch, pushChanged computes toDelete as every index entry not present under <cache>/<namespace>/. With the files stranded at the root, that set is effectively the whole datastore. It is gated only on dirtyPathsOverflowed, which flips true when one command exceeds DIRTY_PATHS_CAP = 200 dirty paths.

My repo sat in this state for 8 days with 7431 live S3 objects one boolean away from deletion, plus 348 files of local run history that could not be pushed the whole time. On a different repo, a related index inconsistency did prune ~1456 live objects for real.

The documented recovery does not work

The error text points at swamp datastore namespace migrate, which is a dead end once both layouts exist: findFileCollisions blocks on the overlap, and with the namespace dir non-empty it prints the preview and exits 0 without migrating. doctor --repair returns {"status":"not_needed"} (it only targets foreign-namespace contamination).

migrate is also actively harmful here: it calls invalidateCatalog(), and the next query triggers bulkReplaceNamespace -> DELETE FROM catalog WHERE namespace = ?, rebuilding from local disk and truncating the catalog. That is the mechanism behind the 3-byte .catalog-export.json I have seen reported elsewhere.

What actually repaired it

Out-of-band, bypassing swamp entirely:

1. stop all other writers
2. aws s3 sync s3://<bucket>/<prefix>/<ns>/ <cache>/<ns>/ — download-only, never --delete, excluding .datastore-index.json, _index/*, .namespace.json
3. rm -rf the six root-level trees (they must be removed, not emptied — both the doctor check and the push guard stat isDirectory)
4. swamp datastore catalog pull --namespaces <ns> to restore the truncated catalog
5. push

The excludes in step 2 matter: isInternalCacheFile matches against the cache-relative path, so a downloaded <ns>/_index/... is not filtered and round-trips back as junk index entries.

Suggested fixes

1. Make pullFile namespace-symmetric with dataKey (the actual bug).
2. Never compute a delete set that large from a layout mismatch — treat "index entry exists but local file is wholly absent from the namespace tree" as needs-investigation, not delete.
3. Make namespace migrate handle the both-layouts-present case, or fail with a message that does not recommend itself.
4. Implement previewPush — sync --push currently has no dry-run and no confirmation, so there is no supported way to inspect a push (including its deletes) before it happens.

Environment

• CLI 20260806.001601.0-sha.5bfddaa2 (also present in 20260730.013940.0)
• @swamp/s3-datastore with both prefix and namespace configured
• macOS arm64

Upstream repository: https://github.com/systeminit/swamp-extensions

Environment

• Extension: @swamp/s3-datastore@2026.07.25.1
• swamp: 20260807.031228.0-sha.9a36314e
• OS: darwin (aarch64)
• Deno: 2.8.3
• Shell: /bin/zsh