#3316 datastore namespace migrate treats the repo-local catalog as user data: it cannot be re-run, resumed, or reversed after a read

Severity: medium — datastore namespace migrate cannot be run twice, cannot finish a migration that stopped part-way, and cannot be reversed once the repo has been read. Each case exits 1 with an error that calls swamp's own catalog index user data. No data is lost and deleting the repo-local catalog files is a workaround, but the errors do not point to it, and the reverse error advises removing the directory that holds the live catalog.

Affects: swamp 20261012.194115.0-sha.9b7f54f4 (the four UAT tests below fail on it). First written up against 20261009.215038.0-sha.986f9022. The code cited is as described on source main at 3b7bb2ee.

Summary

The catalog _catalog.db is repo-local: it always lives at .swamp/data/ and is not part of the namespaced layout. Namespace migration nevertheless treats it, and its SQLite sidecar files, as datastore content. That shows up as three failures, through two separate checks:

A and B are the same defect reached from two starting states. C is a second defect in the same area. A fix needs both sites: adding _catalog.db-shm to INFRASTRUCTURE_FILES fixes only C, because the forward check never reads that set; making the forward check skip INFRASTRUCTURE_FILES fixes A and B only, and only fully if -shm is added too.

Steps to reproduce

A. Second migrate after a successful one

mkdir /tmp/ns-migrate-twice && cd /tmp/ns-migrate-twice
swamp init
swamp model create command/shell m1
swamp model method run m1 execute --input run='echo hi'
swamp datastore namespace set half-ns
swamp datastore namespace migrate --confirm ; echo "first: $?"    # 0
find .swamp -maxdepth 3 -name '_catalog*'
swamp datastore namespace migrate ; echo "second (preview): $?"   # 1
swamp datastore namespace migrate --confirm ; echo "second (confirm): $?"  # 1

B. Resume after a partial migration

A plain file is placed where outputs has to go, so the rename of the fourth directory fails; the file is then removed and migrate is retried.

mkdir /tmp/ns-resume && cd /tmp/ns-resume
swamp init
swamp model create command/shell m1
swamp model method run m1 execute --input run='echo hi'
swamp datastore namespace set half-ns

touch .swamp/half-ns/outputs                      # make the 'outputs' rename fail
swamp datastore namespace migrate --confirm; echo "exit=$?"   # exit=1, partially migrated
rm .swamp/half-ns/outputs                         # obstacle removed

swamp datastore namespace migrate --confirm; echo "exit=$?"   # exit=1, should be 0
ls .swamp .swamp/half-ns

C. Reverse migration after a read

mkdir repro && cd repro && swamp init

mkdir -p extensions/models/test
# Any model that writes data will do. This used the swamp-uat fixture
# tests/cli/fixtures/extensions/models/test/bulk-producer.ts
cp <swamp-uat>/tests/cli/fixtures/extensions/models/test/bulk-producer.ts extensions/models/test/
swamp model create @test/bulk-producer seed --global-arg count=2 --global-arg sizeBytes=64
swamp model method run seed produce

swamp datastore namespace set ns1
swamp datastore namespace migrate --confirm      # exit 0, 9 directories moved
ls -a .swamp/data                                # _catalog.db

swamp data list seed                             # exit 0
ls -a .swamp/data                                # _catalog.db _catalog.db-shm _catalog.db-wal

swamp datastore namespace unset --migrate --confirm ; echo $?   # exit 1, expected 0
swamp datastore namespace migrate --reverse --confirm ; echo $? # exit 1

# Showing the sidecar is the cause:
rm .swamp/data/_catalog.db-shm
swamp datastore namespace unset --migrate --confirm ; echo $?   # exit 0

Expected behaviour

• A: the second and later runs exit 0 with nothing to move.
• B: the retry exits 0 and moves the directories the first run did not reach, leaving .swamp/data/_catalog.db alone.
• C: namespace unset --migrate --confirm exits 0, moves the subdirectories back from .swamp/ns1/ to .swamp/, and removes the namespace from .swamp.yaml.

What establishes it:

• design/enablers/datastores.md:350-358: "Two things are not namespaced: The _catalog.db catalog is repo-local at {repoDir}/.swamp/data/". The catalog is not datastore content that should be migrated into, or collide inside, a namespace, and that directory is expected to exist and hold the catalog after a forward migration.
• design/primitives/data.md:377: "Catalog rows are derived state. Deleting _catalog.db loses no local data". The forward error's claim that the files "would be lost during merge" is not true of these files.
• design/enablers/datastores.md:343-348: --reverse "refuses if the un-namespaced path already holds data files". The catalog and its sidecars are not data files.
• src/libswamp/datastores/namespace_migrate.ts:98-102 already defines INFRASTRUCTURE_FILES so that the catalog is ignored when deciding whether a path holds data. The catalog runs in WAL mode (src/infrastructure/persistence/catalog_store.ts, PRAGMA journal_mode=WAL), and SQLite in WAL mode keeps a -shm file beside the -wal.
• Commit d9afef4c (#2104), "fix(datastore): make namespace migrate idempotent when nothing to move", states: "The command is now safe to run unconditionally as an idempotent operation".
• The command reports a partial result itself ("Partially migrated: 3 directory(s) moved before failure") and has no rollback, so re-running it is the only way the CLI offers to finish. No design doc names resume-after-failure explicitly.

Actual behaviour

A

The first migrate succeeds (exit 0, "Migration complete: 8 directory(s) moved", "Catalog invalidated — will rebuild on next access"). Immediately afterwards:

$ find .swamp -maxdepth 3 -name '_catalog*'
.swamp/data/_catalog.db
.swamp/half-ns/data/_catalog.db-shm
.swamp/half-ns/data/_catalog.db-wal
.swamp/half-ns/data/_catalog.db

$ swamp datastore namespace migrate

Error: Cannot migrate: 1 file(s) already exist at the destination and would be lost during merge. Remove or rename the conflicting files first:
  data/_catalog.db
$ echo $?
1

--confirm and --json give the same error and exit 1. In the UAT run the same failure lists three files (data/_catalog.db-shm, data/_catalog.db-wal, data/_catalog.db).

B

First run (obstacle in place), exit 1:

  ✓ definitions-evaluated
  ✓ workflows-evaluated
  ✓ data

Partially migrated: 3 directory(s) moved before failure.
  Succeeded: definitions-evaluated, workflows-evaluated, data
  Failed: outputs

Error: Failed to migrate "outputs": Not a directory (os error 20): rename '.../.swamp/outputs' -> '.../.swamp/half-ns/outputs'

After it, .swamp/data/ exists again and contains only _catalog.db. Retry (obstacle removed), exit 1:

Error: Cannot migrate: 1 file(s) already exist at the destination and would be lost during merge. Remove or rename the conflicting files first:
  data/_catalog.db

ls .swamp still shows audit data outputs secrets vault-bundles workflow-runs beside half-ns; .swamp/half-ns holds only data definitions-evaluated workflows-evaluated. In this state swamp model output search --json returns "results": [] although the method run's output is still on disk under .swamp/outputs.

C

$ swamp datastore namespace unset --migrate --confirm

Error: Cannot reverse-migrate: "data" already exists at "<repo>/.swamp/data". Remove or rename it first.
$ echo $?
1

.swamp/data at that moment contains exactly _catalog.db, _catalog.db-shm and _catalog.db-wal. Nothing is moved and .swamp.yaml still has namespace: ns1. swamp datastore namespace migrate --reverse --confirm prints the same error and exits 1. After rm .swamp/data/_catalog.db-shm, the same command prints Reverse migration complete: 9 directory(s) moved and exits 0.

Workaround

For A and B: rm .swamp/data/_catalog.db*, then migrate exits 0. The failure returns once the catalog is recreated at .swamp/data/. For C: rm .swamp/data/_catalog.db-shm.

Root cause

Lines as on source main at 3b7bb2ee.

Site 1 — the forward collision check (A and B)

1. The forward path renames each whole subdirectory, so .swamp/data/ moves to .swamp/<ns>/data/ including _catalog.db and its -shm/-wal files (renameDir, src/libswamp/datastores/namespace_migrate.ts:405), where nothing reads it.
2. deps.invalidateCatalog() then runs, on success (namespace_migrate.ts:438) and on the failure path (:422). The CLI wiring implements it by opening a catalog store at the repo-local path (createCatalogStore(repoDir) then invalidate(), src/cli/commands/datastore_namespace.ts:404-411), which recreates .swamp/data/_catalog.db at once.
3. On the next run .swamp/data exists as a source and .swamp/<ns>/data as a destination, so the collision check runs (namespace_migrate.ts:250-259). findFileCollisions (src/infrastructure/persistence/directory_merge.ts:24-62) skips only extension staging entries; it does not skip INFRASTRUCTURE_FILES. The two catalogs differ byte for byte, so compareFiles reports them non-identical and the command returns "Cannot migrate" (namespace_migrate.ts:297).

The collision check was added in 0496ad5b (#1553, "prevent data loss when mergeDirInto encounters file collisions"). The idempotency fix d9afef4c (#2104) only covers the case where no un-namespaced source directory exists at all, which is not the state the command itself leaves behind. Whether the catalog should be left behind when data is moved, or skipped by the collision check, was not determined.

Site 2 — the reverse "already holds data" check (C)

• src/libswamp/datastores/namespace_migrate.ts:98-102 defines

  export const INFRASTRUCTURE_FILES = new Set([
    "_catalog.db",
    "_catalog.db-journal",
    "_catalog.db-wal",
  ]);

  _catalog.db-shm is missing.

• dirHasDataFiles (src/cli/commands/datastore_namespace.ts:390) returns true for any directory entry whose name is not in that set. It is the set's only consumer.

• namespace_migrate.ts:235-241 turns that into the Cannot reverse-migrate refusal for any subdirectory not in MERGEABLE_ON_REVERSE.

The set was introduced in e3a85856 (#1515) with these three names and has not changed since. Reverse migration directly after a forward migration passes only because the migrate command's own catalog invalidation leaves a lone _catalog.db; the sidecars appear on the next command that opens the catalog.

No fix was attempted.

How the UAT found it

All four tests were added by the swamp-uat test audit and fail against 9b7f54f4. Before the audit, the only forward test ran one migrate --confirm from a clean layout, which never has a destination data directory; the only reverse test ran set, migrate and unset --migrate back to back with no command in between, so .swamp/data held only _catalog.db.

The B test produces the half-migrated state by moving definitions-evaluated, workflows-evaluated and data into the namespace directory and running data list once, then requires migrate --confirm to exit 0 and move the rest. It fails with "Cannot migrate: 2 file(s) already exist" naming data/_catalog.db-shm and data/_catalog.db-wal.

Run them from the swamp-uat repo (branch test-investigation):

deno test --allow-read --allow-write --allow-env --allow-run --allow-net \
  tests/cli/adversarial/namespace_isolation_test.ts tests/cli/datastore/namespace/migrate_test.ts \
  --filter '/second time|interrupted after|after the namespaced data was read|again after a migration/'