Relationships
#1271 `datastore setup filesystem` never transfers content when leaving a sync-based (remote) datastore — copies only the catalog and hardcodes `filesPulled: 0`
Opened by kneel · 7/19/2026· Shipped 7/23/2026
Component
src/libswamp/datastores/setup.ts (datastoreSetupFilesystem), dispatched from src/cli/commands/datastore_setup.ts.
Severity
Medium — no data loss, but a silent incomplete migration. The filesPulled: 0 reads like success.
Summary
swamp datastore setup filesystem is handled by a dedicated datastoreSetupFilesystem function that does a pure local-disk copy of <repo>/.swamp into the destination and returns filesPulled: 0 unconditionally. It constructs no provider and no sync service for either side, so when the repo is currently on a remote sync-based datastore (content in the backend + the ~/.swamp/repos/<repoId> cache, not under <repo>/.swamp), the content is never read and never relocated. The operation reports success with only the catalog moved.
The repo bootstrap may hydrate the source cache as a side effect (running a pullChanged against the outgoing datastore), but datastoreSetupFilesystem copies from <repo>/.swamp, not from that cache, so the hydrated content still is not transferred.
Nothing is lost — content remains in the outgoing backend and in the local cache — but the switch does not transfer it.
Repro
- Set up a repo on a sync-capable remote datastore; populate it (push content so the remote holds N blobs).
swamp datastore setup filesystem.- Observe
filesPulled: 0and that the new filesystem datastore contains only catalog metadata, not the N content files.
Expected
Leaving a sync-capable datastore should relocate the content (from the hydrated cache / outgoing backend) into the new filesystem datastore — or at least refuse with a clear message and a documented manual procedure — rather than silently completing with a hardcoded filesPulled: 0.
Suggested direction
In datastoreSetupFilesystem, when the outgoing datastore is remote/sync-based, copy the hydrated working copy from the source cache (~/.swamp/repos/<repoId>) into the destination instead of (or in addition to) <repo>/.swamp, and report the real transferred count rather than a hardcoded 0. Alternatively, detect the sync-capable → filesystem transition and emit an actionable error pointing at the manual off-ramp.
Workaround
Manual: hydrate the cache with swamp datastore sync --pull, then cp -a ~/.swamp/repos/<repoId>/. <new-filesystem-datastore-path>/ after the switch, then verify before decommissioning the remote tables.
Discovered while
Building @shrug/rethinkdb-datastore (a sync-based remote datastore) and exercising the off-ramp path during test-drive. Draft write-up and workaround live in that package's docs/off-ramp.md.
Shipped
Click a lifecycle step above to view its details.
kneel commented 7/20/2026, 6:56:15 PM
Another repro path for the same root cause — and it's not limited to leaving a sync-based remote; it also bites filesystem → filesystem when you try to move an external filesystem datastore back into the repo.
Second repro (filesystem → filesystem, same-path collision)
- Repo currently on a filesystem datastore at an external path (mine:
/mnt/swamp/, same local xfs volume — not a remote backend). Has real content: namespaced data dirs, 684 KB catalog, 4 models visible viaswamp model list. swamp datastore setup filesystem --path <repo>/.swamp(to relocate back into the repo).- Every file errors:
Failed to copy _catalog.db: Source and destination paths refer to the same file: copy '<repo>/.swamp/data/_catalog.db' -> '<repo>/.swamp/data/_catalog.db'. Same for_catalog.db-wal,_catalog.db-shm,audit/commands-*.jsonl. - Result:
filesCopied: 0,bytesCopied: 0,directoriesMigrated: [], config rolls back to the external path. The actual external data at/mnt/swamp/is never even touched as a source.
What this adds to #1271
- Scope: the hardcoded-source bug isn't "sync-based remote → filesystem only." It's "any case where the outgoing datastore's real path isn't
<repo>/.swamp." When outgoing is remote/sync-based, it silently copies an empty skeleton (this issue). When outgoing is a different filesystem path that happens to collide with the target, it hard-errors and rolls back. Both are the same defect:datastoreSetupFilesystemresolves the source from a hardcoded path instead of the current datastore's actual location. - Symptom axis: the existing repro produces a silent incomplete migration (config switches, content missing). This one produces a loud failure (config rolls back). The loud variant is arguably more recoverable for users because it self-blocks — but the silent variant is the scarier default because it looks like success.
- Workaround (matches #1271's direction, adapted for filesystem):
rsync -a --delete /mnt/swamp/<dir>/ <repo>/.swamp/<dir>/for each subdir indatastore.directories, verify checksums, thenswamp datastore setup filesystem --path <repo>/.swamp --skip-migrationto flip config. Same shape as the existing workaround but no cache hydration step needed (content already on disk at the outgoing path).
swamp version hit: 20260716.194319.0-sha.df9fc306, Linux.
Sign in to post a ripple.