Relationships
#1557 swamp serve writes outputs/ outside the bound namespace — silently never persisted to the remote datastore
Opened by sntxrr · 8/7/2026
Summary
Under swamp serve with a namespaced remote datastore, method and workflow outputs/ are written to the datastore cache root instead of the bound namespace. Push only walks <cache>/<namespace>/**, so those outputs are silently never persisted to the remote. data/ is unaffected and syncs correctly, which is what makes this so easy to miss — the datastore looks healthy, run history looks fine, and only the outputs quietly stop arriving.
On an ephemeral or recreated container this is permanent, silent data loss.
Evidence
A containerised swamp serve running one workflow on a */15 schedule (4 outputs/hour). Counting outputs/ objects that reached the remote, by day:
2026-07-23: 2 <- namespace bound 2026-07-30
2026-07-24: 4
2026-07-26: 19
2026-07-27: 30
2026-07-29: 2
2026-07-30: 4
2026-07-31: 0 <- 96 produced locally
2026-08-01: 0 <- 96 produced locally
2026-08-02: 0 <- 96 produced locally
2026-08-03: 0 <- 96 produced locally
2026-08-04: 0 <- 96 produced locally
2026-08-05: 0 <- 96 produced locally
2026-08-06: 0 <- 96 produced locallySeven consecutive days, ~670 outputs, zero delivered. Cache state at the point of discovery:
tree cache ROOT cache <ns>/ remote
data 7861 7593 7868 <- fine
outputs 812 4 128 <- broken
workflow-runs 1515 759 773outputs/ had 812 at root vs 4 in the namespace. The stranded files are ordinary method outputs, e.g. outputs/<pkg>/<model>/sync/<instance>-2026-08-01T00-30-00-037Z.yaml.
The onset lines up exactly with the namespace being bound. Before that the repo was solo-layout and outputs reached the remote normally.
Mechanism
The entrypoint runs swamp datastore setup extension @swamp/s3-datastore --namespace <ns> ... on every container start. swamp serve writes outputs/, workflow-runs/, definitions-evaluated/ and workflows-evaluated/ into the repo-local .swamp/ directory during operation:
/data/repo/.swamp/outputs/<pkg>/<model>/sync/*.yaml <- accumulates while serve runsOn the next start, datastore setup relocates those into the cache root, reported as:
Datastore Setup Complete
Files: 969 copied (3.4MB)
Dirs: definitions-evaluated, workflows-evaluated, data, outputs, workflow-runs
Relocated: ... now resolve under /home/[REDACTED]/.swamp/repos/<repoId>
Moved from: /data/repo/.swampThat destination is outside the bound namespace, so push never reads it. The files are neither pushed nor reported as skipped in a way an operator would notice.
Why this is worse than it looks
- Silent. No error, no warning.
swamp doctor datastoresdoes flag "un-migrated data found at root level", but that message points atswamp datastore namespace migrate, which cannot fix it (see below), and it reads as a stale-leftovers problem rather than "your outputs are not being saved". data/still works, so every obvious health check passes.- Unrecoverable on ephemeral compute. The stranded files live only on the container's volume. Anyone running
swamp serveon k8s/ECS/a recreated container loses all execution outputs with no indication it happened. - The suggested remedy is unavailable.
datastore namespace migraterequires the namespace dir to be effectively empty; with both layouts populated it refuses on collisions. It also callsinvalidateCatalog(), and the next query rebuilds the catalog from local disk — so following the advice can truncate the catalog. - Root is written live, not only at restart, so it cannot be cleaned up under a running server.
Reproduce
- Configure a repo with
@swamp/s3-datastore, bothprefixandnamespaceset. - Run
swamp servewith a scheduled workflow. - Let several scheduled runs complete, restart the server once.
- Compare
<cache>/outputs/against<cache>/<namespace>/outputs/and against the remote — outputs accumulate at root and never appear remotely.
Secondary observation (may be intentional)
workflow-runs/*.yaml run records DO reach the remote (771 present). workflow-runs/*.log files do not — 743 at root, 1 remote. If logs are deliberately local-only that is fine, but it would be worth stating explicitly, because the current split makes it hard to tell a design decision from the bug above.
Suggested fixes
- Write
outputs/(and the other execution artifacts) to the bound namespace directly, the same waydata/already is. - Failing that, have the
datastore setuprelocation target<cache>/<namespace>/rather than<cache>/. - Make push loud about files it skips outside the bound namespace — a count and a path sample. Today the skip is effectively invisible, and that is what turned this into a seven-day silent loss rather than a same-day bug report.
- Reword the
doctorfinding so it distinguishes "stale leftovers" from "live artifacts are being written outside the namespace and will not be saved".
Environment
- swamp
20260730.013940.0-sha.7e40de69(container),20260806.001601.0(host) @swamp/s3-datastorewithprefix+namespaceswamp servein Docker (debian trixie), datastore cache on a named volume
Distinct from the bulk-pull path issue I filed as #1554, though both stem from root-vs-namespace path handling.
Upstream repository: https://github.com/systeminit/swamp-extensions
Environment
- Extension:
@swamp/s3-datastore@2026.07.25.1 - swamp:
20260807.031228.0-sha.9a36314e - OS:
darwin(aarch64) - Deno:
2.8.3 - Shell:
/bin/zsh
Closed
No activity in this phase yet.
stack72 commented 8/7/2026, 11:03:47 PM
@sntxrr — great bug report, thank you. The root cause was in the @swamp/s3-datastore extension: pullFile/pullChanged were writing pulled objects to {cachePath}/{rel} instead of {cachePath}/{namespace}/{rel}, so outputs landed outside the bound namespace directory and were silently skipped by push.
This was fixed in @swamp/s3-datastore@2026.07.31.1 (extension commit 82a19e64c). Updating the extension should resolve the ongoing data loss:
swamp extension pull @swamp/s3-datastoreYour suggestion #3 (make push loud about skipped files) is already in place — the extension now warns with "Skipped N solo-layout file(s) outside the bound namespace" during push.
sntxrr commented 8/8/2026, 1:23:12 AM
Thanks for the quick turnaround. The layout half is genuinely fixed and I can confirm it — but the outputs half is still reproducible on @swamp/s3-datastore@2026.08.07.1, so I think this was closed early.
What IS fixed (confirmed)
pullFile now uses this.localRelPath(relativePath). Natural experiment across three repos on the same bucket:
| repo | extension | root-level files | namespaced |
|---|---|---|---|
| A | 2026.07.31.1 | 0 | 6249 |
| B | 2026.07.31.1 | 0 | 6833 |
| C | 2026.07.25.1 | 9301 | 8966 |
A and B ran hourly cron workloads for a week with zero root-level files. C, left on the old version, accumulated 9301. After upgrading C to 2026.08.07.1 and repairing it, a subsequent workflow run left root at 0 — so the repair is now durable, which it was not before. That part is solid, thank you.
What is NOT fixed
Outputs still never reach the remote. The determining factor is execution path, not version.
Live test on 2026.08.07.1, just now:
- Ran a workflow from the CLI at
01:21:10Z. - Its output was written to
<repo>/.swamp/outputs/@sntxrr/openwebui/instance/drift/…-2026-08-08T01-21-10-541Z.yaml. - The newest output object in the remote at that moment was
…01-15-00-044Z.yaml— written byswamp serve, not by my run. - The CLI run's output never appeared in
<cache>/<ns>/outputs/or in the remote.
Splitting the same repo's outputs by which process produced them:
reach the namespace + remote (written by swamp serve)
@sntxrr/home-ip/sync 93
@sntxrr/oci-instance/sync 14
@magistr/pihole/list 10
swamp/server-token/* 8
never leave <repo>/.swamp/outputs (written by CLI / cron)
@sntxrr/apprise-notify/notify 7
@sntxrr/swamp-version/target/drift 6
@magistr/pihole/sync 6
@sntxrr/openwebui/instance/{drift,sync} 10
@sntxrr/unifi/dhcp_reservation/{drift,device_drift,inventory} 12Two other repos already on 2026.07.31.1 show the same thing, and it is not a small tail:
repo <repo>/.swamp/outputs <cache>/<ns>/outputs remote
A 596 newest 08-07T21:54Z 217 stale 217
B 604 newest 08-07T22:35Z 198 (08-07T02:39Z) 198A verification trap worth flagging
<cache>/<ns>/outputs and the remote agree exactly — 217=217, 198=198 — which reads as healthy. It only proves the two are in sync with each other; both are stale. I initially concluded this was fixed on exactly that basis. Any check needs to compare against what was actually produced (<repo>/.swamp/outputs), not the two stores against each other.
Relationship to #1559
I think #1559 is the same defect seen from one angle, and closing it leaves this unfixed. Outputs are written to the repo-local path, and the only thing that ever moves them into the datastore is datastoreSetupExtension's migration — which targets bare cachePath. I verified setup.ts:461 still passes cachePath in CLI 20260807.031228.0-sha.9a36314e:
const config = { type: "filesystem" as const, path: cachePath };
const result = await deps.migrateData(sourceDir, cachePath, config);Under cron this is worse than the serve case I originally reported. datastore setup never runs at all, so there is no point at which outputs could migrate — they accumulate in .swamp/outputs/ forever. Five scheduled drift watchers here have produced ~600 outputs each; none is in the remote, so none is queryable, visible in the UI, or backed up. A swamp serve deployment at least gets a partial migration on restart.
Suggested framing
- Route output writes through the datastore directly, the same way
data/already is — rather than writing repo-local and relying on a later migration. - If the migration is meant to be the mechanism, fix #1559's target (
join(cachePath, ns)) and run it somewhere other thandatastore setup, or cron-driven repos never benefit. - Make push report skipped-outside-namespace files with a count and sample. An invisible skip is why this ran seven days here before anyone noticed.
Happy to test a build.
Environment
- CLI
20260807.031228.0-sha.9a36314e @swamp/s3-datastore@2026.08.07.1(also reproduced on 2026.07.31.1)- Remote datastore with
prefix+namespace
sntxrr commented 8/10/2026, 9:26:24 PM
Update from the reporter: swamp-club#1559 is fixed in CLI 20260809.004828.0-sha.b61c9de2, and with it the outputs stranding this issue was about.
Verified across two container restarts. The entrypoint still reports a migration, but the root layout now stays empty:
Files: 12 copied (5.9MB)
Relocated: data, outputs, workflow-runs now resolve under /home/[REDACTED]/.swamp/repos/<repoId>
*** ROOT TOTAL: 0 ***On the previous build (20260730.013940.0) the same restart left 14 files at the cache root, and two days of uptime left 535 — all outside the bound namespace where push could never see them. doctor datastores now passes on both writers.
So migrateData is targeting the namespace, which is what #1559 asked for. Combined with the pullFile/localRelPath fix in @swamp/s3-datastore@2026.07.31.1, the root-vs-namespace class looks resolved end to end here.
Correcting my own earlier comment on this issue: I had reported outputs were still stranded on 2026.07.31.1/2026.08.07.1. That was accurate at the time, but the missing piece was core-side (#1559), not the extension — the extension fix alone could not have closed it. Both were needed.
Recovered ~850 previously stranded outputs across the two hosts by migrating them into the namespace and pushing; remote outputs/ went 164 → 1,118. Nothing further needed from my side.
Separately filed swamp-club#1584 for a migrate-index partitioning bug found while enabling shard-first commits — unrelated to this, and much lower severity.
Sign in to post a ripple.