Under swamp serve with a namespaced remote datastore, method and workflow outputs/ are written to the datastore cache root instead of the bound namespace. Push only walks <cache>/<namespace>/**, so those outputs are silently never persisted to the remote. data/ is unaffected and syncs correctly, which is what makes this so easy to miss — the datastore looks healthy, run history looks fine, and only the outputs quietly stop arriving.

On an ephemeral or recreated container this is permanent, silent data loss.

Evidence

A containerised swamp serve running one workflow on a */15 schedule (4 outputs/hour). Counting outputs/ objects that reached the remote, by day:

2026-07-23:  2      <- namespace bound 2026-07-30
2026-07-24:  4
2026-07-26: 19
2026-07-27: 30
2026-07-29:  2
2026-07-30:  4
2026-07-31:  0      <- 96 produced locally
2026-08-01:  0      <- 96 produced locally
2026-08-02:  0      <- 96 produced locally
2026-08-03:  0      <- 96 produced locally
2026-08-04:  0      <- 96 produced locally
2026-08-05:  0      <- 96 produced locally
2026-08-06:  0      <- 96 produced locally

Seven consecutive days, ~670 outputs, zero delivered. Cache state at the point of discovery:

tree             cache ROOT   cache <ns>/   remote
data                   7861          7593     7868   <- fine
outputs                 812             4      128   <- broken
workflow-runs          1515           759      773

outputs/ had 812 at root vs 4 in the namespace. The stranded files are ordinary method outputs, e.g. outputs/<pkg>/<model>/sync/<instance>-2026-08-01T00-30-00-037Z.yaml.

The onset lines up exactly with the namespace being bound. Before that the repo was solo-layout and outputs reached the remote normally.

Mechanism

The entrypoint runs swamp datastore setup extension @swamp/s3-datastore --namespace <ns> ... on every container start. swamp serve writes outputs/, workflow-runs/, definitions-evaluated/ and workflows-evaluated/ into the repo-local .swamp/ directory during operation:

/data/repo/.swamp/outputs/<pkg>/<model>/sync/*.yaml     <- accumulates while serve runs

On the next start, datastore setup relocates those into the cache root, reported as:

Datastore Setup Complete
  Files:    969 copied (3.4MB)
  Dirs:     definitions-evaluated, workflows-evaluated, data, outputs, workflow-runs
Relocated: ... now resolve under /home/[REDACTED]/.swamp/repos/<repoId>
  Moved from: /data/repo/.swamp

That destination is outside the bound namespace, so push never reads it. The files are neither pushed nor reported as skipped in a way an operator would notice.

Why this is worse than it looks

Silent. No error, no warning. swamp doctor datastores does flag "un-migrated data found at root level", but that message points at swamp datastore namespace migrate, which cannot fix it (see below), and it reads as a stale-leftovers problem rather than "your outputs are not being saved".
data/ still works, so every obvious health check passes.
Unrecoverable on ephemeral compute. The stranded files live only on the container's volume. Anyone running swamp serve on k8s/ECS/a recreated container loses all execution outputs with no indication it happened.
The suggested remedy is unavailable. datastore namespace migrate requires the namespace dir to be effectively empty; with both layouts populated it refuses on collisions. It also calls invalidateCatalog(), and the next query rebuilds the catalog from local disk — so following the advice can truncate the catalog.
Root is written live, not only at restart, so it cannot be cleaned up under a running server.

Reproduce

Configure a repo with @swamp/s3-datastore, both prefix and namespace set.
Run swamp serve with a scheduled workflow.
Let several scheduled runs complete, restart the server once.
Compare <cache>/outputs/ against <cache>/<namespace>/outputs/ and against the remote — outputs accumulate at root and never appear remotely.

Secondary observation (may be intentional)

workflow-runs/*.yaml run records DO reach the remote (771 present). workflow-runs/*.log files do not — 743 at root, 1 remote. If logs are deliberately local-only that is fine, but it would be worth stating explicitly, because the current split makes it hard to tell a design decision from the bug above.

Suggested fixes

Write outputs/ (and the other execution artifacts) to the bound namespace directly, the same way data/ already is.
Failing that, have the datastore setup relocation target <cache>/<namespace>/ rather than <cache>/.
Make push loud about files it skips outside the bound namespace — a count and a path sample. Today the skip is effectively invisible, and that is what turned this into a seven-day silent loss rather than a same-day bug report.
Reword the doctor finding so it distinguishes "stale leftovers" from "live artifacts are being written outside the namespace and will not be saved".

Environment

swamp 20260730.013940.0-sha.7e40de69 (container), 20260806.001601.0 (host)
@swamp/s3-datastore with prefix + namespace
swamp serve in Docker (debian trixie), datastore cache on a named volume

Distinct from the bulk-pull path issue I filed as #1554, though both stem from root-vs-namespace path handling.

Upstream repository: https://github.com/systeminit/swamp-extensions

Environment

Extension: @swamp/s3-datastore@2026.07.25.1
swamp: 20260807.031228.0-sha.9a36314e
OS: darwin (aarch64)
Deno: 2.8.3
Shell: /bin/zsh

02Bog Flow

Closed

8/7/2026, 11:03:49 PM

No activity in this phase yet.

03Sludge Pulse

stack72 assigned stack728/7/2026, 10:45:01 PM

stack72 commented 8/7/2026, 11:03:47 PM

@sntxrr — great bug report, thank you. The root cause was in the @swamp/s3-datastore extension: pullFile/pullChanged were writing pulled objects to {cachePath}/{rel} instead of {cachePath}/{namespace}/{rel}, so outputs landed outside the bound namespace directory and were silently skipped by push.

This was fixed in @swamp/s3-datastore@2026.07.31.1 (extension commit 82a19e64c). Updating the extension should resolve the ongoing data loss:

swamp extension pull @swamp/s3-datastore

Your suggestion #3 (make push loud about skipped files) is already in place — the extension now warns with "Skipped N solo-layout file(s) outside the bound namespace" during push.

sntxrr commented 8/8/2026, 1:23:12 AM

Thanks for the quick turnaround. The layout half is genuinely fixed and I can confirm it — but the outputs half is still reproducible on @swamp/s3-datastore@2026.08.07.1, so I think this was closed early.

What IS fixed (confirmed)

pullFile now uses this.localRelPath(relativePath). Natural experiment across three repos on the same bucket:

repo	extension	root-level files	namespaced
A	2026.07.31.1	0	6249
B	2026.07.31.1	0	6833
C	2026.07.25.1	9301	8966

A and B ran hourly cron workloads for a week with zero root-level files. C, left on the old version, accumulated 9301. After upgrading C to 2026.08.07.1 and repairing it, a subsequent workflow run left root at 0 — so the repair is now durable, which it was not before. That part is solid, thank you.

What is NOT fixed

Outputs still never reach the remote. The determining factor is execution path, not version.

Live test on 2026.08.07.1, just now:

Ran a workflow from the CLI at 01:21:10Z.
Its output was written to <repo>/.swamp/outputs/@sntxrr/openwebui/instance/drift/…-2026-08-08T01-21-10-541Z.yaml.
The newest output object in the remote at that moment was …01-15-00-044Z.yaml — written by swamp serve, not by my run.
The CLI run's output never appeared in <cache>/<ns>/outputs/ or in the remote.

Splitting the same repo's outputs by which process produced them:

reach the namespace + remote   (written by swamp serve)
  @sntxrr/home-ip/sync            93
  @sntxrr/oci-instance/sync       14
  @magistr/pihole/list            10
  swamp/server-token/*             8

never leave <repo>/.swamp/outputs (written by CLI / cron)
  @sntxrr/apprise-notify/notify           7
  @sntxrr/swamp-version/target/drift      6
  @magistr/pihole/sync                    6
  @sntxrr/openwebui/instance/{drift,sync} 10
  @sntxrr/unifi/dhcp_reservation/{drift,device_drift,inventory} 12

Two other repos already on 2026.07.31.1 show the same thing, and it is not a small tail:

repo   <repo>/.swamp/outputs      <cache>/<ns>/outputs   remote
A      596  newest 08-07T21:54Z   217  stale             217
B      604  newest 08-07T22:35Z   198  (08-07T02:39Z)    198

A verification trap worth flagging

<cache>/<ns>/outputs and the remote agree exactly — 217=217, 198=198 — which reads as healthy. It only proves the two are in sync with each other; both are stale. I initially concluded this was fixed on exactly that basis. Any check needs to compare against what was actually produced (<repo>/.swamp/outputs), not the two stores against each other.

Relationship to #1559

I think #1559 is the same defect seen from one angle, and closing it leaves this unfixed. Outputs are written to the repo-local path, and the only thing that ever moves them into the datastore is datastoreSetupExtension's migration — which targets bare cachePath. I verified setup.ts:461 still passes cachePath in CLI 20260807.031228.0-sha.9a36314e:

const config = { type: "filesystem" as const, path: cachePath };
const result = await deps.migrateData(sourceDir, cachePath, config);

Under cron this is worse than the serve case I originally reported. datastore setup never runs at all, so there is no point at which outputs could migrate — they accumulate in .swamp/outputs/ forever. Five scheduled drift watchers here have produced ~600 outputs each; none is in the remote, so none is queryable, visible in the UI, or backed up. A swamp serve deployment at least gets a partial migration on restart.

Suggested framing

Route output writes through the datastore directly, the same way data/ already is — rather than writing repo-local and relying on a later migration.
If the migration is meant to be the mechanism, fix #1559's target (join(cachePath, ns)) and run it somewhere other than datastore setup, or cron-driven repos never benefit.
Make push report skipped-outside-namespace files with a count and sample. An invisible skip is why this ran seven days here before anyone noticed.

Happy to test a build.

Environment

CLI 20260807.031228.0-sha.9a36314e
@swamp/s3-datastore@2026.08.07.1 (also reproduced on 2026.07.31.1)
Remote datastore with prefix + namespace

sntxrr commented 8/10/2026, 9:26:24 PM

Update from the reporter: swamp-club#1559 is fixed in CLI 20260809.004828.0-sha.b61c9de2, and with it the outputs stranding this issue was about.

Verified across two container restarts. The entrypoint still reports a migration, but the root layout now stays empty:

Files:    12 copied (5.9MB)
Relocated: data, outputs, workflow-runs now resolve under /home/[REDACTED]/.swamp/repos/<repoId>
*** ROOT TOTAL: 0 ***

On the previous build (20260730.013940.0) the same restart left 14 files at the cache root, and two days of uptime left 535 — all outside the bound namespace where push could never see them. doctor datastores now passes on both writers.

So migrateData is targeting the namespace, which is what #1559 asked for. Combined with the pullFile/localRelPath fix in @swamp/s3-datastore@2026.07.31.1, the root-vs-namespace class looks resolved end to end here.

Correcting my own earlier comment on this issue: I had reported outputs were still stranded on 2026.07.31.1/2026.08.07.1. That was accurate at the time, but the missing piece was core-side (#1559), not the extension — the extension fix alone could not have closed it. Both were needed.

Recovered ~850 previously stranded outputs across the two hosts by migrating them into the namespace and pushing; remote outputs/ went 164 → 1,118. Nothing further needed from my side.

Separately filed swamp-club#1584 for a migrate-index partitioning bug found while enabling shard-first commits — unrelated to this, and much lower severity.

--ca-cert / SWAMP_CA_CERT are ignored when serve runs --auth-mode token — only DENO_CERT reaches the connection's TLS config (found validating #1748)

Only documented non-interactive auth puts the password in argv; the safe SWAMP_API_KEY path is undocumented

Retyping a model instance strands catalog rows no command can reclaim — prune matches on model_id, delete resolves the current type (follow-up to #1009)

Docs: update cancel endpoint response statuses in REST API reference and workflows page

docs: update vault audit trail reference for write operation support

docs: document --dashboard flag for swamp serve

ExtensionWorkflowRepository scans manifest.yaml as a workflow, emitting 'Unknown key manifestVersion' warning

Relative path for custom agent skills setup doesn't provision skills

swamp serve daemon enable: add --dashboard flag

CLI writes runtime state (lastStalenessWarning) into tracked .swamp.yaml, dirtying a committed config file

extension install serializes files[] in readdir order, producing machine-dependent reorder diffs in upstream_extensions.json

Workflow YAML rewritten in place: block-scalar description truncated, file left unparseable, and validate still reports PASSED

workflow validate does not resolve modelType on factory-pattern steps — a non-existent extension validates clean

vault.get() arguments are verbatim, so a per-target secret cannot be expressed in workflow YAML

Cancellation/--timeout skips remaining steps including `completed`/`always` branches; --timeout waits for the in-flight subprocess

Remote "write-bearing" classification is inferred from data-plane writes, not declared; external side effects can be re-executed on worker disconnect (docs say the opposite)

Audit timeline is not populated by swamp CLI commands unless the AI-tool hook fires (manual says recording is automatic)

manual_approval suspends the run while parallel sibling jobs are mid-flight; in-flight steps are torn down and left `running`

`workflow resume --from` does nothing for forEach steps whose template name contains `${{ }}`

Step-level dependsOn on a forEach step never evaluates true (except `always`)

rum/site-info: lookup GETs a non-existent endpoint, always 405

vault read-secret appends a newline to piped output

wss:// direct to serve's own TLS listener still fails on 20260817 — client ALPN lands on h2, WS-over-h2 unimplemented (follow-up to #1363)

serve: vault audit repository not wired up — audit trail always empty

Support Jira webhooks with X-Hub-Signature HMAC-SHA256 verification

Add typed read-only remote ref lookup to @swamp/git

pullChanged skips same-size updated files — size-only change detection ignores the index sha256

A run's terminal status reaches the shared datastore only when a later run happens to push it

workflow cancel --all on the serve host shuts down the serve process

Readability of tables in documentation

repo init/upgrade --tool kiro references .kiro/skills/ but installs skills to ~/.kiro/skills/

vault: audit trail should record put, delete, and annotate operations

data.findBySpec() returns only the first in-run record, not all of them, so an .all() assert silently checks one element

repo upgrade --tool kiro reports wrong changedFiles paths

Suppress boot-time "pulled extension(s) have missing source files" WRN when invoked command is `swamp extension install`

s3-datastore: a stale machine's sync pushes its old 'latest' pointer over the newer remote one, and the rollback is permanent

`search` → `list` aliases are hand-rolled Command clones, not aliases: 10 of 11 have drifted from their parent's option set

Make official Docker image OCI-compliant

Method-name collision between a base model type and an extending extension silently drops the whole extension file — no error, doctor reports pass

data prune still misclassifies live records as orphans on 20260820 (after #1396) — serve-minted token/grant records, and report data of every live workflow

vault put prompts for the secret before validating the vault name exists

workflow validate: false 'Missing required inputs' for CEL-templated globalArgs (${{ inputs.* }})

Add SWAMP_CLI_OUTPUT env var to control default output mode (plaintext/tui/json)

@swamp/git fetch: support local path as remote

Dispatched steps over 15 minutes cannot write results: dispatch credentials expire and never refresh

Concurrent pushes to one namespace fail with HTTP 409 ConditionalRequestConflict — the conditional PUT is never retried

serve: triggerSource not persisted to workflow run summary

dataOutputOverrides not applied during remote execution — ephemeral resources GC'd before override takes effect

extension rm also deletes the lockfile entry, and install on an empty lockfile is a silent success — rm/install is not a round trip

--max-run-duration only applies to detached runs, not attached runs

Cancel endpoint reports status: cancelled for runs it did not actually cancel

@swamp/git: no way to amend a commit, but push can --force

Installation with new account creation times out

pre-flight checks run on orchestrator, not on worker — breaks remote execution with filesystem-dependent checks

docs: document --remote-only flag for swamp serve

Swamp serve daemon doesn't run with --hot-reload

serve: include workflowName in HealthSnapshotSchedule

knowledge-base retrieve persists query output into the state resource, with lifetime infinite and the query text in the instance name

Skip 50MB monolithic index rewrite on v2 shard-first path

Share single AwsS3Client instance across lock, sync, and verifier

Double shard assembly per command lifecycle (pull + push both read all shards)

pullChanged never records commitSeq for shard-first (v2) index

localHasAllRemoteEntries() permanently blocks sidecar update on partial caches

serve: workflow.history.get cannot find CLI-created runs or look up by run ID

"TRIAL ENDED" modal shown to team plan user when toggling extension visibility

Concurrent suspended workflow runs can't be disambiguated from 'approvals'; fresh runs should supersede matching-input suspended runs

I _think_ I might like a guard in the trigger yaml

swamp cmds don't realise the user is sshed into the system

serve dashboard: add cluster.instances and serve.config endpoints

CEL expressions have timestamp/duration but no current-time value, so no guard can express "older than N"

Support direct type execution (@type) with --server

access token mint writes to user vault, not serve's _token-secrets control plane in HA mode

Generic OTLP endpoint appends signal path after query or fragment

Issue lifecycle start drops top-level issue comments

serve dashboard: add instanceId, triggerSource, and step progress to WorkflowRunSummary

Document worker affinity (affinity: true) for workflows and jobs

Datastore MongoDB driver leaks connections — 5960 accumulated over 8 days

workflow approvals displays the latest run's prompt for every suspended run

@swamp/git: add a read-only working-tree diff for pre-commit review

docs: document --oauth-client-name flag for swamp serve