Two machines sharing one @swamp/s3-datastore namespace never converge. Each data record is a directory of immutable version dirs (1/, 2/, ...) plus one mutable 2-byte latest file naming the current version, and reads resolve through latest. The version dirs are new keys so they always replicate correctly, but latest is synced by index/mtime bookkeeping with no version comparison.

Result: a machine whose local cache is behind will push its OLD latest over the newer remote one during a routine swamp datastore sync, silently rolling back what every other machine reads. The machine that had the correct value then refuses to re-push it, so the rollback is permanent.

Environment

swamp 20260819.011806.0-sha.a9c7ee9a (both machines)
@swamp/s3-datastore 2026.08.12.1
Backend: Cloudflare R2 via endpoint: https://<ACCOUNT_ID>.[HOST-1], forcePathStyle: true, region: auto, single namespace
Machine A: macOS. Machine B: Ubuntu. Identical .swamp.yaml, same repoId, same bucket and namespace.

Reproduction

On machine A, run model methods that advance a record several versions. Flush with swamp datastore sync.
Confirm the remote pointer is current by reading the object directly: GET <bucket>/<ns>/data/<type>/<model-id>/<record>/latest returns 8.
On machine B, whose cache is behind, run a plain swamp datastore sync.
Read the same remote object again. It now returns 6.

Machine B pushed its stale pointer over the newer remote value. No error, no warning, and the sync reports success.

Evidence

Observed on the record lease-current, where R2 held version dirs 1..8:

Step	remote `latest`
after machine A flushed	`8`
after machine B ran `sync`	`6`

Machine A will not repair it. Its local latest still reads 8, but swamp datastore sync --push reports Pushed 0 files, because its index entry for that key absorbed machine B's push metadata (lastModified set to B's push time) and it therefore believes the remote is already current.

So the steady state is: remote 6, machine A local 8 and refusing to push, machine B local 6. Neither side self-corrects.

Scope at the time of diagnosis: 17 of 135 records on machine B had latest pointing below the highest version present on its own disk. The affected records were exactly those the other machine had written most recently.

Why this matters

The pointer is what every read resolves through, so a rolled-back pointer silently serves stale content for a record whose newer versions are present on disk and in the bucket. In our case one of the affected records was a mutual-exclusion lease, where a stale read means two machines can disagree about who holds it.

Additional observations, possibly the same root cause

swamp model method run does not flush. It leaves entries in dirtyPaths, and only swamp datastore sync uploads them. Until the writer syncs, its most recent records are invisible to every other machine. If that is intended, it is worth documenting, because "the first swamp command that touches data will auto-hydrate" in the sharing docs led us to expect reads to be self-healing.
Deleting a local latest does not cause the next sync to re-fetch it, because the index still lists the key as present. It is only restored if that sync happens to have other work to do. Deleting all 138 pointers and syncing produced Sync complete: 0 pulled, 0 pushed, leaving every record unreadable until we restored from a backup. A sync that verifies local presence, or a repair command, would prevent that footgun.
A full re-hydration from scratch (removing the local cache entirely and letting it rebuild) does not help, since it faithfully inherits whatever stale pointer the last syncer pushed.

Suggested fix

Give the pointer write precedence rather than last-writer-wins. Either compare versions before overwriting latest and refuse to move it backwards, or derive the current version from the highest version directory present and drop the mutable pointer entirely. Conditional writes work correctly against R2 (we verified If-None-Match: * returns 200 then 412), so a compare-and-swap on the pointer is available on this backend.

Upstream repository: https://github.com/systeminit/swamp-extensions

Environment

Extension: @swamp/s3-datastore@2026.08.12.1
swamp: 20260820.072114.0-sha.05d20b6a
OS: darwin (aarch64)
Deno: 2.8.3
Shell: /bin/zsh

02Bog Flow

Open

8/21/2026, 7:08:11 AM

No activity in this phase yet.

03Sludge Pulse

--ca-cert / SWAMP_CA_CERT are ignored when serve runs --auth-mode token — only DENO_CERT reaches the connection's TLS config (found validating #1748)

Only documented non-interactive auth puts the password in argv; the safe SWAMP_API_KEY path is undocumented

Retyping a model instance strands catalog rows no command can reclaim — prune matches on model_id, delete resolves the current type (follow-up to #1009)

Docs: update cancel endpoint response statuses in REST API reference and workflows page

docs: update vault audit trail reference for write operation support

docs: document --dashboard flag for swamp serve

ExtensionWorkflowRepository scans manifest.yaml as a workflow, emitting 'Unknown key manifestVersion' warning

Relative path for custom agent skills setup doesn't provision skills

swamp serve daemon enable: add --dashboard flag

CLI writes runtime state (lastStalenessWarning) into tracked .swamp.yaml, dirtying a committed config file

extension install serializes files[] in readdir order, producing machine-dependent reorder diffs in upstream_extensions.json

Workflow YAML rewritten in place: block-scalar description truncated, file left unparseable, and validate still reports PASSED

workflow validate does not resolve modelType on factory-pattern steps — a non-existent extension validates clean

vault.get() arguments are verbatim, so a per-target secret cannot be expressed in workflow YAML

Cancellation/--timeout skips remaining steps including `completed`/`always` branches; --timeout waits for the in-flight subprocess

Remote "write-bearing" classification is inferred from data-plane writes, not declared; external side effects can be re-executed on worker disconnect (docs say the opposite)

Audit timeline is not populated by swamp CLI commands unless the AI-tool hook fires (manual says recording is automatic)

manual_approval suspends the run while parallel sibling jobs are mid-flight; in-flight steps are torn down and left `running`

`workflow resume --from` does nothing for forEach steps whose template name contains `${{ }}`

Step-level dependsOn on a forEach step never evaluates true (except `always`)

rum/site-info: lookup GETs a non-existent endpoint, always 405

vault read-secret appends a newline to piped output

wss:// direct to serve's own TLS listener still fails on 20260817 — client ALPN lands on h2, WS-over-h2 unimplemented (follow-up to #1363)

serve: vault audit repository not wired up — audit trail always empty

Support Jira webhooks with X-Hub-Signature HMAC-SHA256 verification

Add typed read-only remote ref lookup to @swamp/git

pullChanged skips same-size updated files — size-only change detection ignores the index sha256

A run's terminal status reaches the shared datastore only when a later run happens to push it

workflow cancel --all on the serve host shuts down the serve process

Readability of tables in documentation

repo init/upgrade --tool kiro references .kiro/skills/ but installs skills to ~/.kiro/skills/

vault: audit trail should record put, delete, and annotate operations

data.findBySpec() returns only the first in-run record, not all of them, so an .all() assert silently checks one element

repo upgrade --tool kiro reports wrong changedFiles paths

Suppress boot-time "pulled extension(s) have missing source files" WRN when invoked command is `swamp extension install`

s3-datastore: a stale machine's sync pushes its old 'latest' pointer over the newer remote one, and the rollback is permanent

`search` → `list` aliases are hand-rolled Command clones, not aliases: 10 of 11 have drifted from their parent's option set

Make official Docker image OCI-compliant

Method-name collision between a base model type and an extending extension silently drops the whole extension file — no error, doctor reports pass

data prune still misclassifies live records as orphans on 20260820 (after #1396) — serve-minted token/grant records, and report data of every live workflow

vault put prompts for the secret before validating the vault name exists

workflow validate: false 'Missing required inputs' for CEL-templated globalArgs (${{ inputs.* }})

Add SWAMP_CLI_OUTPUT env var to control default output mode (plaintext/tui/json)

@swamp/git fetch: support local path as remote

Dispatched steps over 15 minutes cannot write results: dispatch credentials expire and never refresh

Concurrent pushes to one namespace fail with HTTP 409 ConditionalRequestConflict — the conditional PUT is never retried

serve: triggerSource not persisted to workflow run summary

dataOutputOverrides not applied during remote execution — ephemeral resources GC'd before override takes effect

extension rm also deletes the lockfile entry, and install on an empty lockfile is a silent success — rm/install is not a round trip

--max-run-duration only applies to detached runs, not attached runs

Cancel endpoint reports status: cancelled for runs it did not actually cancel

@swamp/git: no way to amend a commit, but push can --force

Installation with new account creation times out

pre-flight checks run on orchestrator, not on worker — breaks remote execution with filesystem-dependent checks

docs: document --remote-only flag for swamp serve

Swamp serve daemon doesn't run with --hot-reload

serve: include workflowName in HealthSnapshotSchedule

knowledge-base retrieve persists query output into the state resource, with lifetime infinite and the query text in the instance name

Skip 50MB monolithic index rewrite on v2 shard-first path

Share single AwsS3Client instance across lock, sync, and verifier

Double shard assembly per command lifecycle (pull + push both read all shards)

pullChanged never records commitSeq for shard-first (v2) index

localHasAllRemoteEntries() permanently blocks sidecar update on partial caches

serve: workflow.history.get cannot find CLI-created runs or look up by run ID

"TRIAL ENDED" modal shown to team plan user when toggling extension visibility

Concurrent suspended workflow runs can't be disambiguated from 'approvals'; fresh runs should supersede matching-input suspended runs

I _think_ I might like a guard in the trigger yaml

swamp cmds don't realise the user is sshed into the system

serve dashboard: add cluster.instances and serve.config endpoints

CEL expressions have timestamp/duration but no current-time value, so no guard can express "older than N"

Support direct type execution (@type) with --server

access token mint writes to user vault, not serve's _token-secrets control plane in HA mode

Generic OTLP endpoint appends signal path after query or fragment

Issue lifecycle start drops top-level issue comments

serve dashboard: add instanceId, triggerSource, and step progress to WorkflowRunSummary

Document worker affinity (affinity: true) for workflows and jobs

Datastore MongoDB driver leaks connections — 5960 accumulated over 8 days

workflow approvals displays the latest run's prompt for every suspended run

@swamp/git: add a read-only working-tree diff for pre-commit review

docs: document --oauth-client-name flag for swamp serve