Lab #3549: S3/GCS datastores: keep v2 sync off v3 buckets (exclude _litestream/, refuse a format-3 marker) for swamp releases before v20261009.154711.0

Background

Datastore rework Phase 4E (swamp-club#3534) makes v3 a datastore format on S3 and GCS. swamp datastore setup extension @swamp/s3-datastore|@swamp/gcs-datastore --format 3 writes a format-3 marker at <prefix>/_control/datastore-format through datastoreControlPlaneStore(). Litestream then replicates each checkout's commits.db under <prefix>/_litestream/<fence>/.

Who is exposed

Only swamp releases from before v20261009.154711.0, the first release with #3189's format guard (c4e65dbe6). Every release from then on reads the marker at <prefix>/_control/datastore-format and refuses format 3 on S3, GCS and extension datastores before it writes anything.

An older binary sees a v3 bucket as v2. With no .datastore-index.json, the v2 sync service lists the whole bucket. isInternalCacheFile skips _control/ and _blobs/ but not _litestream/, so it publishes a v2 index over Litestream's objects, pulls the LTX files into its cache, and can push them back as files to sync.

Why the fix belongs in the extensions

Extensions are pinned per repo. A clone of a v3 bucket repo has a lockfile pinning the extension at 2026.10.16.1 or later, and an old binary loads that pinned extension. So a guard in the extension covers binaries that core can no longer fix.

Scope (both on S3 and GCS, one extension release)

Exclude _litestream/ from the file sync, in isInternalCacheFile, as _blobs/ already is (#3492). Do this regardless of part 2: today nothing stops a v2 pull from downloading LTX files into a cache, or a push from treating them as files to sync.
Refuse a v3 bucket in the sync service itself, as defence in depth for binaries older than the guard. createSyncService (or the first pull or push) reads the root marker and refuses format 3 or higher with an "upgrade swamp" message, before writing anything.

Not a blocker

It does not block 4E (swamp-club#3534). E's setup keeps relying on the core guard for every current binary. Ideally this release lands before v3 on buckets is announced, alongside Phase 4 H and I.

Review note on part 2: refuse at the first pull or push, not at createSyncService construction. A construction-time refusal turns any path in new core that still builds a sync service for a v3 bucket (a status or health check, a doctor path, something #3534 misses) into a hard failure for current binaries. Refusing at the first pull or push still stops a pre-guard binary before it publishes an index or moves LTX files. Add a test that a sync service built for a format-3 bucket constructs fine, and refuses pullChanged / pushChanged / preparePush with the upgrade message, writing nothing. Ordering: swamp-club#3550 (the #3546 regression) touches the same *_cache_sync.ts files and should land first; rebase onto it.

Docs: run-workflows-on-a-schedule tells readers to curl /health for queue state, which no longer lists schedules

Inbox API contract and SwampClubClient inbox methods: summary, list, get and mark-read, pinned by a shared fixture and design/surfaces/inbox.md

Datastore rework Phase 5 (team mode) 2/3: the CLI's serve upstream — with serverAddress, a v3 checkout commits, reads chunks and catches up through serve's datastore API, with no storage credentials

Scheduled flaky-test detection is failing

v3 read model: guard against aliased key spellings on case-insensitive filesystems, without slowing replay

Scheduled flaky-test detection is failing

stagecraft remote studio: one long-lived connection to serve instead of a swamp process per read

stagecraft remote studio poll: follow-ups from #3658 review (flaky test, mid-poll first page, quiet-poll test)

docs(manual): the manual documents swamp subscription status and start — standings, call to action, flags and --json output

swamp subscription start opens the Swamp Club billing page, or a collective's with --collective, and status points its call to action at it

v3: the compaction policy bounds how much history a fresh clone replays (retainMs, minCommits, retainCommits) and the base load skips files the tail removes

datastore_v3_replica_verify_test stall case flakes under the full integration run: restore.pid is read before the stub writes it

v3 on S3/GCS: a replica restore yields a corrupt commits.db (fails SQLite's integrity check), so fresh clones and restores of the datastore fail; reproduces with Litestream alone from its first compacted L1 file (found by #3916)

swamp subscription status reports the plan and billing standing of the signed-in user and each collective, worded as auth whoami words it

/o/<slug>/billing sends a signed-out visitor to the collective page, or a 404 when private, instead of through login and back like /billing

swamp-extensions GCS storage test 'put: an abort mid-upload leaves no object and no resumable session' fails about 1 run in 5 (fixed 50 ms sleeps)

refactor(cli): share auth whoami's collective-standing helpers ahead of swamp subscription, and pin Personal License and Admin standings

@swamp/cve/tailworm: check the reachability matrix against a Tailscale-hosted tailnet

AWS codegen treats nested primaryIdentifier pointers as top-level fields

AWS codegen collapses nested createOnly/readOnly paths to their last segment

Generated state schemas require every child of nested objects, so normal resources warn on create/get

swamp-club.com/cve/tailworm states two things its sources do not say: finding 5's memory growth and no exploitation in the wild

cve/tailworm extension README says the findings were not seen in the wild, which no source states

Datastore rework Phase 5 (team mode) 1/3: serve's datastore API — commit, chunk and catch-up endpoints, authorisation on each op's intent, principal stamping and actionable conflicts

@swamp/cve/tailworm: reachability follow-ups from #3919 review

@swamp/cve/tailworm: opt-in reachability matrix from each node's packet filter

Datastore rework Phase 5: content-level authorisation for forwarded definition and evaluated-file writes

swamp-club.com/cve/tailworm: cover grade, probeScript, gradeFleet and the reachability matrix (extension 2026.10.25.5)

Docs: --trusted-hosts accepts host:port for public addresses on non-default ports

@swamp/cve/tailworm: after mitigate caps headscale, the rescan still tells you to set MemoryMax and Restart=always

@swamp/cve/tailworm grade: unix serve targets the probe could not resolve are graded as ordinary sockets

kubernetes: externalsecret and normalizeMeta follow-ups from #3907

v3 crash recovery's progress line breaks --json stderr: a failing command that recovered prints text before its JSON error (regression from #3363 / #3893)

Docs: vaults reference should say a wide local_encryption key file gets a warning

Docs: manual states the old pre-flight check rule and scores fast-check by deno doc --lint

Docs: define the sensitive-output fallback vault order and document the doctor vaults fallback warning

v3 long silent waits: crash recovery runs ~24 s with no output, data prune is silent 6-10 s in log mode, and a lagging read hangs ~112 s when Litestream's listing is cut (found by #3756)

Add @swamp/cve/http2shell: scan and mitigate the Brotli decoder RCE (CVE-2026-111497, Node2Shell/HTTP2Shell)

extension_tarball_analyzer_test fails on macOS: tar --mode is GNU-only

S3/GCS datastore: a push that fails after uploading drops its recorded deletes, so the retry brings deleted data back

v2 S3: about 4-5% of serial workflow runs fail with datastore_unpublished_data_version ('a concurrent run or the serve poller synced') when nothing else is running (found by #3756)

Datastore rework Phase 4 I: dogfood and benchmark solo v3 (filesystem, S3, GCS) in a throwaway repo with the released binary, check the kill criteria, and decide whether to announce

v3 bucket outage: a command cut off while becoming the writer says its changes are saved locally when nothing was saved (workflow approve), and the suggested recover fails for a run cut mid-step (found by #3756)

auth login browser flow prints a fatal stack trace when swamp-club cannot be reached

Scheduled flaky-test detection is failing

v3 first clone and restore of a bucket datastore take 2.6-3x v2, missing #3418's 1.5x target (found by #3756)

CVE pages: horizontally scrolling code blocks are not keyboard-reachable

Homepage club plate link has no visible keyboard focus

Scheduled flaky-test detection is failing

swamp update downloads before failing in a sticky directory with another user's binary, and blames an immutable flag

Docs: extra-headers how-to systemd snippet puts a path in SWAMP_SERVE_EXTRA_HEADERS

Docs: install-swamp says SWAMP_HOME moves everything, but agent skill folders stay in the user's home folder

Docs: dashboard Approvals view shows run context, reason and per-viewer decide state

v3 bucket outage: a workflow run whose checkpoint committed is told nothing was committed, three outage errors have no stable code, and exit takes 15-60 s (found by #3756)

v3 identity on buckets: S3 checkouts with different endpoint spellings fork the datastore silently, and a hostname change blocks every write for ~5 minutes (found by #3756)

typed_workflow_passes_test child-sentinel case flakes under full-suite load: f.received.at(-1) reads the compare step's empty payload

extension push drops local modules when the repo under a node_modules folder is reached through a symlinked path (macOS /var, the swamp-club#3782 fix)

Teach defineBehavior in the extension skill now that swamp-testing 0.20261013.52 exports it

Docs: issue redaction keeps git ids and non-address colon runs, and asks before posting (swamp-club#3686)

Docs: worker daemon enable reference is missing --ca-cert, --concurrency and SWAMP_SERVE_EXTRA_HEADERS

v3 on S3/GCS: a write whose commit cannot reach the bucket prints "succeeded" on --json, exits 1, and is lost (confirm-before-output, found by #3756)

swamp-club /cve/dirtyfrag page shows --input targetHost/sshUser/sshKey/dockerContainer, which the CLI rejects

v3: datastore config migrate skips every workflow that has run (name-committed), and they are then "not found" (found by #3756)

@swamp/cve/dirtyfrag report suggests commands for dirtyfrag-scanner whatever the model is named

v3 on a bucket: after a commit fails because the bucket is unreachable, the closing sync still waits its full 60 s before the command exits

Interactive history and output search pickers cannot open a run under a non-default datastore

v3 recovery sweep: dot-prefixed data names escape it, HFS+ non-ASCII names are deleted by it, and an old mtime keeps a dropped write's file (bug hunt group 8)

Source glob expansion crashes when a directory under the glob parent disappears mid-walk

v3 chunk barrier on macOS: use F_BARRIERFSYNC (ordering) instead of F_FULLFSYNC (full drive-cache flush), cutting ~10-25 ms (200 ms+ on a busy disk) per command

extension_source_add_test globs the real home directory and flakes when a file in ~ disappears mid-walk

v3 join check refuses a fresh managedConfig checkout on S3/GCS: the datastore extension's auto-install writes the managed lockfile first (regression of #3690)

Docs: doctor datastores reference for the v3 content check and lost format marker repair

Docs: step dataOutputOverrides vaultName and the field-by-field override merge (swamp-club#3606)

Docs: data name portability rule and warning

Docs: vault create and vault migrate refuse a local_encryption name with leftover storage

v3 namespaces: cross-namespace reads return the reader's own content, a namespaced local tier is overwritten by solo commits, and namespaces can collide with swamp's own .swamp/ directories (bug hunt group 7)

v3: a checkout holding v2 records joins a v3 datastore through .swamp.yaml unguarded, overwriting its records and diverging for good (bug hunt group 5)

v3 on S3/GCS: the namespace refusal names setup filesystem, and an unreadable marker on a format-3 repo says "will not open … as format 2" (found by swamp-uat#715)

v3 on S3/GCS: setup on a prefix holding only swamp's format-3 marker fails with "this swamp release can't open" format 3 (found by swamp-uat#715)

Workflow runs ignore defaultVault: step sensitive outputs land in the first vault, and validate predicts a vault the run doesn't use