Session-start 'swamp model search --json' in the AGENTS.md template is unfulfillable - wire it to a hook or drop it

4h ago

#3124

Flaky property test: analyzeExpression dot/bracket agreement fails when the generated model name is a CEL keyword

9h ago

#3123

Accept usernames for user subjects in grant files

10h ago

#3122

Docs: enable-managed-config should say to re-run config migrate when its push fails

10h ago

#3121

Docs: grant file reference shows a stale format and omits resources and subjects

10h ago

#3120

Flaky property test: analyzeExpression hyphenated dot access fails when modelName generates a CEL keyword ("in")

11h ago

#3119

extension quality and push crash with IsADirectory when additionalFiles lists a directory

11h ago

#3118

extension push: a required global argument renders with a stray "" ("text": "string""")

11h ago

#3116

Docs: extension publishing reference is stale on bare specifiers, quality output and fmt config (swamp-club#3022)

11h ago

#3114

swamp-extensions: replace bare import specifiers with npm:/jsr: specifiers in first-party extensions

12h ago

#3112

extension push: 'Requested visibility' line prints to stdout without the logger prefix, twice per run

12h ago

#3111

A nested structural swamp that outlives the run that started it keeps skipping that run's lock while the holder writes

12h ago

#3110

extension push/quality: structured acceptances for agents, a clearer title than For next time, and an optional reason

12h ago

#3104

extension push: the undecided private-entitlement verdict still says the registry may start a trial at publish

13h ago

#3103

seed:collective cannot produce a superseded personal trial or a returned trial, so the UAT cannot cover them

13h ago

#3102

Replayable state fragments: Git-reviewable desired state for resources changed by model methods

13h ago

#3100

extension push: registry metadata and content hash label files relative to the repo dir, so a push with --extensions-dir records ../ paths

14h ago

#3095

registry: store and show declared acceptances sent in contentMetadata.acceptances at confirm (swamp-club#3021)

15h ago

#3088

extension push --dry-run: authentication verdict prints 'Signed in as .' when the credential is an API key with no username

17h ago

#3080

serve --remote-only: step output from workers is never persisted to the run log (history logs always empty)

18h ago

#3074

serve HA: worker enrollment on a replica that lacks the enrollment token's definition may create a second definition (unverified)

18h ago

#3073

serve HA: pull a token's definition on an auth miss so a peer accepts a new token at once

18h ago

#3069

extension push: version-drift check reports 'no previously published version found' when the registry call failed

19h ago

#3067

Search and list output follows directory iteration order in the definition, workflow, vault and output repositories

19h ago

#3065

extension push: accept attestation-backed adversarial-review evidence (lookup by extension and content hash), generated provenance, and carry the evidence to the registry at confirm (design: Lab #3023)

20h ago

#3062

Workflow load errors print the raw Zod issue dump instead of a readable message

21h ago

#3060

Decide a stricter naming rule for workflow step and job names

12h ago

#3046

serve: run scheduled workflows concurrently (per-workflow serialization), and report queue delay

1d ago

#3045

Show expired approval gates as expired, with a Cancel action in the dashboard

1d ago

#3041

digitalocean codegen leaves an orphaned security_secret.ts model file after its endpoint left the spec

1d ago

#3040

extension promote accepts a yanked version and reports it promoted

1d ago

#3036

Docs: doctor install reports a stale or failing autoupdate scheduler

1d ago

#3017

extension push output: double-quoted adjacent interpolations, repo-relative paths, and --json emits three JSON documents

1d ago

#3013

auto-resolve: the Installing line prints the extension's entire multi-line description

1d ago

#3010

s3-datastore/serve: readResource without a version returns an older version; the latest marker is rewritten backwards

2d ago

#3004

Publish agent-facing projections of the manual: /llms.txt, .md pages, and llms-full.txt

Team3d ago

#3000

Let agent-runner drive more agent CLIs (Kilo Code and others)

4d ago

#2998

s3-datastore: every fast-path miss re-downloads every _index shard, so a busy serve pulls the whole index every poll (~230 GB/day S3 egress)

4d ago

#2993

Feedback: Swamp as an approval-gated control plane for a small fleet

1d ago

#2965

extension safety analyzer's Deno.Command( warning is a plain substring match

4d ago

#2922

Tell timeouts apart from cancels in method-run records, and review the hidden 30 s fallback timer for step-called model methods

5d ago

#2908

Run from a git worktree: definitions from the worktree, state from the shared repo

5d ago

#2907

S3/GCS datastore: a push that fails after uploading drops its recorded deletes, so the retry brings deleted data back

5d ago

#2906

Tests: bring the in-memory remote's default semantics up to @swamp/s3-datastore and @swamp/gcs-datastore 2026.10.01.1

5d ago

#2900

Tell the pushing client when its extension contentMetadata fails validation

5d ago

#2892

Extension datastore: query still returns an item after its delete is pulled

5d ago

#2867

Cancel, reject and supersede of a parent run should settle its suspended nested child runs

6d ago

← Back to list

01Issue

BugShippedExtensionsPublic

Assigneesstack72

Relationships

#2999 s3-datastore: pull never removes files deleted on the remote, so a peer's next push re-uploads them and undoes `swamp data gc`

Opened by sntxrr · 10/3/2026· Shipped 10/6/2026

PR #463: https://git.swamp-club.com/swamp-club/swamp-extensions/pulls/463

Summary

When one peer runs swamp data gc against a shared @swamp/s3-datastore, the GC removes the files from S3 (delete markers) and from that peer's index shards. A second peer (here swamp serve) pulls afterwards, but its pull does not delete the GC'd files from its own local cache. The second peer's next push then uploads those files again and merges their index entries back into the shards, restoring the pre-GC state.

Measured (two peers on one namespace: a CLI host and a `swamp serve` container)

GC on the CLI host: 120,250 excess versions reclaimed. The home-ip model's index shard went 14.2 MB → 28 KB (38,252 → 80 entries).
8–12 min later, a scheduled workflow run on serve pushed. The same shard went 28 KB → 14.18 MB in a single write, restoring 38,172 entries.
Sampled restored files: each has the GC's delete markers (23:04–23:09Z) followed by a new current version (23:16–23:17Z). They were all present in serve's local cache and absent from the CLI host's.
Cache drift at that point: CLI host 89,436 files / 627 MB; serve 297,986 files / 2.5 GB. serve's sync-state showed localDirty: false, 0 dirty paths.
The same files show many earlier bursts of identical re-uploads (on 2026-10-02 at 17:58Z, 10:17Z and 01:02Z, on 2026-10-01 at 21:57Z and 04:00Z, four times on 09-30, and earlier). Each burst stores another full copy under bucket versioning. That roughly matches the bucket's ~50 GB/day storage growth and ~2M PUT/day.
Running the same swamp data gc --yes on serve itself (dry run: 118,031 excess versions) removed the stale copies: serve's cache dropped to 70,486 files / 872 MB and the index stayed down across the next scheduled runs.
- With SWAMP_DATASTORE_SYNC_TIMEOUT_MS=1800000. On the CLI host, the GC's push had timed out at the 300 s default.

Expected

A pull removes from the local cache any file whose index entry no longer exists remotely, or that has a remote delete marker. A file deleted by a peer should not reappear on the next push.
Alternatively, push should not upload files that were never dirty locally.

Also observed

Retention differs by peer: after GC, serve kept 10,096 home-ip files, while the CLI host kept about 139. serve's next push merged its set into the shard (28 KB → 3.4 MB). If both ran the same GC config, I'd expect the same retained set.

#2683: pullChanged overwrites dirty, unpushed cache files. Same area, opposite direction.
#2888: keep a failed push's changes dirty through a pull (shipped in 2026.10.01.1). This would have protected the timed-out GC push, but not this case.
Filed alongside: every fast-path miss downloads every _index shard. The two compound: re-bloated shards make every pull more expensive.

Environment

@swamp/s3-datastore 2026.09.24.1 at the time of the re-upload, now 2026.10.01.1 on both peers
swamp 20261002.202148.0-sha.0fc928c8 (serve, linux container); CLI on darwin aarch64
AWS S3, versioning enabled

Upstream repository: https://github.com/systeminit/swamp-extensions

Environment

Extension: @swamp/s3-datastore@2026.10.01.1
swamp: 20261002.194016.0-sha.c0c751a1
OS: darwin (aarch64)
Deno: 2.9.7
Shell: /bin/zsh

02Bog Flow

Shipped

10/6/2026, 7:34:53 PM

Click a lifecycle step above to view its details.

03Sludge Pulse

stack72 assigned stack7210/6/2026, 4:55:25 PM

system commented 10/3/2026, 12:36:03 AM

Classified automatically when this issue was filed.

Source: Extensions

If you feel this classification is incorrect, add a ripple to tell us so.

stack72 commented 10/6/2026, 7:34:59 PM

Thanks @sntxrr for reporting this! We shipped: Pull and push in both @swamp/s3-datastore and @swamp/gcs-datastore reconcile peer deletions: a local file whose index entry existed at the last sync, is gone from an authoritatively read remote index, and is unchanged locally (size plus mtime or sha256) is removed from the cache with its now-empty parent dirs, so a later push walk no longer re-uploads it and undoes swamp data gc. Locally modified copies are kept; nothing is removed when the remote index is cached, missing, synthetic, or empty.. The fix has been merged and a release is on its way. We appreciate your contribution to swamp.

Session-start 'swamp model search --json' in the AGENTS.md template is unfulfillable - wire it to a hook or drop it

Flaky property test: analyzeExpression dot/bracket agreement fails when the generated model name is a CEL keyword

Accept usernames for user subjects in grant files

Docs: enable-managed-config should say to re-run config migrate when its push fails

Docs: grant file reference shows a stale format and omits resources and subjects

Flaky property test: analyzeExpression hyphenated dot access fails when modelName generates a CEL keyword ("in")

extension quality and push crash with IsADirectory when additionalFiles lists a directory

extension push: a required global argument renders with a stray "" ("text": "string""")

Docs: extension publishing reference is stale on bare specifiers, quality output and fmt config (swamp-club#3022)

swamp-extensions: replace bare import specifiers with npm:/jsr: specifiers in first-party extensions

extension push: 'Requested visibility' line prints to stdout without the logger prefix, twice per run

A nested structural swamp that outlives the run that started it keeps skipping that run's lock while the holder writes

Show signal waits to remote clients and the dashboard

Settle expired signal waits from swamp serve

Resume signalled workflow runs automatically under swamp serve

extension push/quality: structured acceptances for agents, a clearer title than For next time, and an optional reason

extension push: the undecided private-entitlement verdict still says the registry may start a trial at publish

seed:collective cannot produce a superseded personal trial or a returned trial, so the UAT cannot cover them

Replayable state fragments: Git-reviewable desired state for resources changed by model methods

extension push: registry metadata and content hash label files relative to the repo dir, so a push with --extensions-dir records ../ paths

registry: store and show declared acceptances sent in contentMetadata.acceptances at confirm (swamp-club#3021)

extension push --dry-run: authentication verdict prints 'Signed in as .' when the credential is an API key with no username

serve --remote-only: step output from workers is never persisted to the run log (history logs always empty)

serve HA: worker enrollment on a replica that lacks the enrollment token's definition may create a second definition (unverified)

serve HA: pull a token's definition on an auth miss so a peer accepts a new token at once

extension push: version-drift check reports 'no previously published version found' when the registry call failed

Search and list output follows directory iteration order in the definition, workflow, vault and output repositories

extension push: accept attestation-backed adversarial-review evidence (lookup by extension and content hash), generated provenance, and carry the evidence to the registry at confirm (design: Lab #3023)

Workflow load errors print the raw Zod issue dump instead of a readable message

Decide a stricter naming rule for workflow step and job names

serve: run scheduled workflows concurrently (per-workflow serialization), and report queue delay

Show expired approval gates as expired, with a Cancel action in the dashboard

digitalocean codegen leaves an orphaned security_secret.ts model file after its endpoint left the spec

extension promote accepts a yanked version and reports it promoted

Docs: doctor install reports a stale or failing autoupdate scheduler

extension push output: double-quoted adjacent interpolations, repo-relative paths, and --json emits three JSON documents

auto-resolve: the Installing line prints the extension's entire multi-line description

s3-datastore/serve: readResource without a version returns an older version; the latest marker is rewritten backwards

Publish agent-facing projections of the manual: /llms.txt, .md pages, and llms-full.txt

Let agent-runner drive more agent CLIs (Kilo Code and others)

s3-datastore: every fast-path miss re-downloads every _index shard, so a busy serve pulls the whole index every poll (~230 GB/day S3 egress)

Feedback: Swamp as an approval-gated control plane for a small fleet

extension safety analyzer's Deno.Command( warning is a plain substring match

Tell timeouts apart from cancels in method-run records, and review the hidden 30 s fallback timer for step-called model methods

Run from a git worktree: definitions from the worktree, state from the shared repo

S3/GCS datastore: a push that fails after uploading drops its recorded deletes, so the retry brings deleted data back

Tests: bring the in-memory remote's default semantics up to @swamp/s3-datastore and @swamp/gcs-datastore 2026.10.01.1

Tell the pushing client when its extension contentMetadata fails validation

Extension datastore: query still returns an item after its delete is pulled

Cancel, reject and supersede of a parent run should settle its suspended nested child runs

Relationships

#2999 s3-datastore: pull never removes files deleted on the remote, so a peer's next push re-uploads them and undoes `swamp data gc`

Summary

Measured (two peers on one namespace: a CLI host and a swamp serve container)

Expected

Also observed

Related

Environment

Environment

Shipped

Measured (two peers on one namespace: a CLI host and a `swamp serve` container)