Skip to main content
← Back to list
01Issue
FeatureShippedSwamp CLI
Assigneesstack72

Relationships

#823 Opt-in automatic garbage collection for datastore data

Opened by stack72 · 6/25/2026· Shipped 6/26/2026

Summary

Datastore data versions accumulate indefinitely because garbage collection only runs when the user manually invokes swamp data gc. For high-churn datastores — particularly those with frequent model method runs — this causes the catalog export (.catalog-export.json) to grow unboundedly, increasing lock-hold time on every write in the namespace.

This issue consolidates two related requests:

  • #806 — Optional scheduled/automatic datastore GC for duration-expired data on high-churn S3 datastores, so users don't have to hand-roll their own swamp data gc scheduler wrappers.
  • #811 — Auto-generated method-summary report artifacts (garbageCollection: 5, lifetime: 30d) dominate the catalog export because their GC policy is never enforced automatically. One profiled namespace had 109,152 catalog rows / 156.6 MB tracked, but only 13,884 rows (7.1 MB) were current state — ~95% was non-latest versions, with method-summary artifacts alone accounting for ~108 MB.

Both issues share the same root cause: GC policies are declared on data metadata but only enforced during manual swamp data gc runs. The catalog export serializes every row (all versions) to JSON on every push, so accumulated versions directly increase lock-hold time for all writers in the namespace.

Why the catalog export matters

Every datastore write triggers a push of .catalog-export.json, which is a full JSON dump of all catalog rows in the namespace. More accumulated versions → more rows → bigger export → longer lock-hold during push. Even unrelated writes (vault operations, small state updates) pay the full cost of the bloated export.

Proposed approach

Add an opt-in automatic GC mechanism that enforces existing GC policies (version-count caps like garbageCollection: 5 and duration-based retention like lifetime: 30d) without requiring manual intervention. This should be opt-in rather than default behavior, since automatic data deletion is a significant behavioral change.

Possible entry points:

  • Repository-level config (.swamp.yaml) — e.g. autoGc: true or autoGc: { afterMethodRun: true } to enable GC after method runs
  • Scheduled/periodic GC — a built-in scheduler or datastore-level config for periodic GC runs
  • Per-write inline GC — GC scoped to the specific data items just written, triggered after each write

Data and catalog entries must be pruned together to avoid orphaning data files on remote datastores.

Context from triage

During triage of #811, we explored several approaches:

  • Inline GC after report writes — effective but risky as a default; deletes data automatically on every method run
  • Latest-only catalog export — would shrink the export but orphans data files on remote datastores since the catalog no longer references non-latest versions
  • Overwrite instead of version — eliminates accumulation for reports but loses version history entirely

The opt-in auto-GC approach was chosen because it addresses both issues cleanly, keeps data and catalog in sync, and lets users control when automatic deletion is acceptable.

Environment

  • swamp: 20260624.181631.0-sha.aa2ae00f
  • Relevant to all datastore backends (local filesystem, S3)
02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 7 MOREREVIEW+ 3 MOREPR_MERGED+ 1 MORENOTIFICATION_SKIPPED

Shipped

6/26/2026, 9:47:52 AM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack726/25/2026, 9:57:07 PM
Editable. Press Enter to edit.

mgreten commented 6/26/2026, 2:02:44 AM

Excited for this one — thanks for picking it up. While profiling our own datastore to prep for auto-GC, I turned up something that might be worth folding into the design: swamp data gc currently doesn't collect ephemeral-lifetime data.

Concretely, on @swamp/s3-datastore@2026.06.24.1 (swamp 20260625.225837.0):

  • One of our models writes a small per-call result artifact declared lifetime: "ephemeral" with garbageCollection: 50.
  • These accumulated to 2,559 distinct ephemeral artifacts that were never collected.
  • A swamp data gc --dry-run reported dataEntriesExpired: 8350 and would reclaim ~128 MB — but 0 of those 2,559 ephemeral artifacts were in the expired set. GC reclaimed expired duration data and version history, but left the ephemeral artifacts entirely untouched.

So today ephemeral doesn't appear to translate into anything GC acts on — an ephemeral artifact lives as long as a lifetime: infinite one. If auto-GC is going to be the mechanism that keeps high-churn datastores lean, it might be worth having it (or data gc) treat ephemeral as collectable — e.g. collect ephemeral artifacts not referenced by the latest run, or honor a max-age/garbageCollection count for them. That would let models opt unimportant per-run scratch data out of the index cleanly, which is exactly the accumulation pattern that bloats the catalog export.

Totally possible this is already in scope for #823 — just flagging the measurement in case it's useful for the design. Happy to share more profiling detail if it'd help, and thanks again for the auto-GC work.

mgreten commented 6/26/2026, 5:21:15 PM

Thanks for shipping this — picked it up right away and I'm hoping it'll help, but I want to make sure I'm enabling it correctly before I trust it.

I set autoGc: true as a top-level key in .swamp.yaml (it sits as a sibling of gitignoreManaged / swampSha, and it survives swamp repo upgrade un-stripped, so it looks recognized). Both my writer machines are on 20260626.102849.0 with the key set.

But I can't tell whether it's actually firing. Peeking at the binary, the GC call that emits auto_gc_completed looks gated on a per-invocation input.autoGc rather than directly on the repo-config field — and swamp model method run doesn't seem to expose an --auto-gc flag. So I'm unsure whether the top-level autoGc: true config propagates into that per-run input automatically, or whether it needs to be set some other way.

Two quick questions when you have a moment (no rush):

  1. Is top-level autoGc: true in .swamp.yaml the intended enable, or does it belong somewhere else (e.g. nested under datastore:, or per-model)?
  2. Is there a way to confirm it's running — a log line / event at a particular log level, or a field in the method-run output I should look for? I tried watching --log output for auto_gc_completed but my namespace is busy enough that I couldn't get a clean observation.

Context for why I care: this is the high-churn / catalog-bloat namespace from the earlier profiling (the one where the index dominates the per-write lock-hold). Auto-GC keeping that index from re-growing between runs is exactly what I'm after. Happy to share whatever would help confirm it. Thanks again.

Sign in to post a ripple.