Skip to main content
← Back to list
01Issue
BugShippedSwamp CLIPublicEnterprise
Assigneesstack72

Relationships

#458 swamp data gc does not remove physical version directories from disk

Opened by webframp · 5/27/2026· Shipped 5/27/2026

Description

After running swamp data gc --force, the GC reports cleaning excess versions (e.g. "version gc: 445 models with 8069 excess versions") and checkpoints the WAL, but the physical version directories remain on the filesystem. Disk usage does not decrease.

Steps to Reproduce

  1. Create a model with garbageCollection: 5 on a resource spec
  2. Run the same method 200+ times (writing to the same instance name each time)
  3. Observe 200+ version directories accumulate on disk
  4. Run swamp data gc --force
  5. GC reports success and excess version cleanup
  6. Check disk: all 200+ version directories still exist

Observed Behavior

$ swamp data gc --force
GC preview: 806 expired data items
version gc: 445 models with 8069 excess versions
GC complete: deleted 806 items
WAL checkpointed and truncated

$ find .swamp/data/@webframp/aws/securityhub-findings/MODEL_ID/INSTANCE/ -type d | wc -l
215   # still all versions on disk

$ du -sh .swamp/data/@webframp/aws/securityhub-findings/
143M  # unchanged

After GC, swamp data get correctly returns only the latest version (logical GC works). But the physical files for old versions remain, causing unbounded disk growth.

Expected Behavior

Either:

  • swamp data gc should delete physical version directories beyond the garbageCollection limit, OR
  • There should be a separate swamp data prune command for filesystem cleanup, OR
  • This is by-design (versions are retained for potential recovery) and should be documented

Impact

In a burn-in test with ~3,200 method executions over 10 hours against a single model instance, disk usage grew to 143MB despite garbageCollection: 5. For swamp serve deployments with frequent polling, this could become significant over time.

Environment

  • swamp: 20260525.222440.0-sha.6b09eef7
  • Extension: @webframp/aws/securityhub-findings (garbageCollection: 5, lifetime: 30m)
  • Platform: linux x86_64
  • Datastore: local (default)
02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 2 MOREREVIEW+ 3 MOREPR_MERGED+ 1 MORECONTRIBUTOR_NOTIFIED

Shipped

5/27/2026, 6:50:48 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack725/27/2026, 5:17:03 PM
Editable. Press Enter to edit.

webframp commented 5/27/2026, 3:23:05 PM

Don't know for certain if this is a valid problem, but the agent found it from a recent extended test so I sent for triage

stack72 commented 5/27/2026, 6:05:32 PM

Investigation & Reproduction Attempt

We investigated the GC code path thoroughly and attempted reproduction at multiple scales:

Code analysis:

  • Traced the full flow: CLI → deleteExpiredData (Phase 1: expired data, Phase 2: version GC) → collectGarbage → Deno.remove(versionDir, { recursive: true })
  • The collectGarbage method correctly computes versions to remove and calls Deno.remove for each
  • Existing integration tests verify filesystem deletion at small scale (6 versions)

Reproduction attempts (all on commit 50eb2502, same GC code as your version 6b09eef7):

  • 6 versions with gc=3 → GC correctly deleted 3 dirs ✓
  • 25 versions with gc=5 via deleteExpiredData end-to-end → correctly deleted 20 dirs ✓
  • 150 versions with gc=10 via CLI (swamp data gc --force) → correctly deleted 280 dirs ✓
  • 220 versions with gc=10 via CLI → correctly deleted 420 dirs ✓
  • Mixed scenario: expired data (Phase 1) + excess versions (Phase 2) → both phases worked correctly ✓

Could not reproduce the bug. Physical version directories were correctly removed in every scenario we tested.

What we DID find

The log renderer hides version GC results. The completed handler only displays dataEntriesExpired — the versionsDeleted and bytesReclaimed fields are completely absent from log output (they only appear with --json). This means there's zero visibility into whether version GC actually ran from the default CLI output.

What we're shipping

  1. Fix the log renderer to display version GC metrics (versionsDeleted, bytesReclaimed) alongside expired item counts
  2. Add end-to-end integration tests for the combined Phase 1 + Phase 2 deleteExpiredData flow with physical filesystem verification

Next steps for diagnosis

If the issue persists after upgrading, could you try running with --json to capture the exact versionsDeleted count?

swamp data gc --force --json

That will show whether the GC code reports versions as deleted but the filesystem disagrees, or whether the GC code itself reports 0 versions deleted. This will help narrow down whether the issue is in the deletion logic or something environment-specific (datastore config, filesystem permissions, concurrent swamp serve process).

stack72 commented 5/27/2026, 6:51:08 PM

Thanks @webframp for reporting this! The fix has been merged and a release is on its way. We appreciate your contribution to swamp.

Sign in to post a ripple.