Lab #811: Could method-summary report artifacts get a default retention cap? They grow to dominate the datastore manifest

Thanks for the quick turnarounds on #666 and #788 — both landed fast and have already helped. Here's a follow-on I turned up while profiling that looks worth addressing on its own.

What I found

Every swamp model method run emits two auto-generated report artifacts — report-swamp-method-summary and report-swamp-method-summary-json — and each is retained as a new data version indefinitely. For high-frequency runners these accumulate steadily and end up dominating the datastore index that gets pulled/pushed under the lock on every write.

Profiling one busy namespace (related to #666): of 109,152 catalog rows / 156.6 MB tracked, only 13,884 rows (7.1 MB) were current state — ~95% was non-latest versions. The auto-generated method-summary artifacts alone were roughly half the manifest:

one frequently-run model: ~67.6 MB of method-summary versions
a polling model: ~37.8 MB
another: ~3.3 MB

These are generated automatically rather than written by the user, so an unbounded version history of per-run summaries is probably not what anyone wants.

Why it matters

It sits upstream of (and independent from) the lock-scoping in #666 and the gc/S3-delete fix in #788:

The full ~110 MB index is pulled/pushed under the lock on every write, so manifest size gates the lock-hold for all writers in the namespace — even a trivial vault write or a 7-entry gc pays the full-index cost.
swamp data gc is the natural mitigation, but on S3 datastores it prunes the catalog without deleting backing objects (#788), so it doesn't shrink the manifest today — and even once that's fixed, a default cap keeps the manifest from re-bloating between gc runs.
Ephemeral lifetime also doesn't appear to be implemented yet, so tagging these artifacts ephemeral isn't currently an escape hatch.

A few directions that could work

A default retention cap on the auto-generated method-summary artifacts (e.g. keep the last N versions or N days), configurable per repo.
Implementing ephemeral lifetime and defaulting these auto-generated artifacts to ephemeral, so they auto-collect.
An opt-out for a model to disable method-summary artifact emission where it isn't useful.

Whichever fits the architecture best — I can share more profiling data if it'd help. Thanks again.

Environment

swamp: 20260624.181631.0-sha.aa2ae00f
@swamp/s3-datastore@2026.06.24.1 (MinIO backend)
OS: darwin (aarch64)

New Release Discord notification links to GitHub Actions run instead of swamp changelog/release notes

/feed: filter and search by author, title/text, and topic

Unclear whether SWAMP_API_KEY auth claims a device's banked "ghost" reputation, or only swamp auth login does

Vault: native secret-read access audit trail

runModel(): options.arguments never reach the child's per-method arguments — merged into globalArgs, then dropped or rejected

runModel(): nested outputs record triggeredBy "manual" with no parentOutputId — docs promise model-trigger lineage

commitPush no-op verification fetches the full monolithic index instead of the v2 shard/commitSeq fast-path (follow-up to #1034)

Catalog export re-uploads the full monolithic object every sync, even with zero changes (sibling to #1034)

Shard-first v2 still re-uploads the full monolithic index + catalog export every commit; silent writeback failure above the 30s request timeout

S3 datastore push walks the entire cache root — nested/foreign namespace dirs become doubled index keys (recurring)

extension update/install reports new version while on-disk pulled files stay old

Orphaned data (migrated-away/deleted model) can't be GC'd or deleted

datastore migrate-index aborts on hardcoded 30s S3 requestTimeout for large shards

datastore migrate-index ignores the repo's configured namespace (migrates bucket-root index instead)

Intra-namespace write concurrency: whole-index sync under the lock serializes fan-out workloads (split from shipped #666)

Pre-sync pull aborts the whole command when an index entry's S3 object is missing — no built-in way to recover

Local model source edit may not change the bundle cache key — rebuilt bundle shadowed by an older same-hash copy on a shared S3 datastore

autoGc emits auto_gc_completed event on --json stdout, breaking single-parse consumers

Batch / prefix delete for swamp data delete (single lock acquisition)

Same-namespace writers fully serialize on the per-namespace lock — could maintenance/append writes avoid holding it?

Could method-summary report artifacts get a default retention cap? They grow to dominate the datastore manifest

Datastore should fail fast on unresolvable credentials instead of stalling on the AWS provider chain

Optional scheduled / automatic datastore GC (retention-policy-driven pruning)

Notify issue author/participants on ripples & status changes — with Discord bot DM as a delivery channel

Single __global__ datastore lock serializes unrelated writes across all repos/namespaces

swamp data gc prunes the catalog but never deletes objects from S3 datastores (markDirty hook not wired) — sync manifest never shrinks

data gc --dry-run ignores the flag and performs a destructive GC

Private / org-scoped collectives with member ACLs

Resident/warm worker mode — `model method run` has ~6s fixed per-invocation overhead that rules out latency-sensitive use

Homepage install command: wrong domain and missing https:// protocol

Homepage install command: wrong domain and missing https:// protocol

Gator-approved feed post did not trigger Discord activity or profile points

swamp CLI commands fail silently or hang when invoked from git worktrees via SWAMP_REPO_DIR

Many CLI commands acquire global .datastore.lock unnecessarily, causing 60s LockTimeoutError under any concurrent writer

Relationships

#811 Could method-summary report artifacts get a default retention cap? They grow to dominate the datastore manifest

What I found

Why it matters

A few directions that could work

Environment

Closed

Single global datastore lock serializes unrelated writes across all repos/namespaces