#1442 First-class scheduled garbage collection policy per datastore namespace

Summary

Make garbage collection a first-class, declarative datastore namespace policy that swamp serve schedules and observes. The policy should cover both existing collectors:

• swamp data gc for versioned datastore data
• swamp run gc for workflow-run history and model-method outputs

Today these are disconnected. autoGc: true incrementally enforces data policies after writes and model runs, while run/output retention remains manual-only. Operators must discover both mechanisms, create external scheduling, and remember to maintain it. High-frequency workflows can therefore make a repository unusable before the default manual retention window is reached.

Incident that motivated this

A valid workflow scheduled every five minutes accumulated 1,528 workflow runs in roughly six days. The local .swamp datastore reached 51 GB:

• 28 GB in workflow-run records
• 2.9 GB in model-method outputs
• 20 GB in versioned data

swamp serve then failed during startup with a fatal V8 out-of-memory error near the 4 GiB heap limit. Even swamp workflow history get <workflow> --json reproduced the OOM. Extensions were healthy and the workflow validated successfully.

swamp data gc --dry-run correctly found nothing eligible because workflow runs and outputs are outside its scope. The default swamp run gc retention of 30 days also found nothing because the repository was only six days old. Running swamp run gc --older-than 1d required a 16 GiB heap and a long scan, but eventually deleted 2,274 run records and 3,575 outputs and reclaimed 27.3 GB. Keeping one day still left 4.8 GB of run history, enough for the default-heap history command to OOM.

This is partly a reader scalability bug, related to #1173, but automatic retention would prevent ordinary swamp serve users from unknowingly reaching this state.

Existing pieces

• #823 shipped opt-in autoGc for datastore data.
• #1172 shipped the manual swamp run gc command.
• #1200 explicitly documents that automated run/output retention is still a planned follow-up.

The missing piece is one policy and lifecycle for all datastore-owned garbage.

Proposed behavior

Add a validated GC policy to repository/datastore namespace configuration, for example:

datastore:
  garbageCollection:
    schedule: "0 * * * *"
    data:
      enabled: true
    runs:
      olderThan: 24h
    outputs:
      olderThan: 24h

The exact schema is open for design. Important semantics:

1. swamp serve registers GC as built-in maintenance, not as a user-authored workflow or shell wrapper.
2. Policy is scoped to the datastore namespace so every client uses the same retention contract.
3. Data GC continues honoring model/report lifetime and version policies.
4. Run/output GC uses explicit age or count retention and always protects active, pending, and suspended work.
5. Only one server performs maintenance for a shared namespace through the existing datastore lock/lease mechanism.
6. Failures are isolated from user workloads and surfaced as structured status rather than silently ignored.
7. A server restart does not create a catch-up storm; at most one overdue maintenance pass runs.
8. Manual swamp data gc and swamp run gc remain available and use the same policy when explicit flags are omitted.

Operability

Expose the policy and runtime state through commands such as swamp datastore status --json or a dedicated GC status command:

• enabled/disabled
• effective namespace policy
• last attempt and last success
• next scheduled run
• entries and bytes reclaimed
• duration and error details

Provide preview and validation before enabling automatic deletion. A generated configuration should choose conservative defaults and explain expected storage based on workflow frequency.

Onboarding and agent guidance

Add a CLI generator, such as swamp datastore gc policy init, or extend datastore setup to ask about retention. Update the swamp agent skill to inspect workflow schedules and current datastore growth, recommend a policy, generate valid configuration, preview its effect, and explain the distinction between data GC and run/output GC.

The skill should improve discovery and configuration, but the feature must remain first-class CLI behavior with a schema, locking, status, and safety guarantees. Garbage collection should not depend on every user or agent independently inventing a maintenance workflow.

Acceptance criteria

• A namespace can declaratively configure periodic data, run-history, and output GC.
• swamp serve safely schedules and executes that policy.
• Multiple servers sharing a namespace cannot run conflicting GC passes.
• Active/pending/suspended work is never removed.
• Effective policy, last/next execution, reclaimed bytes, and failures are inspectable.
• Configuration has validation, preview, documentation, and agent-skill guidance.
• GC implementations scan incrementally or stream metadata so cleanup itself does not require heap proportional to retained history.
• Existing autoGc and manual commands have a documented migration/compatibility path.