#3473 Retention: one timeframe, enforced automatically, after which nothing older remains anywhere in the datastore

Problem

Nothing in swamp deletes old history automatically. swamp run gc and swamp data gc only run when someone types them, autoGc only trims model data versions, and serve's background cleanup covers tokens, worker records and its audit store. So every datastore grows with its age.

On ops.swamp-club.com, 18 days produced 13,226 runs, 138k method outputs and 238k data files (2 GB). That slowed searches and writes until serve stalled: data writes reached a 21-minute median, a release build failed, and UAT runs timed out. A team that wants a bounded datastore has no way to get one.

Proposal

A single setting in .swamp.yaml:

retention: 30d

It's off unless set. When it's set, swamp guarantees that nothing older than the timeframe remains in the datastore, by any means of looking.

What it covers

• Everything with a timestamp:
  • workflow runs, evaluated snapshots, run and step logs;
  • method outputs;
  • all data versions, including a model's latest version (strict);
  • audit events;
  • worker bookkeeping (step leases, pending dispatches), signal and wait records.
• Every reference to those items:
  • catalog rows, per-workflow run indexes;
  • _index shards and _meta.json on remote datastores;
  • sync and dirty queues;
  • commit-log entries and content chunks that only expired items reference (#2865).
• Leftover files: files and objects that no index points to, left by crashes or interrupted writes. GC sweeps storage itself, not just the indexes.
• Remote copies: objects, and their old versions, on S3/GCS datastores.

Out of scope: model and workflow definitions, vault secrets and repo config. These are configuration, not history.

How it runs

• Serve applies it on a timer, at least hourly, like token GC.
• A plain CLI repo with no serve applies it at the end of a command, at most once per hour.
• It never deletes running or suspended runs, or anything they reference.
• It must not stall serve. It works in small batches, yields between them, backs off when serve is busy, and is traced. On ops, a manual run gc pushed worker data writes to minutes while it ran, and an automatic job must not do that.

What a reaped item looks like

• A watermark per kind. One record per kind (runs, outputs, data, audit), overwritten on each pass: "everything before <cutoff> was reaped by retention (30d), last pass <time>". It's constant-size, so it doesn't grow.
• Lookups say "reaped". history get, history logs, dashboard run pages and data reads of reaped items answer with that, not a generic not-found. A data.latest() whose only version was reaped reports reaped too.
• New run IDs are time-ordered (UUIDv7). Then "reaped" can be told apart from "never existed" without keeping any per-run record. Older UUIDv4 IDs get "not found; runs before <cutoff> have been reaped".
• Listings mark the cutoff. run search, history search and dashboard history end with "older runs reaped by retention before <cutoff>".
• Each pass writes one audit summary: cutoff, counts per kind, bytes freed. These expire under the same timeframe.

Proof

• A doctor check. swamp doctor datastore --retention scans storage and every index, and fails with an exact list if anything older than the cutoff exists. Serve reports the result in its health output.
• A swamp-uat test. Write history of every kind, crash some writes partway, apply retention, then require that a full scan finds nothing older than the cutoff, on filesystem, S3 and GCS.

Related

• #3408 (storage tiers): with tiers configured, "reaped" means "moved to the next tier", and retention applies to the last tier.
• #3407 (run search): that fixes read cost; this fixes size.
• #2865 (datastore rework) and #3051.