#3408 Configurable storage tiers: runs, outputs, data versions and audit move through an ordered list of user-defined tiers, and stay queryable across them

Problem

History grows without limit, on disk and in every read path. Workflow runs and their evaluated snapshots, method outputs, data versions beyond each model's retention, and audit events all build up, and the only control is a manual, delete-only swamp run gc. Serve has no retention of its own except token GC and worker and bookkeeping GC. autoGc covers model data only.

So a team that wants to keep all history has to choose between deleting it and putting up with reads that slow down as it grows.

On ops.swamp-club.com (local filesystem datastore), after 18 days:

• 13,226 runs (239 MB). Most are from three high-frequency workflows: 5,196 from a 5-minute reconcile, 3,547 from an archive sync, and 2,448 from a webhook router at about 64 KB each.
• data/ holds 238k files (2 GB) and outputs/ 138k files.
• Because run search scans all history (#3407), serve slowed to the point where worker data writes reached a 21-minute median and a release build failed.

Proposed: user-defined tiers

There are no fixed tiers, just an ordered list. Each tier has:

• name: what queries and policies refer to.
• store: the local datastore, a local path, or any datastore type (GCS, S3, NFS, and MongoDB once it lands in #2865).
• after: the age at which items move into it. Optional rules can route by kind (runs, outputs, data versions, audit), workflow or model.
• defaultRead: whether default reads include it. Normally only the first tier does.
• index: each tier keeps its own index or manifest, partitioned by kind, time and workflow/model, so a query can skip what doesn't match without listing or downloading the tier.
• cache and fetchLimit: for slow remote tiers, a local read cache plus limits on how much a query may fetch.
• delete (optional, last tier only): without it, nothing is ever deleted.

tiers:
  - name: hot
    store: local
    defaultRead: true
  - name: warm
    store: local:.swamp/tiers/warm
    after: 7d
  - name: archive
    store: { type: '@swamp/gcs-datastore', bucket: example-archive, prefix: ops/ }
    after: 90d
    cache: 7d
  - name: deep
    store: { type: '@swamp/s3-datastore', bucket: example-deep, storageClass: GLACIER }
    after: 2y
tieringInterval: 1h
overrides:
  - workflow: github-event-router
    tiers: { warm: 1d, archive: 30d }

Behaviour

• One mechanism for runs (with their snapshots), method outputs, old data versions and audit events.
• Serve moves items on a timer, like token GC. Each move is a verified copy, and the item is removed from the previous tier only after that check.
• Read cost grows only with the default-read tiers. Everything else is physically outside their trees and indexes, so search, history, dashboard and data.latest stay fast however much history exists.
• Queries can span tiers: --tiers warm,archive, --tiers all, or a CEL tier field. Results stream, labelled with their tier, so fast tiers answer straight away while slow tiers keep arriving.
  • Slow-tier queries can run as a background job with an id to check later.
  • --dry-run estimates how many items and bytes a query would read, and per-tier fetchLimit caps it.
• A lookup by id checks the tiers in order and returns the item without moving it. Only an explicit restore brings an item back into a default-read tier.
• The default config is a single hot tier, which is today's behaviour, so nothing changes until a repo opts in.
• It builds on the datastore rework (#2865), so tiering is part of the commit-log design rather than a second storage path.

Alternatives considered

• swamp run gc with a short retention: loses history.
• Exporting from outside swamp (e.g. copying .swamp/ files to a bucket, then GC): unsupported, not queryable, and out of step with the datastore's own sync and catalog.
• Only fixing the run index (#3407): bounds search cost, but disk use and the other read paths still grow without limit.

Related: #3407, #2865, #3051, #3399, #3405.