#3914 Annotate and retain past workflow runs and data after the fact

Problem

You often learn that a run matters only after it finishes. An access-review audit becomes the baseline for the next review. A one-off change workflow becomes the change record someone asks for a month later. By then swamp offers no way to mark either one.

Tags are the only labels, and they are set when a run starts (behavior run --tag, workflow run --tag, or tags in a definition). Nothing adds a label or a note to a finished workflow run, a method run, or a data version. workflow history has get, search, logs and outputs. The data commands read, rename, delete and collect. The one annotate command, swamp vault annotate, works only on vault secrets.

Retention removes the record you would have labelled. swamp run gc deletes workflow runs and method outputs older than 30 days by default. A resource's garbageCollection count drops old data versions as new ones arrive. To find a result again you have to keep its run ID somewhere outside swamp, and that ID stops resolving after the next gc.

Example

An operator grants a role on two resource groups through an approval-gated workflow, then runs a read-only RBAC audit workflow to record the result. Both runs succeed. A day later the operator wants the audit marked as the post-change baseline and the grant run linked to the change ticket. Neither is possible, and the audit snapshot will age out after about ten more audits.

Proposal

Add annotations, kept apart from tags and modelled on vault annotate.

• swamp workflow history annotate <run> --label key=value --note "...", with the same for method runs.
• swamp data annotate '<CEL predicate>' --label ... --note ... applies one annotation to every data version the predicate matches, such as all the data a run produced.
• Annotations do not create a data version or change stored content. They live in their own field (annotations.labels.*, annotations.notes) and queries can filter on them: swamp data query 'annotations.labels.change == "CHG-123"'.
• An optional retain marker (--retain or a separate retain command) exempts the run or matched versions from run gc and version-count GC until removed.
• The audit log records who added or removed an annotation or retain marker, and when.

Open questions (annotations)

1. Where annotations attach. To the run, to each data version, or to both, with run annotations inherited by the data that run produced? Inheritance keeps one call per run but adds a resolution rule to every query.
2. Mutability. Can an annotation be edited or removed, or only added to, so the full history stays visible?
3. Retain lifecycle. Does a retain marker expire, or last until someone removes it? Should run gc and data gc report what they skipped and why? Does retaining a run also retain the data, logs and files it produced?
4. Permissions under swamp serve. Is annotating a new grant action (annotate, and separately retain), or implied by write the way tag is today? Retain decides what survives GC, so it may need its own explicit grant.
5. Datastore model. In the v3 datastore rework (#2865) every change is a commit to a partition. Is an annotation a commit in the owning partition, or its own append-only partition? How do annotations sync to S3 and GCS datastores?
6. Size and abuse limits. What caps apply to label count, note length and the number of versions one predicate may annotate?
7. Interaction with data rename, data delete and data prune. Do annotations follow a renamed instance? Does a retain marker block delete, or only GC?

Open questions if this were done with tags instead

Making tags editable after the fact looks simpler, since tags are already queryable. It changes several assumptions, and I don't think it should be the default answer.

1. Immutable versions. Each data version is written once. If its tags can change, is the version still immutable? Does a retag create a new version, changing the version count and GC? Or does it change metadata in place, so the same version reads differently over time?
2. Access control. Under swamp serve, tags feed authorization. Data is judged on its owner's current tags, and --tag-requires-explicit-grant exists because whoever can retag can satisfy any tag-conditioned data: grant. Retagging past data or runs extends that power to records already written. A principal could tag an old audit to read it, or untag it to hide it. Does a retag need tag on both the old and new values, as model edit does now?
3. Owner tags versus data tags. Read grants are judged on the behavior's or workflow's tags, not each version's. Would retagging a version have any effect on access, or only on queries? Two kinds of tag with different rules would be harder to reason about than a separate annotation field.
4. Query and history stability. workflow history search --filter and data query 'tags.*' return results that hold over time. With mutable tags, a saved query or report could return different sets without any new run. Should reports record the tag state they read?
5. Reserved keys. _swamp.* tag keys are reserved, and runs refuse them. What stops a retag from adding or removing one, and does anything rely on current _swamp.* tags staying fixed?
6. Provenance. Tags set at run time record the operator's intent at that moment. Retagged values mix intent at run time with later reinterpretation. Should the original tags be kept and shown beside the current ones?
7. Retention. Tags do not affect GC today. If a tag could exempt data from GC, anyone allowed to tag could also control retention, which joins two permissions that are separate now.

My read is that annotations plus a separate retain marker answer the audit and change-record need without touching immutability or tag-based access. Let's weigh the trade-offs before settling on either design.