Relationships
#4943 Store large values once: content-addressed sharing across resources and versions
Opened by zack · 11/20/2026
Problem
Behaviors that snapshot state naturally repeat the same large values: the same multi-MB string embedded in a state resource, in every history entry, in a handoff copy and in each new version. Swamp stores and reloads every copy independently. In our case about 5 MB of distinct data was stored as several hundred MB per version, because one 3.6 MB text value appeared dozens of times across six resources. Storage, query output and parse memory all scale with the number of copies, not with the distinct data.
Proposal
- Store string (or subtree) values above a size threshold out of line, keyed by content hash,
shared across all resources and versions of a repository's data, with garbage collection
through the existing
swamp data gc. - On read, either return a shared single instance per distinct value within a run, or return a handle that can be resolved or streamed on demand (combining well with partial reads from the feature above).
- Keep this transparent to existing behaviors: written values round-trip unchanged.
Expected outcome
Storage and memory scale with distinct data. Redundant snapshots stay cheap, so a design mistake like ours degrades gracefully instead of making a lifecycle unrecoverable.
Open
No activity in this phase yet.
Sign in to post a ripple.