← Back to list
01Issue
BugShippedSwamp CLIPublic
AssigneesNone
Relationships
↔ sibling #3179#3178 Lazy hydration: content readers bypass hydration (stream, stale raw after append); add one ensuring read path and a fitness test
Opened by zack · 10/8/2026· Shipped 10/8/2026
Summary
Under hydrationStrategy: lazy the local cache can hold a data item's metadata without its content. Readers are repaired one at a time (getContent has a hydrate hook, data.latest() materialises the path), and there is no list of them, so some still assume the content is local. Found while working swamp-club#2435, which is blocked on this.
Readers that are wrong today
FileSystemUnifiedDataRepository.stream()(src/infrastructure/persistence/unified_data_repository.ts) catches NotFound and yields nothing, without calling the hydrate hook. The serve data plane (src/serve/data_plane.ts) streams artifacts to remote workers through it, so a worker gets an empty or truncated body for content that is not local.- A
rawfile that exists locally is never checked against its metadata.append()rewrites the latest version'srawat the same path and updatessizeinmetadata.yaml. A metadata-only pull brings the new metadata and skips theraw(the S3 and GCS extensions skip every lazily skippable file, whether or not a local copy exists), so the old bytes stay and are served as current.
Proposed change
- One content-ensuring step in the repository: the file is present and matches the size in its metadata, or it is downloaded through the hydrate hook.
getContent,streamand path materialisation all go through it. - An architecture fitness test in
integration/that pins the list of direct readers of content paths and ofgetContentSynccallers, using the ratchet pattern inarch_fitness_helpers.ts, so a new reader that bypasses the step fails a test instead of waiting for a review to find it.
Notes
- A fix for
stream()with a unit test exists on theissue-2435branch (commitac58584a) and can be reused. - Not verified: every place a content file path is handed to extension code, which may read it directly.
02Bog Flow
Shipped
Click a lifecycle step above to view its details.
03Sludge Pulse
Sign in to post a ripple.