Relationships
#2590 s3-datastore: a model-scoped pull still lists, walks and indexes the whole namespace
Opened by hammz · 9/28/2026
Defect
A model-scoped pullChanged (core passes context.models because the extension advertises scopedSync: true) still does work proportional to the whole namespace, not the model. Core runs this pull while holding the per-model lock, so on a large namespace a 40ms method holds the lock for ~70s and any concurrent run of the same model hits the 60s lock timeout (swamp-club#2553).
Where
datastore/s3/extensions/datastores/_lib/s3_cache_sync.ts, pullChanged:
- When
context.modelsis set it triespullPartitionedIndex(models), but falls back to the monolithicpullIndex({ forceRemote: true })when partition files are missing: the full index is downloaded. - The bulk-diff listing uses
bulkDiffPrefixes()(the whole namespace subtree) unlesssubdirsis set.context.modelsdoes not narrow it. - The walk over
this.index.entriesonly skips out-of-scope entries whensubdirsis set (scoped && !isInSubdirs(rel, subdirs)).context.modelsis ignored, so every index entry is considered for pruning and download.
Evidence
Reporter on swamp-club#2553: @swamp/s3-datastore 2026.09.24.1, one namespace with ~98k catalog rows. The "Syncing model ... from datastore..." step for a single model took ~66s before the method ran.
Expected
A pull with context.models lists, walks, prunes and downloads only that model's subtree (data/outputs/definitions for the given type/id) and reads only that model's index partition, so its cost is bounded by the model rather than the namespace. A model-scoped pull must, like a subdir-scoped pull, not advance the fast-path sidecar (see #2246).
Core-side mitigations (lock ordering, exit code) are tracked on swamp-club#2553. This issue is the extension half.
Upstream repository: https://github.com/systeminit/swamp-extensions
Environment
- Extension:
@swamp/s3-datastore@2026.09.24.1 - swamp:
20260928.150833.0-sha.6858940b - OS:
linux(x86_64) - Deno:
2.9.7 - Shell:
/usr/bin/zsh
Open
No activity in this phase yet.
system commented 9/28/2026, 4:09:52 PM
Classified automatically when this issue was filed.
- Source: Extensions
If you feel this classification is incorrect, add a ripple to tell us so.
Sign in to post a ripple.