Relationships
#3198 s3-datastore: model-scoped pullChanged lists and walks the whole namespace
Opened by stack72 · 10/8/2026
Problem
@swamp/s3-datastore reports scopedSync: true (datastore/s3/extensions/datastores/_lib/s3_cache_sync.ts:4615), but a model-scoped pullChanged still does work proportional to the whole namespace. The scope is applied only to the index read.
swamp core takes a scoped pull for every workflow step. After taking the step's model lock, acquireModelLocks (src/cli/repo_context.ts:2654 in swamp) calls pullChanged({ context: { models: [{ modelType, modelId }] } }). This pull runs on the serve host before any worker dispatch, so worker-placed steps pay it too.
Inside the S3 slow path, with context.models set:
- Remote listing is namespace-wide.
listPrefixesisbulkDiffPrefixes()(around :2570), built from every index entry. Each step pays a fulllistAllObjectsof the namespace (1,000 keys per call). - The walk covers every index entry. The loop around :2612-2690 iterates
this.index.entriesfor all models: aDeno.staton each, plus a sha256 when the mtime differs. - A model with no data shard reads the whole index. If
pullPartitionedIndexfinds nodata--shard, it falls back toreadWholeIndex()(around :2526), which assembles every shard. - The fast path is off under lazy hydration.
skipFastPath = this.lazyPullActive && !metadataOnly(around :2446).lazyPullActiveis cleared only by an unscoped, non-subdir pull, and step pulls are scoped.
Impact
Reported in swamp-club#3046 (serve 20261003.192630.0, s3-datastore on Ceph RGW, 7 scheduled workflows): median pull of 32.3 s before a step dispatch and push of 5.0 s after it, so a run with 1 s of work takes 40-110 s. The split of those 32 s between this pull and waiting on serve's sync gate (swamp.serve.sync_gate.wait span) is not yet known.
Proposed approach (measure first, no regressions)
- Benchmark the scoped step pull before changing anything. Use
datastore/benchmarks/s3_bench.tsat roughly 50 models x 20 versions, namespaced, v2 shard-first, and record the existingtracePhasebreakdown: fastpath, pullIndex, remoteListing, walk, download. - Change only the phases the numbers show matter. The safest first step is narrowing the remote listing to the model's
data/,outputs/anddefinitions-evaluated/prefixes, and pruning only in-scope entries against it. - Keep these as they are, because other behaviour depends on them:
- The walk: lazy-hydration users get their bulk raw download on the first step pull from it.
- The no-data-shard whole-index fallback: it refreshes other models' entries and keeps the swamp-club#2999 peer-delete reconciliation timing. Do not use the
_meta.jsonpartitions list to skip it: a failed compare-and-swap can leave committed shards unlisted. - The #2242 rule: a partial or failed listing never drives pruning.
- Prove no regression. Compare the released extension with the changed one across {full, lazy} x {namespaced, root} x {v2, v1} x {CLI run, serve run}. Cover a normal step, a model with no data shard, a peer deleting another model's data, the first lazy step, and cross-model CEL reads. Compare the cache tree, the index and command output.
GCS (gcs_cache_sync.ts) has the same pullChanged shape. It is out of scope here.
Open
No activity in this phase yet.
system commented 10/8/2026, 9:34:33 PM
Classified automatically when this issue was filed.
- Source: Extensions
If you feel this classification is incorrect, add a ripple to tell us so.
Sign in to post a ripple.