Skip to main content
← Back to list
01Issue
BugOpenExtensionsPublic
AssigneesNone

Relationships

#3198 s3-datastore: model-scoped pullChanged lists and walks the whole namespace

Opened by stack72 · 10/8/2026

Problem

@swamp/s3-datastore reports scopedSync: true (datastore/s3/extensions/datastores/_lib/s3_cache_sync.ts:4615), but a model-scoped pullChanged still does work proportional to the whole namespace. The scope is applied only to the index read.

swamp core takes a scoped pull for every workflow step. After taking the step's model lock, acquireModelLocks (src/cli/repo_context.ts:2654 in swamp) calls pullChanged({ context: { models: [{ modelType, modelId }] } }). This pull runs on the serve host before any worker dispatch, so worker-placed steps pay it too.

Inside the S3 slow path, with context.models set:

  • Remote listing is namespace-wide. listPrefixes is bulkDiffPrefixes() (around :2570), built from every index entry. Each step pays a full listAllObjects of the namespace (1,000 keys per call).
  • The walk covers every index entry. The loop around :2612-2690 iterates this.index.entries for all models: a Deno.stat on each, plus a sha256 when the mtime differs.
  • A model with no data shard reads the whole index. If pullPartitionedIndex finds no data-- shard, it falls back to readWholeIndex() (around :2526), which assembles every shard.
  • The fast path is off under lazy hydration. skipFastPath = this.lazyPullActive && !metadataOnly (around :2446). lazyPullActive is cleared only by an unscoped, non-subdir pull, and step pulls are scoped.

Impact

Reported in swamp-club#3046 (serve 20261003.192630.0, s3-datastore on Ceph RGW, 7 scheduled workflows): median pull of 32.3 s before a step dispatch and push of 5.0 s after it, so a run with 1 s of work takes 40-110 s. The split of those 32 s between this pull and waiting on serve's sync gate (swamp.serve.sync_gate.wait span) is not yet known.

Proposed approach (measure first, no regressions)

  1. Benchmark the scoped step pull before changing anything. Use datastore/benchmarks/s3_bench.ts at roughly 50 models x 20 versions, namespaced, v2 shard-first, and record the existing tracePhase breakdown: fastpath, pullIndex, remoteListing, walk, download.
  2. Change only the phases the numbers show matter. The safest first step is narrowing the remote listing to the model's data/, outputs/ and definitions-evaluated/ prefixes, and pruning only in-scope entries against it.
  3. Keep these as they are, because other behaviour depends on them:
    • The walk: lazy-hydration users get their bulk raw download on the first step pull from it.
    • The no-data-shard whole-index fallback: it refreshes other models' entries and keeps the swamp-club#2999 peer-delete reconciliation timing. Do not use the _meta.json partitions list to skip it: a failed compare-and-swap can leave committed shards unlisted.
    • The #2242 rule: a partial or failed listing never drives pruning.
  4. Prove no regression. Compare the released extension with the changed one across {full, lazy} x {namespaced, root} x {v2, v1} x {CLI run, serve run}. Cover a normal step, a model with no data shard, a peer deleting another model's data, the first lazy step, and cross-model CEL reads. Compare the cache tree, the index and command output.

GCS (gcs_cache_sync.ts) has the same pullChanged shape. It is out of scope here.

02Bog Flow
◉OPEN○TRIAGED○IN PROGRESS○SHIPPED

Open

10/8/2026, 9:34:33 PM

No activity in this phase yet.

03Sludge Pulse
Editable. Press Enter to edit.

system commented 10/8/2026, 9:34:33 PM

Classified automatically when this issue was filed.

  • Source: Extensions

If you feel this classification is incorrect, add a ripple to tell us so.

Sign in to post a ripple.