Skip to main content
← Back to list
01Issue
BugOpenExtensionsPublic
AssigneesNone

Relationships

#2590 s3-datastore: a model-scoped pull still lists, walks and indexes the whole namespace

Opened by hammz · 9/28/2026

Defect

A model-scoped pullChanged (core passes context.models because the extension advertises scopedSync: true) still does work proportional to the whole namespace, not the model. Core runs this pull while holding the per-model lock, so on a large namespace a 40ms method holds the lock for ~70s and any concurrent run of the same model hits the 60s lock timeout (swamp-club#2553).

Where

datastore/s3/extensions/datastores/_lib/s3_cache_sync.ts, pullChanged:

  1. When context.models is set it tries pullPartitionedIndex(models), but falls back to the monolithic pullIndex({ forceRemote: true }) when partition files are missing: the full index is downloaded.
  2. The bulk-diff listing uses bulkDiffPrefixes() (the whole namespace subtree) unless subdirs is set. context.models does not narrow it.
  3. The walk over this.index.entries only skips out-of-scope entries when subdirs is set (scoped && !isInSubdirs(rel, subdirs)). context.models is ignored, so every index entry is considered for pruning and download.

Evidence

Reporter on swamp-club#2553: @swamp/s3-datastore 2026.09.24.1, one namespace with ~98k catalog rows. The "Syncing model ... from datastore..." step for a single model took ~66s before the method ran.

Expected

A pull with context.models lists, walks, prunes and downloads only that model's subtree (data/outputs/definitions for the given type/id) and reads only that model's index partition, so its cost is bounded by the model rather than the namespace. A model-scoped pull must, like a subdir-scoped pull, not advance the fast-path sidecar (see #2246).

Core-side mitigations (lock ordering, exit code) are tracked on swamp-club#2553. This issue is the extension half.

Upstream repository: https://github.com/systeminit/swamp-extensions

Environment

  • Extension: @swamp/s3-datastore@2026.09.24.1
  • swamp: 20260928.150833.0-sha.6858940b
  • OS: linux (x86_64)
  • Deno: 2.9.7
  • Shell: /usr/bin/zsh
02Bog Flow
◉OPEN○TRIAGED○IN PROGRESS○SHIPPED

Open

9/28/2026, 4:09:52 PM

No activity in this phase yet.

03Sludge Pulse
Editable. Press Enter to edit.

system commented 9/28/2026, 4:09:52 PM

Classified automatically when this issue was filed.

  • Source: Extensions

If you feel this classification is incorrect, add a ripple to tell us so.

Sign in to post a ripple.