Relationships
#1730 pullChanged never records commitSeq for shard-first (v2) index
Opened by magistr · 8/19/2026
Summary
When using shard-first (v2) index, pullChanged assembles the index from shards and sets indexETag = null (s3_cache_sync.ts:1685). The sidecar update at line ~1824 is guarded by if (indexETag), so markSynced is never called on the shard path. No equivalent commitSeq recording happens.
The sidecar can ONLY be updated via the push path (pushChanged), but that path is gated on localHasAllRemoteEntries() (see companion issue). Together these two bugs form a vicious cycle: pull never writes commitSeq, push can't write commitSeq on partial caches, so the fast path is permanently dead.
Impact
The tryCommitSeqFastPath() probe at the top of pullChanged always fails because v2.commitSeq === undefined in the sidecar. Every pull takes the full slow path: read _meta.json + all shard files from S3 + rebuild 129k-entry index + rewrite 50MB local index file.
Suggested fix
After successful shard assembly in pullChanged, record the assembled commitSeq in the sidecar:
const assembled = await this.assembleIndexFromShards(signal);
if (assembled) {
this.index = { version: 1, lastPulled: ..., entries: assembled.entries };
this.scrubIndex();
await atomicWriteTextFile(this.indexPath, ...);
indexETag = null;
// NEW: record commitSeq for pull fast-path
try {
const sidecar = this.buildV2State();
sidecar.commitSeq = assembled.commitSeq;
await this.writeSyncState(sidecar);
} catch { /* non-fatal */ }
}References
s3_cache_sync.ts:1672-1688(shard assembly in pullChanged)s3_cache_sync.ts:1824(if (indexETag)guard skipping markSynced)s3_cache_sync.ts:1207-1220(tryCommitSeqFastPath— checks sidecar.commitSeq)
Upstream repository: https://github.com/systeminit/swamp-extensions
Environment
- Extension:
@swamp/s3-datastore@2026.08.12.1 - swamp:
20260819.011806.0-sha.a9c7ee9a - OS:
darwin(x86_64) - Deno:
2.8.3 - Shell:
/bin/zsh
Closed
No activity in this phase yet.
stack72 commented 8/27/2026, 4:48:38 PM
Hey @magistr, the observation that the shard assembly path sets indexETag = null and therefore skips the if (indexETag) sidecar update is correct at a surface level. However, pushChanged runs immediately after pullChanged in the same command lifecycle and performs its own shard assembly — it can and does record commitSeq when localHasAllRemoteEntries() passes.
So the claim that commitSeq is "never recorded" isn't accurate — it's recorded by the push path when the local cache is complete. The real issue here is the double shard assembly (which you filed separately as #1731), not a missing write.
Line references are also off by ~14 lines from the current source. Closing this — the behavior is working as designed.
Sign in to post a ripple.