Skip to main content
← Back to list
01Issue
BugClosedSwamp CLIPublic
AssigneesNone

Relationships

#1730 pullChanged never records commitSeq for shard-first (v2) index

Opened by magistr · 8/19/2026

Summary

When using shard-first (v2) index, pullChanged assembles the index from shards and sets indexETag = null (s3_cache_sync.ts:1685). The sidecar update at line ~1824 is guarded by if (indexETag), so markSynced is never called on the shard path. No equivalent commitSeq recording happens.

The sidecar can ONLY be updated via the push path (pushChanged), but that path is gated on localHasAllRemoteEntries() (see companion issue). Together these two bugs form a vicious cycle: pull never writes commitSeq, push can't write commitSeq on partial caches, so the fast path is permanently dead.

Impact

The tryCommitSeqFastPath() probe at the top of pullChanged always fails because v2.commitSeq === undefined in the sidecar. Every pull takes the full slow path: read _meta.json + all shard files from S3 + rebuild 129k-entry index + rewrite 50MB local index file.

Suggested fix

After successful shard assembly in pullChanged, record the assembled commitSeq in the sidecar:

const assembled = await this.assembleIndexFromShards(signal);
if (assembled) {
    this.index = { version: 1, lastPulled: ..., entries: assembled.entries };
    this.scrubIndex();
    await atomicWriteTextFile(this.indexPath, ...);
    indexETag = null;
    // NEW: record commitSeq for pull fast-path
    try {
        const sidecar = this.buildV2State();
        sidecar.commitSeq = assembled.commitSeq;
        await this.writeSyncState(sidecar);
    } catch { /* non-fatal */ }
}

References

  • s3_cache_sync.ts:1672-1688 (shard assembly in pullChanged)
  • s3_cache_sync.ts:1824 (if (indexETag) guard skipping markSynced)
  • s3_cache_sync.ts:1207-1220 (tryCommitSeqFastPath — checks sidecar.commitSeq)

Upstream repository: https://github.com/systeminit/swamp-extensions

Environment

  • Extension: @swamp/s3-datastore@2026.08.12.1
  • swamp: 20260819.011806.0-sha.a9c7ee9a
  • OS: darwin (x86_64)
  • Deno: 2.8.3
  • Shell: /bin/zsh
02Bog Flow
✓OPEN○TRIAGED○IN PROGRESS◉CLOSED

Closed

8/27/2026, 4:48:38 PM

No activity in this phase yet.

03Sludge Pulse
Editable. Press Enter to edit.

stack72 commented 8/27/2026, 4:48:38 PM

Hey @magistr, the observation that the shard assembly path sets indexETag = null and therefore skips the if (indexETag) sidecar update is correct at a surface level. However, pushChanged runs immediately after pullChanged in the same command lifecycle and performs its own shard assembly — it can and does record commitSeq when localHasAllRemoteEntries() passes.

So the claim that commitSeq is "never recorded" isn't accurate — it's recorded by the push path when the local cache is complete. The real issue here is the double shard assembly (which you filed separately as #1731), not a missing write.

Line references are also off by ~14 lines from the current source. Closing this — the behavior is working as designed.

Sign in to post a ripple.