Skip to main content
← Back to list
01Issue
FeatureShippedExtensions
Assigneesstack72

Relationships

#906 Shard-first index: eliminate monolithic .datastore-index.json from commitPush critical section

Opened by stack72 · 7/1/2026· Shipped 7/1/2026

Summary

Follow-up to #812 and #829 (two-phase sync). The two-phase sync moved file uploads outside the global lock, but commitPush still reads/merges/writes the full monolithic .datastore-index.json under the lock. On a high-churn namespace with a ~98 MB index, this takes 26–35s — enough that only ~2 concurrent writers fit within the 60s lock timeout.

Design

A full design doc has been written at design/shard-first-index.md in the core repo (PR incoming). The design covers:

  • Extend partitioning to all datastore subdirectories — not just data/. Per-model shards for data/, outputs/, definitions-evaluated/; per-workflow for workflow-runs/, workflows-evaluated/; single shard for low-cardinality dirs (audit/, telemetry/, etc.)
  • Make _index/ shards the sole source of truth — commitPush reads/merges/writes only the touched shard(s) under the lock. Lock-hold time drops from ~30s to sub-second.
  • Replace monolith ETag fast path with a commitSeq counter in _meta.json (version 2 format)
  • Unscoped pulls assemble from shards — read _meta.json, fetch all shards in parallel
  • Backward compatibility — phase 1 dual-writes monolith + shards; phase 2 (future major version) drops the monolith
  • Migration — first shard-first commit partitions the existing monolith; _meta.json version field gates the transition
  • Recovery — self-healing _meta.json rebuild from S3 ListObjects; per-shard fallback to monolith

Scope

This is entirely @swamp/s3-datastore extension work — no core CLI changes needed. The core's DatastoreSyncService interface is unchanged; the PushManifest is opaque and the extension can embed partition keys in it during preparePush.

User-reported measurements (mgreten, #812)

Setup: CLI 20260701.044223.0, @swamp/s3-datastore@2026.06.30.2, MinIO, ~98 MB index, 4 concurrent writers:

  • Before two-phase sync: 1 succeed + 3 LockTimeout
  • After two-phase sync: 2 succeed + 2 LockTimeout
  • Expected after shard-first: 4 succeed + 0 LockTimeout (lock-hold drops from ~30s to sub-second)

Implementation sequence

  1. Extend partitionKeyFromPath to cover all datastore subdirectories
  2. Add commitSeq to _meta.json (version 2 format)
  3. Implement shard-first commitPush: read/merge/write touched shards + bump commitSeq under lock; dual-write monolith outside lock (phase 1)
  4. Update zero-diff fast path to use commitSeq instead of monolith ETag
  5. Update unscoped pullChanged to assemble from shards when _meta.json version is 2
  6. Add _meta.json self-healing recovery
  7. Test all dimensions (see design doc)
  8. Phase 2 (future): stop writing monolith, add cleanup tooling
  • #812 — original issue (intra-namespace lock contention)
  • #829 — two-phase sync (narrowed critical section to index merge)
  • #666 — per-namespace lock split
  • #520 — original LockTimeout cascade

Upstream repository: https://github.com/systeminit/swamp-extensions

Environment

  • Extension: @swamp/s3-datastore@2026.06.03.2
  • swamp: 20260701.044223.0-sha.8d598bd4
  • OS: darwin (aarch64)
  • Deno: 2.8.3
  • Shell: /bin/zsh
02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 5 MOREREVIEW+ 4 MOREPR_MERGED+ 1 MORENOTIFICATION_SKIPPED

Shipped

7/1/2026, 9:21:05 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack727/1/2026, 6:04:48 PM
Editable. Press Enter to edit.

stack72 commented 7/1/2026, 6:01:23 PM

Design Doc: Shard-First Index

Design for eliminating the monolithic .datastore-index.json in the @swamp/s3-datastore extension and replacing it with a fully partitioned shard-based index under _index/.

Motivation

The two-phase sync (swamp-club#829) moved file uploads outside the global lock, narrowing the critical section to the index merge in commitPush. But commitPush still reads, parses, merges, and rewrites the full monolithic .datastore-index.json under the lock. On a high-churn namespace with a ~98 MB index, this takes 26–35 seconds — enough that only ~2 concurrent writers fit within the 60-second lock timeout.

User-reported measurements (@swamp/s3-datastore@2026.06.30.2, MinIO, 4 concurrent writers to one namespace):

  • Before two-phase sync: 1 succeed + 3 LockTimeout
  • After two-phase sync: 2 succeed + 2 LockTimeout

The improvement is real — file uploads no longer hold the lock — but lock-hold time is still proportional to the total index size, not the write size.

Current State

Monolithic index: .datastore-index.json is a JSON object mapping every relative path in the bucket to its size, mtime, and optional SHA-256 hash. It is the source of truth for both pull and push change detection.

Existing partition shards: The extension already writes per-model partition shards under _index/:

  • _index/{partitionKey}.json — entries for one model's data
  • _index/_meta.json — list of all partition keys

Partition key derivation: partitionKeyFromPath(rel) extracts a key from data/{modelType}/{modelId}/... paths. Only data/ entries have partition keys; all other subdirectories return undefined and are not partitioned.

Current usage:

  • Pull (scoped): pullPartitionedIndex() reads specific model shards, falls back to the monolith if any shard is missing.
  • Push: Writes the full monolith first (source of truth), then dual-writes only dirty partition shards afterward.
  • commitPush: Does not use partitions — reads/merges/writes the full monolith.

Target Architecture

Eliminate the monolithic index. Make the _index/ shards the sole source of truth. Every push writes only the touched shard(s); every pull assembles the full picture from shards on demand.

Partition Scheme

Extend partitioning to cover all datastore subdirectories, not just data/.

Subdirectory Partition key pattern Granularity
data/ data--{type}--{modelId} Per model
outputs/ outputs--{type}--{modelId} Per model
definitions-evaluated/ definitions-evaluated--{type}--{modelId} Per model
workflows-evaluated/ workflows-evaluated--{workflowId} Per workflow
workflow-runs/ workflow-runs--{workflowId} Per workflow
auto-definitions/ auto-definitions Single shard
audit/ audit Single shard
telemetry/ telemetry Single shard
logs/ logs Single shard
files/ files Single shard

_meta.json Structure (Version 2)

{
  "version": 2,
  "partitions": ["data--mytype--abc123", "outputs--mytype--abc123", "audit"],
  "commitSeq": 42,
  "lastCompacted": "2026-07-01T12:00:00.000Z"
}
  • version: 2 distinguishes shard-first from the current version: 1 (dual-write) format.
  • commitSeq is a monotonic counter bumped on every commitPush. Replaces the monolith ETag for the zero-diff fast path.

Push Path (commitPush)

Under the global lock:

  1. Read _meta.json to get the current partition list.
  2. Read only the shard(s) touched by the manifest entries.
  3. Merge the manifest entries into those shard(s).
  4. Write the updated shard(s) back.
  5. If new partition keys were introduced, append them to _meta.json.
  6. Bump commitSeq and write _meta.json.
  7. Clear the dirty flag.

Lock-hold time is proportional to the shard size (typically KB, not MB).

preparePush is unchanged — it uploads files outside the lock and returns an opaque manifest. The manifest should include the list of affected relative paths so commitPush can derive partition keys.

Pull Path

Scoped pull (pullChanged with context.models): unchanged — read the relevant model shard(s) from _index/. No monolith fallback needed once _meta.json version is 2.

Unscoped pull (pullChanged without context): read _meta.json to discover all partition keys, then fetch all shards in parallel (batched at MAX_CONCURRENCY = 10). Merge into the in-memory index.

Zero-Diff Fast Path

The sidecar stores { commitSeq: 42 }. On the next pull, HEAD _meta.json → parse → compare commitSeq. If unchanged, skip. If changed, fetch dirty shards (or all shards for an unscoped pull).

Optional optimization: Store each shard's ETag in _meta.json so scoped pulls can skip unchanged shards. This can be added after the MVP.

Backward Compatibility

Phase 1 (this release): commitPush writes shards as the source of truth and also writes the monolith as a derived artifact. The monolith write happens after the shards, outside the critical section if possible. Old clients continue to work. New clients read shards.

Phase 2 (next major version): Stop writing the monolith. Old clients must upgrade.

Old writers: On unscoped pull, if _meta.json is missing or version: 1, fall back to the monolith. This is the existing behavior and should be preserved as a permanent fallback.

Migration

On the first commitPush by a shard-first extension:

  1. Read the existing .datastore-index.json.
  2. Partition all entries into shards using the new partition key scheme.
  3. Write all shard files to _index/.
  4. Write _meta.json with version: 2 and commitSeq: 1.
  5. Continue writing the monolith during phase 1.

Detection via _meta.json: missing = pre-partition, version: 1 = dual-write, version: 2 = shard-first.

Recovery

_meta.json missing/corrupted: List _index/ prefix via S3 ListObjectsV2 to discover all shard files. Rebuild _meta.json from discovered filenames. Self-healing, no operator intervention. Log a warning.

Individual shard missing/corrupted: Fall back to the monolith for that shard's entries (if monolith exists). If no monolith, rebuild the shard by listing S3 objects under the shard's path prefix.

Full rebuild: swamp datastore compact rebuilds all shards from bucket contents via ListObjectsV2 + HeadObject.

Namespace Interaction

In namespaced mode, the shard path becomes {namespace}/_index/. The partition scheme is unchanged — the namespace prefix is applied by existing path resolution logic. _meta.json and all shards are per-namespace.

Test Dimensions

  1. Concurrent-writer throughput: 4+ concurrent commitPush calls. Measure lock-hold time, verify zero LockTimeout.
  2. Unscoped pull correctness: After N concurrent writes, unscoped pull assembles all shards correctly.
  3. Scoped pull correctness: Write to model A, scoped pull for A returns updated entries, pull for B unaffected.
  4. Backward compatibility (phase 1): Old reader sees all entries via monolith. New reader sees all entries via shards. Both identical.
  5. Migration: Bucket with only a monolith is correctly migrated on first shard-first commit.
  6. _meta.json recovery: Deleting _meta.json triggers automatic rebuild from _index/ listing.
  7. Zero-diff fast path: After a commit, commitSeq in sidecar matches _meta.json. No-op push/pull fast-paths to 0.
  8. Mixed partition types: A commit touching both data/ (per-model) and audit/ (single shard) updates both correctly.

Implementation Sequence

  1. Extend partitionKeyFromPath to cover all datastore subdirectories.
  2. Add commitSeq to _meta.json (version 2 format).
  3. Implement shard-first commitPush: read/merge/write touched shards + bump commitSeq under lock. Dual-write monolith outside lock (phase 1).
  4. Update zero-diff fast path to use commitSeq instead of monolith ETag.
  5. Update unscoped pullChanged to assemble from shards when _meta.json version is 2.
  6. Add _meta.json self-healing recovery.
  7. Test all dimensions above.
  8. Phase 2 (future): stop writing monolith, add cleanup tooling.

Core CLI Interface (unchanged)

No core changes needed. The DatastoreSyncService interface is unchanged:

  • preparePush(options?) → PushManifest (opaque, extension-defined)
  • commitPush(manifest, options?) → number | void
  • capabilities() → { twoPhaseSync: true, scopedSync: true, ... }

The extension embeds partition keys in the PushManifest during preparePush and uses them in commitPush. Core passes the manifest through without inspection.

stack72 commented 7/1/2026, 6:36:19 PM

Benchmark Setup Context

This comment provides everything needed to set up a before/after benchmark for the shard-first index change. The implementing agent should use this to validate that the change actually reduces lock-hold time.

Goal

Measure commitPush lock-hold time and concurrent-writer success rate with:

  1. Before: Published @swamp/s3-datastore extension (monolithic index)
  2. After: Local extension source with shard-first changes (partitioned index)

Target: 4 concurrent writers to the same namespace, zero LockTimeout errors after the change.

Prerequisites

  • swamp CLI installed (which swamp must succeed)
  • swamp auth login completed (needed for extension pull)
  • Docker available (for MinIO)

Step 1: Start MinIO

docker run -d --name swamp-minio \
  -p 9000:9000 -p 9001:9001 \
  -e MINIO_ROOT_USER=minioadmin \
  -e MINIO_ROOT_PASSWORD=minioadmin \
  minio/minio server /data --console-address ':9001'

# Create the bucket
docker exec swamp-minio mc alias set local http://localhost:9000 minioadmin minioadmin
docker exec swamp-minio mc mb local/swamp-benchmark

Step 2: Create a benchmark repo

mkdir -p /tmp/swamp-shard-benchmark
cd /tmp/swamp-shard-benchmark
swamp repo init
swamp datastore setup extension @swamp/s3-datastore \
  --config '{"bucket":"swamp-benchmark","endpoint":"http://localhost:9000","region":"us-east-1","credentials":{"accessKeyId":"minioadmin","secretAccessKey":"minioadmin"},"forcePathStyle":true}'

Step 3: Create a simple model type for generating data

Create extensions/models/bench.ts:

import { z } from "npm:zod@4";

export const model = {
  type: "bench/writer",
  name: "Benchmark writer",
  description: "Generates data entries for index benchmarking",
  configSchema: z.object({}),
  methods: {
    write: {
      description: "Write some data",
      inputSchema: z.object({
        payload: z.string().optional(),
      }),
      async run({ save }, { input }) {
        const data = input.payload ?? "x".repeat(100);
        await save("result", data);
        return { wrote: data.length };
      },
    },
  },
};

Step 4: Seed the index with bulk data

The key is building up a large .datastore-index.json. Each model method run creates ~5-10 index entries. To reach a meaningfully large index (target: 20-50MB, ~100K-250K entries), create many model instances and run methods on them in a tight loop.

# Create 200 model instances
for i in $(seq 1 200); do
  swamp model create bench/writer "bench-$i"
done

# Run 500 method invocations (each creates versioned data entries)
# Run sequentially to avoid contention during seeding
for run in $(seq 1 500); do
  instance="bench-$((( (run - 1) % 200 ) + 1))"
  swamp model bench/writer method run write "bench-$instance" \
    --input payload="$(head -c 200 /dev/urandom | base64)" \
    --quiet 2>/dev/null
done

# Check the index size
ls -lh ~/.swamp/repos/*/\*.datastore-index.json 2>/dev/null || \
  echo "Check the S3 bucket for .datastore-index.json size"

Important: The seeding loop may take a while. The goal is an index large enough that commitPush takes multiple seconds. Check progress by looking at the index size. If 500 runs aren't enough, keep going. Each run adds several entries, so 500 runs x 200 models x ~5 entries = ~500K entries should yield a ~50-100MB index.

Alternative — synthetic seeding: If the organic approach is too slow, you can build a large index synthetically:

  1. After the first few organic writes, download .datastore-index.json from MinIO
  2. Programmatically expand it with generated entries following the same path pattern (data/bench--writer/{uuid}/{data-name}/{version}/raw)
  3. Upload the expanded index back to MinIO
  4. Run swamp datastore sync --pull to hydrate the local cache

The organic approach is more realistic but slower. The synthetic approach is faster but the entries won't have corresponding files in the bucket (which is fine — we're measuring index I/O time, not file transfer).

Step 5: Benchmark harness

Create benchmark.sh:

#!/bin/bash
set -euo pipefail

CONCURRENCY=${1:-4}
REPO_DIR=/tmp/swamp-shard-benchmark
RESULTS_DIR=/tmp/swamp-benchmark-results
mkdir -p "$RESULTS_DIR"

echo "=== Concurrent writer benchmark (concurrency=$CONCURRENCY) ==="
echo "Start: $(date -u +%Y-%m-%dT%H:%M:%S.%3NZ)"

# Pick random existing model instances for each writer
MODELS=($(swamp model search --json 2>/dev/null | \
  python3 -c "import sys,json,random; d=json.load(sys.stdin); ms=[m['name'] for m in d[:20]]; random.shuffle(ms); print(' '.join(ms[:$CONCURRENCY]))" 2>/dev/null))

if [ ${#MODELS[@]} -lt $CONCURRENCY ]; then
  echo "ERROR: Not enough models. Need $CONCURRENCY, found ${#MODELS[@]}"
  exit 1
fi

# Launch concurrent writers
pids=()
for i in $(seq 0 $(( CONCURRENCY - 1 ))); do
  model="${MODELS[$i]}"
  (
    start=$(date +%s%3N)
    swamp model bench/writer method run write "$model" \
      --input payload="concurrent-write-$i-$(date +%s)" \
      --verbose 2>&1 | tee "$RESULTS_DIR/writer-$i.log"
    end=$(date +%s%3N)
    echo "Writer $i ($model): $(( end - start ))ms" >> "$RESULTS_DIR/summary.txt"
  ) &
  pids+=($!)
done

# Wait for all writers
successes=0
failures=0
for pid in "${pids[@]}"; do
  if wait $pid; then
    successes=$((successes + 1))
  else
    failures=$((failures + 1))
  fi
done

echo ""
echo "=== Results ==="
echo "Succeeded: $successes"
echo "Failed (LockTimeout): $failures"
echo "End: $(date -u +%Y-%m-%dT%H:%M:%S.%3NZ)"
echo ""

# Extract timing from verbose logs
echo "=== Phase timings (from verbose logs) ==="
for f in "$RESULTS_DIR"/writer-*.log; do
  writer=$(basename "$f" .log)
  echo "--- $writer ---"
  grep -E 'Preparing push|Committing index|Committed|LockTimeout|Pushing changes' "$f" 2>/dev/null || echo "  (no phase markers found)"
done

cat "$RESULTS_DIR/summary.txt" 2>/dev/null
chmod +x benchmark.sh
./benchmark.sh 4

Step 6: Record BEFORE results

Run the benchmark 3 times and record:

  • Success/failure counts
  • Wall-clock time per writer
  • Phase timings from verbose logs (Preparing push → Committing index update → Committed N file(s))
  • Lock-hold time (time between Committing index update and Committed)

Step 7: Switch to local extension source (AFTER)

Clone the s3-datastore extension source and point the benchmark repo at it:

# The pulled extension is at:
#   /tmp/swamp-shard-benchmark/.swamp/pulled-extensions/@swamp/s3-datastore/

# Copy it to a working directory for modification
cp -r /tmp/swamp-shard-benchmark/.swamp/pulled-extensions/@swamp/s3-datastore \
  /tmp/s3-datastore-shard-first

# Add it as a local extension source (local source takes precedence over pulled)
cd /tmp/swamp-shard-benchmark
swamp extension source add /tmp/s3-datastore-shard-first --only datastores

Now apply the shard-first changes to /tmp/s3-datastore-shard-first/datastores/_lib/s3_cache_sync.ts.

The key file is s3_cache_sync.ts — that's where partitionKeyFromPath, pushChanged, pullChanged, writePartitionedIndex, and pullPartitionedIndex live. The commitPush method will need to be added here (if not already present in the current published version, implement it following the two-phase sync contract).

Step 8: Record AFTER results

Run the same benchmark 3 times with the shard-first extension and compare:

  • Lock-hold time should drop from ~seconds to sub-second
  • All 4 concurrent writers should succeed (zero LockTimeout)
  • Total wall-clock time should decrease significantly

Key metrics to report

Metric Before After
Index size (MB)
Concurrent writers 4 4
Successes /4 /4
LockTimeout failures /4 /4
Avg lock-hold time (ms)
Max lock-hold time (ms)
Total wall-clock (ms)

Debugging tips

  • Use --verbose on all swamp commands to see phase timing markers
  • Check swamp datastore lock status if writes hang
  • Use swamp datastore lock release --force to clear stuck locks
  • The local cache is at ~/.swamp/repos/{repoId}/
  • The index file is at the cache root: .datastore-index.json
  • Use ls -lh on the index to check its size
  • Check MinIO console at http://localhost:9001 (minioadmin/minioadmin) to inspect bucket contents

Sign in to post a ripple.