Skip to main content
← back to extensions

Valkey Datastore

@webframp/valkey-datastorev2026.07.25.1· 1d agoDATASTORES
01README

Stores swamp runtime data in Valkey/Redis with sorted-set path indexing for O(log n + k) prefix lookups and SET NX distributed locking. Compatible with local Valkey, AWS ElastiCache Serverless, and AWS MemoryDB. Implements two-phase sync to minimize time under the global lock. Emits OpenTelemetry spans for Valkey commands, lock acquisition/release, and push/pull sync so datastore activity is visible in traces when a TracerProvider is configured.

02Release Notes

2026.07.25.1

Added: OpenTelemetry spans for every layer of the datastore. Valkey round trips emit one span each (Valkey SET, Valkey ZRANGEBYLEX, Valkey GETBUFFER, Valkey EVAL, …) carrying db.system.name, db.operation.name, and the key. Pipeline flushes emit Valkey pipeline writeFiles / deleteFiles / fetchMetadata / fetchHashes with the batched command count. The lock emits valkey-datastore lock acquire / release / withLock / inspect / forceRelease, with acquire recording wait duration and whether it contended. The sync service emits valkey-datastore pullChanged / pushChanged / hydrateFile / preparePush / commitPush with file counts, path counts, the remote sequence number, and fast-path indicators.

Added: Lock contention retries are recorded as retry span events on the acquire span.

Added: Pipeline spans report valkey.pipeline.failed_commands and are marked as errors when any command in the batch failed. pipeline.exec() resolves with per-command errors rather than rejecting, so a partially failed batch would otherwise have left a span reporting success. pullChanged also reports datastore.files_skipped for paths whose metadata read failed, which were previously dropped silently.

Changed: pushChanged and commitPush now report datastore.files_pushed and datastore.files_deleted separately. applyChanges returns a single count covering writes and deletes together, and reporting that as a file count would have overstated pushes whenever tombstones were involved. The value returned to callers is unchanged.

Changed: Nothing else observable without tracing configured. The extension depends on @opentelemetry/api only; the host process owns the TracerProvider, and every span is a no-op when none is registered.

Note: Spans are placed on the round trips that carry latency — index range scans, blob reads, and pipeline flushes — rather than on every one of the seventeen command call sites. Heartbeat renewals are deliberately uninstrumented so a long-held lock does not bury the trace in periodic PEXPIRE spans.

Note on secrets: Blob values are file content and the connection URL embeds a password. Neither is ever recorded; span attributes carry only command names, key names, and counts.

03Datastores1
Valkey Datastoreconfigurable
@webframp/valkey-datastorevalkey_datastore/mod.ts

Stores swamp runtime data in Valkey/Redis with sorted-set path indexing

Config Fields

FieldTypeDescription
urlstringValkey/Redis connection URL (redis:// or rediss:// for TLS)
prefixstringKey namespace prefix for all swamp data
dbnumberRedis database number
password?stringAuth password (prefer vault expression for production use)
connectTimeoutMsnumberConnection timeout in milliseconds
maxRetriesPerRequestnumberMax retries per command before failing
04Previous Versions8
2026.07.21.1

2026.07.21.1

Changed: Bumped ioredis from 5.6.1 to 5.11.1.

Upgrade note: No behavioral changes. Routine dependency maintenance.

2026.07.18.2

2026.07.18.2

Changed: Version bump only, no code changes.

2026.07.18.1

Changed

  • Pinned the zod import specifier to npm:zod@4.4.3 (was npm:zod@4) for hermetic dependency resolution. No runtime behavior change.

2026.07.05.5

Fixed

  • collectFullWalkDiff silently discarded the truncated flag from allPaths(), causing deletions beyond the 50k cap to be invisible. Now throws when truncated, refusing to run an incomplete diff.
  • applyChanges pipeline errors were silently swallowed — pipeline.exec() returns per-command [Error, result] tuples that were never inspected. Dirty state was cleared despite partial write failures. Now inspects every result tuple and throws with the list of failed paths.
  • allPaths() truncation check used a separate ZCARD after ZRANGEBYLEX (TOCTOU race). Replaced with LIMIT+1 pattern: fetch one extra entry and check length.
  • createRedisClient TLS CA read: wrapped readTextFileSync with contextual error message for missing CA files.

2026.07.05.4

Fixed

  • collectOneRelDiff still used ZRANGEBYLEX prefix scan when a file was deleted locally (stat === null fell through to the else branch). Inverted the branch: prefix scan only for stat?.isDirectory, ZSCORE point lookup for everything else (existing files and deleted files).
  • collectOneRelDiff directory branch had no LIMIT clause — added 50k cap matching allPaths/pathsForPrefixes.
  • Removed dead deleted === "true" branch in pullFiles — applyChanges hard-deletes via redis.del/zrem, so no meta with deleted:true is ever written.

2026.07.05.3

Fixed

  • Lock acquire() assigned nonce before the retry loop; if redis.set() threw (e.g. ElastiCache failover), nonce was left set, permanently blocking re-acquire. Now uses a candidate variable and only promotes to nonce on success.
  • pathsForPrefixes used charCode + 1 inclusive upper-bound, which could match an adjacent path (e.g. data/foo0 when scanning data/foo/). Aligned to the 0xFF exclusive sentinel used in collectOneRelDiff.
  • Removed RELEASE_NOTES.md from manifest additionalFiles per project policy.
  • Aligned sidecar.ts isTraversal("") to return true, matching mod.ts behavior.

2026.07.05.2

Fixed

  • collectOneRelDiff used ZRANGEBYLEX prefix range for single-file dirty pushes, which over-fetched sibling paths (e.g. v1 matching v10, v11) and tombstoned them. Now uses ZSCORE point lookup for files; range scan only for directories.
  • Verifier called redis.connect() explicitly, which throws on an already-connected client after sync operations. Removed — ioredis lazy connect triggers on first command.
  • allPaths() and pathsForPrefixes() were unbounded. Added 50k LIMIT clause and truncated flag.
  • Pinned zod to exact version (4.4.3) in deno.json.

Added

  • RELEASE_NOTES.md for CI publish workflow.
  • Regression test: verifier works after sync operation (double-connect).

2026.07.05.1

Added

  • Valkey/Redis datastore extension for swamp
  • Sorted-set path index (ZRANGEBYLEX) for O(log n + k) prefix lookups
  • SET NX with Lua-guarded release for distributed locking
  • Two-phase sync (preparePush/commitPush) to minimize time under global lock
  • Sidecar dirty-path tracking with sequence-counter fast path for pull
  • TLS support for AWS ElastiCache Serverless and MemoryDB
  • Configurable key prefix, database number, connection timeout, and retry limits
2026.07.18.1

2026.07.18.1

Changed

  • Pinned the zod import specifier to npm:zod@4.4.3 (was npm:zod@4) for hermetic dependency resolution. No runtime behavior change.

2026.07.05.5

Fixed

  • collectFullWalkDiff silently discarded the truncated flag from allPaths(), causing deletions beyond the 50k cap to be invisible. Now throws when truncated, refusing to run an incomplete diff.
  • applyChanges pipeline errors were silently swallowed — pipeline.exec() returns per-command [Error, result] tuples that were never inspected. Dirty state was cleared despite partial write failures. Now inspects every result tuple and throws with the list of failed paths.
  • allPaths() truncation check used a separate ZCARD after ZRANGEBYLEX (TOCTOU race). Replaced with LIMIT+1 pattern: fetch one extra entry and check length.
  • createRedisClient TLS CA read: wrapped readTextFileSync with contextual error message for missing CA files.

2026.07.05.4

Fixed

  • collectOneRelDiff still used ZRANGEBYLEX prefix scan when a file was deleted locally (stat === null fell through to the else branch). Inverted the branch: prefix scan only for stat?.isDirectory, ZSCORE point lookup for everything else (existing files and deleted files).
  • collectOneRelDiff directory branch had no LIMIT clause — added 50k cap matching allPaths/pathsForPrefixes.
  • Removed dead deleted === "true" branch in pullFiles — applyChanges hard-deletes via redis.del/zrem, so no meta with deleted:true is ever written.

2026.07.05.3

Fixed

  • Lock acquire() assigned nonce before the retry loop; if redis.set() threw (e.g. ElastiCache failover), nonce was left set, permanently blocking re-acquire. Now uses a candidate variable and only promotes to nonce on success.
  • pathsForPrefixes used charCode + 1 inclusive upper-bound, which could match an adjacent path (e.g. data/foo0 when scanning data/foo/). Aligned to the 0xFF exclusive sentinel used in collectOneRelDiff.
  • Removed RELEASE_NOTES.md from manifest additionalFiles per project policy.
  • Aligned sidecar.ts isTraversal("") to return true, matching mod.ts behavior.

2026.07.05.2

Fixed

  • collectOneRelDiff used ZRANGEBYLEX prefix range for single-file dirty pushes, which over-fetched sibling paths (e.g. v1 matching v10, v11) and tombstoned them. Now uses ZSCORE point lookup for files; range scan only for directories.
  • Verifier called redis.connect() explicitly, which throws on an already-connected client after sync operations. Removed — ioredis lazy connect triggers on first command.
  • allPaths() and pathsForPrefixes() were unbounded. Added 50k LIMIT clause and truncated flag.
  • Pinned zod to exact version (4.4.3) in deno.json.

Added

  • RELEASE_NOTES.md for CI publish workflow.
  • Regression test: verifier works after sync operation (double-connect).

2026.07.05.1

Added

  • Valkey/Redis datastore extension for swamp
  • Sorted-set path index (ZRANGEBYLEX) for O(log n + k) prefix lookups
  • SET NX with Lua-guarded release for distributed locking
  • Two-phase sync (preparePush/commitPush) to minimize time under global lock
  • Sidecar dirty-path tracking with sequence-counter fast path for pull
  • TLS support for AWS ElastiCache Serverless and MemoryDB
  • Configurable key prefix, database number, connection timeout, and retry limits
2026.07.05.5
2026.07.05.4
2026.07.05.3
2026.07.05.2
2026.07.05.1
05Stats
A
100 / 100
Downloads
1
Archive size
117.2 KB
  • Has README or module doc2/2earned
  • README has a code example1/1earned
  • README is substantive1/1earned
  • Most symbols documented1/1earned
  • No slow types (deprecated)1/1earned
  • Dependencies pass trust audit2/2earned
  • Has description1/1earned
  • Platform support declared (or universal)2/2earned
  • License declared1/1earned
  • Verified public repository2/2earned
Repository
https://github.com/webframp/swamp-extensions
06Platforms
07Labels
#valkey#redis#elasticache#memorydb#datastore#opentelemetry