Skip to main content

Postgres Datastore

@webframp/postgres-datastorev2026.07.30.1· 7d agoDATASTORES
01README

Stores swamp runtime data in PostgreSQL with row-based distributed locking. Compatible with AWS RDS, Aurora, and Aurora Serverless v2. Uses fencing tokens for split-brain safety across failover events. Emits OpenTelemetry spans for SQL operations, lock acquisition/release, and push/pull sync so datastore activity is visible in traces when a TracerProvider is configured.

02Release Notes

2026.07.30.1

Fixed: Subtree lookup query in the incremental push path forced sequential scans on the files table. The WHERE path = $1 OR path LIKE $2 pattern made PostgreSQL unable to use the btree primary key for the LIKE branch, causing O(n^2) write latency as the catalog grew past ~30K files.

Changed: The subtree query now uses UNION ALL to split the exact-match and prefix-match predicates into independent branches, each with its own query plan. A new text_pattern_ops btree index on the path column is created automatically on first connect, enabling index scans for all prefix-anchored LIKE queries (both the push and pull paths).

Upgrade note: The new index is created via CREATE INDEX IF NOT EXISTS on startup. For large tables (>100K rows) the initial index build may take a few seconds on first connection after upgrade. Subsequent connections are instantaneous. No manual migration required.

03Datastores1
PostgreSQL Datastoreconfigurable
@webframp/postgres-datastorepostgres_datastore/mod.ts

Stores swamp runtime data in PostgreSQL with row-based distributed locking. Compatible with AWS RDS, Aurora, and Aurora Serverless v2.

Config Fields

FieldTypeDescription
connectionStringstringPostgreSQL connection URI (supports RDS, Aurora, Aurora Serverless v2)
schemastringPostgreSQL schema for swamp tables
sslenumSSL mode: disable (no TLS), require (TLS without CA verify), verify-ca (TLS with CA bundle)
sslCaPath?stringPath to CA certificate bundle (e.g., RDS global-bundle.pem). Required when ssl=verify-ca.
04Previous Versions14
2026.07.30.1

2026.07.30.1

Fixed: Subtree lookup query in the incremental push path forced sequential scans on the files table. The WHERE path = $1 OR path LIKE $2 pattern made PostgreSQL unable to use the btree primary key for the LIKE branch, causing O(n^2) write latency as the catalog grew past ~30K files.

Changed: The subtree query now uses UNION ALL to split the exact-match and prefix-match predicates into independent branches, each with its own query plan. A new text_pattern_ops btree index on the path column is created automatically on first connect, enabling index scans for all prefix-anchored LIKE queries (both the push and pull paths).

Upgrade note: The new index is created via CREATE INDEX IF NOT EXISTS on startup. For large tables (>100K rows) the initial index build may take a few seconds on first connection after upgrade. Subsequent connections are instantaneous. No manual migration required.

2026.07.29.1

2026.07.29.1

Added: Monotonic commitSeq via PostgreSQL sequence. Pull fast-path now compares a single integer instead of parsing timestamps, eliminating clock-skew vulnerabilities across concurrent writers.

Changed: Dirty-path pushes are batched into a single transaction (was N separate BEGIN/COMMIT pairs). One watermark write instead of N. Expected 10-50x latency improvement for incremental pushes.

Changed: Pull captures commitSeq before the metadata scan (not after) to prevent TOCTOU races where concurrent pushes during the fetch window could be silently missed.

Changed: Content-fetch batches during pull run up to 3 in parallel via Promise.all.

Changed: Manifest query for push uses batch path lookups (IN clause) instead of a full table scan when lastPulledAt provides a bound.

Added: Tombstone garbage collection — every push transaction deletes tombstones older than 7 days, preventing unbounded table growth.

2026.07.27.1

Changed: Bump @opentelemetry/api 1.9.0 → 1.9.1

2026.07.25.1

Added: OpenTelemetry spans for every layer of the datastore. Each SQL round trip emits a span named for the operation it performs (PostgreSQL acquireLock, PostgreSQL scanFileMetadata, PostgreSQL commitPushTransaction, …) carrying db.system.name, db.operation.name, db.collection.name, and returned row counts. The lock emits postgres-datastore lock acquire / release / withLock / inspect / forceRelease, with acquire recording wait duration and whether it contended. The sync service emits postgres-datastore pullChanged / pushChanged / hydrateFile / preparePush / commitPush with file counts and fast-path indicators.

Added: Retries are recorded as retry span events on the enclosing operation — both the transient-error backoff in retryable and the lock contention loop.

Changed: pushChanged now reports datastore.files_pushed and datastore.files_deleted separately, and pullChanged likewise separates downloads from local deletions. The internal counters increment for both writes and tombstones, so reporting either as a file count would have overstated it. The values returned to callers are unchanged.

Changed: Nothing else observable without tracing configured. The extension depends on @opentelemetry/api only; the host process owns the TracerProvider, and every span is a no-op when none is registered.

Note: The existing SWAMP_PG_SYNC_TRACE=1 phase tracer is untouched and still writes its own timing lines to stderr. The two are independent — enable either, both, or neither.

Note on secrets: Statement text and bound parameters are never recorded. Query parameters carry file content and the connection string carries a password, so span attributes hold only hand-written operation labels, table names, and counts.

2026.07.27.1

2026.07.27.1

Changed: Bump @opentelemetry/api 1.9.0 → 1.9.1

2026.07.25.1

Added: OpenTelemetry spans for every layer of the datastore. Each SQL round trip emits a span named for the operation it performs (PostgreSQL acquireLock, PostgreSQL scanFileMetadata, PostgreSQL commitPushTransaction, …) carrying db.system.name, db.operation.name, db.collection.name, and returned row counts. The lock emits postgres-datastore lock acquire / release / withLock / inspect / forceRelease, with acquire recording wait duration and whether it contended. The sync service emits postgres-datastore pullChanged / pushChanged / hydrateFile / preparePush / commitPush with file counts and fast-path indicators.

Added: Retries are recorded as retry span events on the enclosing operation — both the transient-error backoff in retryable and the lock contention loop.

Changed: pushChanged now reports datastore.files_pushed and datastore.files_deleted separately, and pullChanged likewise separates downloads from local deletions. The internal counters increment for both writes and tombstones, so reporting either as a file count would have overstated it. The values returned to callers are unchanged.

Changed: Nothing else observable without tracing configured. The extension depends on @opentelemetry/api only; the host process owns the TracerProvider, and every span is a no-op when none is registered.

Note: The existing SWAMP_PG_SYNC_TRACE=1 phase tracer is untouched and still writes its own timing lines to stderr. The two are independent — enable either, both, or neither.

Note on secrets: Statement text and bound parameters are never recorded. Query parameters carry file content and the connection string carries a password, so span attributes hold only hand-written operation labels, table names, and counts.

2026.07.25.1

2026.07.25.1

Added: OpenTelemetry spans for every layer of the datastore. Each SQL round trip emits a span named for the operation it performs (PostgreSQL acquireLock, PostgreSQL scanFileMetadata, PostgreSQL commitPushTransaction, …) carrying db.system.name, db.operation.name, db.collection.name, and returned row counts. The lock emits postgres-datastore lock acquire / release / withLock / inspect / forceRelease, with acquire recording wait duration and whether it contended. The sync service emits postgres-datastore pullChanged / pushChanged / hydrateFile / preparePush / commitPush with file counts and fast-path indicators.

Added: Retries are recorded as retry span events on the enclosing operation — both the transient-error backoff in retryable and the lock contention loop.

Changed: pushChanged now reports datastore.files_pushed and datastore.files_deleted separately, and pullChanged likewise separates downloads from local deletions. The internal counters increment for both writes and tombstones, so reporting either as a file count would have overstated it. The values returned to callers are unchanged.

Changed: Nothing else observable without tracing configured. The extension depends on @opentelemetry/api only; the host process owns the TracerProvider, and every span is a no-op when none is registered.

Note: The existing SWAMP_PG_SYNC_TRACE=1 phase tracer is untouched and still writes its own timing lines to stderr. The two are independent — enable either, both, or neither.

Note on secrets: Statement text and bound parameters are never recorded. Query parameters carry file content and the connection string carries a password, so span attributes hold only hand-written operation labels, table names, and counts.

updated labels

2026.07.18.1

2026.07.18.1

Changed: Version bump only, no code changes.

2026.07.16.1

Changed: Internal-only version bump. PR #183 added a fmt:check task to deno.json, which is not part of the published bundle — this release's published content is identical to 2026.06.30.1.

2026.07.16.1

2026.07.16.1

Changed: Internal-only version bump. PR #183 added a fmt:check task to deno.json, which is not part of the published bundle — this release's published content is identical to 2026.06.30.1.

2026.06.30.1

2026.06.30.1

Two-Phase Datastore Sync

Adds preparePush/commitPush two-phase sync protocol alongside the existing fullWalkPush and pushOneRel single-shot paths.

Added

  • TwoPhaseSyncService interface with preparePush() and commitPush() methods
  • PushManifest opaque branded type for safe manifest passing between phases
  • preparePush() captures sidecar snapshot and collects diff without transaction
  • commitPush() executes batched inserts in a transaction, clears sidecar
  • capabilities() now reports twoPhaseSync: true

Changed

  • Extracted collectFullWalkDiff() and collectOneRelDiff() helper functions
  • Refactored fullWalkPush/pushOneRel to use extracted helpers (behavior preserved)
  • No breaking changes to existing sync behavior
2026.06.15.1
2026.06.11.1
2026.05.30.1
2026.05.25.1
2026.05.20.1
2026.05.05.1
2026.05.04.1
05Stats
A
100 / 100
Downloads
61
Archive size
72.5 KB
  • Has README or module doc2/2earned
  • README has a code example1/1earned
  • README is substantive1/1earned
  • Most symbols documented1/1earned
  • No slow types (deprecated)1/1earned
  • Dependencies pass trust audit2/2earned
  • Has description1/1earned
  • Platform support declared (or universal)2/2earned
  • License declared1/1earned
  • Verified public repository2/2earned
06Platforms
07Labels