Skip to main content
← back to extensions

Mongodb Datastore

@keeb/mongodb-datastorev2026.09.04.1· 6d agoMODELSWORKFLOWSDATASTORES
01README

MongoDB-backed swamp datastore — distributed locks with TTL, heartbeat, and nonce fencing, plus a manifest+content-addressed-blob sync protocol that streams the datastore tier as inline-bytes cursor batches. Dirty tracking rides an append-only journal with ancestor coalescing (O(1) markDirty) and pushes reconcile dirty roots in batches rather than one round-trip each. Blobs over 15 MB are transparently chunked across multiple docs so they fit under MongoDB's 16 MB BSON limit. Ships a companion maintenance model and workflow that inventory, sweep orphaned blobs and aged tombstones, and compact reclaimed space — fanning out over every namespace in one execution. Scoped by tenant + repo namespace. Requires MongoDB 4.0+ as a replica set.

02Models1
@keeb/mongodb-datastore/maintenancev2026.08.19.1maintenance.ts

Global Arguments

ArgumentTypeDescription
uristringMongoDB URI for the cluster backing the datastore (no auth baked in)
usernamestringMongoDB username
passwordstringMongoDB password — supply a vault reference, never a literal
databasestringDatabase shared by all tenants/repos
tenantIdstringTenant identifier; the collection prefix is t_<tenantId>_r_<namespace>
fn inventory(namespaces: array)
Snapshot every namespace in the cluster: live paths, tombstones, blob count, allocated vs reusable bytes, and how long since the last write. Fans out over all namespaces in one execution.
ArgumentTypeDescription
namespacesarrayNamespaces to inspect; empty means every namespace found
fn sweep(namespaces: array, dryRun: boolean, graceMinutes: number, tombstoneDays: number, skipBlobs: boolean, activeIdleHours: number)
Prune tombstones past their grace window and delete blobs no live path references. Fans out over namespaces in one execution. Refuses to sweep blobs in a namespace written recently whose blobs predate the createdAt field, since nothing can distinguish an in-flight push there.
ArgumentTypeDescription
namespacesarrayNamespaces to sweep; empty means every namespace found
dryRunbooleanReport without deleting. Defaults true — opt in to deletion.
graceMinutesnumberSpare blobs created within this many minutes (in-flight pushes)
tombstoneDaysnumberSpare tombstones deleted within this many days. Must exceed the longest gap between any peer's syncs, or a dormant peer resurrects deleted files.
skipBlobsbooleanPrune tombstones only, leaving bytes for a maintenance window
activeIdleHoursnumberA namespace written more recently than this counts as active; its blobs are swept only if they carry createdAt
fn compact(namespaces: array, minReusableMb: number)
Return freed space to the filesystem. Deleting documents only moves space to WiredTiger's free list, so a swept cluster still reports its old size until this runs. Uses force:true, which is required on a replica-set primary and slows concurrent operations — schedule it.
ArgumentTypeDescription
namespacesarrayNamespaces to compact; empty means every namespace found
minReusableMbnumberSkip collections holding less reusable space than this

Resources

inventory(infinite)— Per-namespace size and activity snapshot for the datastore cluster
sweep(infinite)— Result of a tombstone + orphaned-blob sweep
compaction(infinite)— Storage returned to the filesystem by a compact pass. bytesReclaimed is a lower bound — storageSize lags WiredTiger's checkpoint, so a pass that freed space may report 0.
03Workflows1
@keeb/datastore-maintenance75098440-81dd-44e9-9515-5a46f67aca0c

Reclaim space in a MongoDB-backed swamp datastore: snapshot every namespace, prune tombstones past grace and blobs no live path references, then return the freed space to the filesystem. Requires a @keeb/mongodb-datastore/maintenance instance named datastoreMaintenance.

reclaimInventory, sweep, then compact — each step fans out over all namespaces
1.inventorydatastoreMaintenance.inventory— Snapshot per-namespace live paths, tombstones, blob counts, and allocated vs reusable bytes. Runs first so the sweep's effect is measurable against a recorded baseline.
2.sweepdatastoreMaintenance.sweep— Prune tombstones older than the grace window and delete blobs no live path references. dryRun is false here; the method itself still refuses to sweep blobs in a namespace that is actively written and whose blobs predate the createdAt field.
3.compactdatastoreMaintenance.compact— Return freed space to the filesystem. Deleting documents only moves it to WiredTiger's free list, so without this the cluster still reports its pre-sweep size. Depends on sweep because there is nothing to reclaim until the sweep has produced free blocks.
04Datastores1
MongoDBconfigurable
@keeb/mongodb-datastoremongodb/mod.ts

Stores swamp runtime coordination and datastore bytes in MongoDB — distributed locks with TTL + heartbeat + nonce fencing, plus manifest + content-addressed blob sync of the datastore tier between local cache and MongoDB. Blobs over 15MB are transparently chunked across multiple docs so they fit under MongoDB's 16MB BSON limit. Scoped by tenant + repo namespace. Requires MongoDB 4.0+ running as a replica set.

05Previous Versions11
2026.09.02.1
2026.08.19.2

Added 1 models. Added 1 workflows

2026.08.19.1
2026.06.13.1
2026.05.25.1
2026.05.23.1
2026.05.21.1
2026.05.05.1

updated labels

2026.05.04.1
2026.04.24.2
2026.04.24.1
06Stats
A
100 / 100
Downloads
310
Archive size
782.1 KB
  • Has README or module doc2/2earned
  • README has a code example1/1earned
  • README is substantive1/1earned
  • Most symbols documented1/1earned
  • No slow types (deprecated)1/1earned
  • Dependencies pass trust audit2/2earned
  • Has description1/1earned
  • Platform support declared (or universal)2/2earned
  • License declared1/1earned
  • Verified public repository2/2earned
Repository
https://github.com/keeb/swamp-mongodb-datastore
07Platforms
08Labels
#mongodb#datastore#distributed-lock#content-addressed#replica-set