DATASTORE PROVIDER

A datastore provider is a TypeScript module that implements data persistence for a custom backend (cloud object storage, networked filesystem, etc.). The file lives in extensions/datastores/ and exports a datastore object that Swamp loads at runtime.

Every Swamp command needs a swamp-club.com account (Swamp Account Requirement). Setting up a datastore that uses a custom provider additionally requires the credential to carry the datastore:* scope when it is a collective token. The built-in filesystem type needs no scope.

For datastore configuration and usage (setup, sync, locking, CLI commands), see the Datastore Configuration reference.

Export Shape

import { z } from "npm:zod@4";

export const datastore = {
  type: "@mycollective/my-datastore",
  name: "My Datastore",
  description: "Store data in my backend",
  configSchema: z.object({
    bucket: z.string().describe("Storage bucket name"),
    region: z.string().optional().describe("Region"),
  }),
  createProvider(config: Record<string, unknown>): DatastoreProvider {
    // validate config, return provider instance
  },
};

Fields

DatastoreProvider Interface

The object returned by createProvider must implement the following methods.

Swamp reads the datastore format marker (datastore-format) through datastoreControlPlaneStore() before every command that opens the datastore, so the method must be cheap:

• Building the store creates no sync service, binds no namespace and sends no request. The first request is sent by get.
• A get is one request, with no separate credentials preflight.

Without the method, Swamp reads the marker through a fresh sync service's controlPlaneStore().

Supporting Interfaces

DistributedLock

Coordinates exclusive access to prevent concurrent writers from corrupting data.

LockOptions

DatastoreVerifier

DatastoreHealthResult

DatastoreSyncService

Implement this if the backend supports bidirectional file synchronization.

pullChanged and pushChanged return the number of files transferred, or void.

markDirty is called before every cache write. Implementations must be idempotent and cheap. When options.relPath is present, it identifies the cache-relative path about to be written. When absent, a bulk mutation occurred and the next pushChanged must perform a full walk. See the markDirty contract in the Swamp source for the complete set of rules.

clearDirty is called after Swamp undoes writes it had marked: when a push is refused because its model lock was lost, Swamp discards the refused versions and restores the datastore's copy of each file. Without the call, the marks those writes left make the next push delete files under a marked directory that the cache lacks, and upload a stale latest marker. It follows these rules:

• A mark is dropped only when its key equals one of relPaths. A mark on a parent or child path is a different key and stays.
• Only marks made before the call are dropped. A markDirty that starts during or after the call keeps its mark.
• A bulk mark and an overflowed path set stay.
• Nothing remote is read or written, and no pull watermark changes.
• relPaths are keys as markDirty received them: cache-relative, forward-slash-normalized, with the namespace prefix when there is one.
• A path with no mark is ignored.

Swamp core treats the presence of clearDirty as the capability, as with fetchContent. Without it, a refused push keeps its marks and warns once, naming the data names whose marks stay. The clear-dirty conformance case holds an implementation to these rules.

capabilities returns a SyncCapabilities object advertising optional features the sync service supports. Omit this method when no optional features are implemented.

hydrateFile downloads a single content file from the remote backend to the local cache. The relPath argument is forward-slash-normalized and cache-relative. Implementations must write atomically (write to a temporary path, then rename). Required only when the extension advertises lazyHydration: true in its capabilities.

fetchContent resolves to the bytes of one file as the remote holds it, or null when the remote has no such file. It follows these rules:

• Only a missing file is null. Any other failure rejects, so an unreachable remote is never read as a deleted file. Honor options.signal as the other methods do.
• Nothing is written locally: no cache file is created, replaced or removed, and no dirty state or pull watermark changes. A local file at relPath that differs from the remote one is neither returned nor modified.
• relPath is cache-relative and forward-slash-normalized, as for hydrateFile. With a namespace it already starts with {namespace}/, so the implementation must not add the namespace again. When options.namespace is unset or empty, relPath is read from the datastore root.
• A relPath that is absolute or has a .. segment is rejected, never resolved.
• The returned bytes are not kept in instance state. In swamp serve the sync service lives as long as the process. The whole file is returned in memory, so the method suits small files such as run records.

Swamp core treats the presence of fetchContent as the capability. There is no SyncCapabilities flag for it.

DatastoreSyncOptions

Options passed to pullChanged, pushChanged, markDirty, hydrateFile, and fetchContent.

SyncContext

Describes what a sync operation is about.

SyncCapabilities

Capabilities a sync service advertises to Swamp core.

ControlPlaneStore

Returned by a sync service's controlPlaneStore(). It reads and writes small records in the datastore directly, bypassing the cache and sync. Keys are slash-delimited paths such as heartbeats/<id>, kept under _control/ in the backend, below the namespace when one is bound.

A store without putIfAbsent cannot hold the records of workflow signal waits. Swamp refuses to start a workflow that contains a wait_for_signal step on such a datastore.

A get that ignores options still works, because Swamp stops waiting at its own deadline either way. Honouring options.signal is recommended.

ControlPlaneReadOptions

Options passed to get on a ControlPlaneStore or a DatastoreControlPlaneStore.

DatastoreControlPlaneStore

Returned by the provider's datastoreControlPlaneStore(). It reads records at the datastore-wide _control/<key>, whatever namespace a repository uses. It binds no namespace and shares no state with any sync service the provider builds, so Swamp can use it before a command's own sync service exists.

get follows these rules:

• Only a missing record returns null. Access denied and every other failure reject, so Swamp can tell an unreadable marker from a missing one.
• options.signal must be honoured: when it aborts, the request and any retry backoff stop and get rejects.

Check an implementation with assertDatastoreControlPlaneStoreConformance.

Conformance Testing

The @swamp-club/swamp-testing package provides conformance assertions for datastore providers and sync services.

assertSyncServiceConformance

import { assertSyncServiceConformance } from "@swamp-club/swamp-testing";

Asserts that a DatastoreSyncService implementation satisfies the behavioral contract. Tests that pullChanged, pushChanged, and markDirty exist and are callable. When capabilities() is present, validates it returns a well-formed SyncCapabilities object.

Deno.test("s3 sync service contract", async () => {
  const syncService = provider.createSyncService!("/repo", "/cache");
  await assertSyncServiceConformance(syncService);
});

With capability assertions:

Deno.test("s3 sync service with scoped sync", async () => {
  const syncService = provider.createSyncService!("/repo", "/cache");
  await assertSyncServiceConformance(syncService, { expectScopedSync: true });
});

SyncServiceConformanceOptions

assertSyncServiceRoundTripConformance

Runs two sync services on one shared backend, as two machines would see it, and asserts that files move between their caches the way Swamp core relies on. It returns the cases that passed and the cases it skipped, each with a reason.

Three cases cover fetchContent. Each is skipped when the second service has no fetchContent.

The clear-dirty case covers clearDirty: a directory mark that clearDirty dropped makes the next push delete and send nothing, and a mark made after the call is still pushed. It is skipped when the first service has no clearDirty.

assertDatastoreControlPlaneStoreConformance

import { assertDatastoreControlPlaneStoreConformance } from "@swamp-club/swamp-testing";

Asserts that a provider's datastoreControlPlaneStore() meets the contract the datastore format check relies on. It checks that get:

• reads a record that a never-pulled sync service's controlPlaneStore() wrote at the datastore-wide root;
• returns null for a missing key;
• does not see a record written only under a namespace;
• rejects when its signal is already aborted.

It also checks that using the store binds no namespace: a sync service built before it can still pull under a namespace afterwards and read that namespace's records. The suite writes only under a key prefix of its own and removes what it wrote.

Deno.test("datastore-wide control-plane store conformance", async () => {
  await assertDatastoreControlPlaneStoreConformance({
    openDatastoreStore: () => provider.datastoreControlPlaneStore!(),
    // a fresh, never-pulled service with its own cache each call
    openSyncService: async () =>
      provider.createSyncService!("/repo", await Deno.makeTempDir()),
  });
});

DatastoreControlPlaneStoreConformanceOptions

Packaging

Declare the datastore in your extension's manifest:

datastores:
  - my_datastore.ts

The file path is relative to the extensions/datastores/ directory within the extension root.