Skip to main content
← Back to list
01Issue
FeatureOpenSwamp CLI
AssigneesNone

Relationships

#663 Implement ephemeral data as in-memory repository

Opened by stack72 · 6/16/2026

Problem

Swamp defines an "ephemeral" lifetime for data artifacts — data that should only exist for the duration of a method invocation or workflow execution. Today, "ephemeral" is accepted by the type system and schema but not implemented. The lifecycle service logs a warning and returns null, and ephemeral data gets written to disk and never cleaned up — behaving identically to "infinite".

Proposed Solution

Implement ephemeral data as a global in-memory store that:

  1. Never writes to disk — content and metadata live only in memory
  2. Is indexed for search — uses an in-memory SQLite catalog (:memory:) so it behaves identically to persistent data from a query perspective (data.latest(), data.query(), data.search() all work transparently)
  3. Is scoped to execution lifetime — when the workflow run or method run that created it completes, the in-memory store is dropped
  4. Is available across remote execution — workers dispatched by the orchestrator read/write ephemeral data through the same data plane interfaces
  5. Requires no new CEL functions — existing data.latest() transparently resolves from the in-memory store; the engine determines whether data is in-memory or on disk

Why

Ephemeral data is the foundation for dynamic data lookups — intermediate results, computed context, and runtime-resolved references that models and workflows produce and consume within a single execution. Without this, there is no way to pass transient computed data between workflow steps without polluting the persistent datastore.

Scope of Changes

The implementation involves these new components:

  • InMemoryUnifiedDataRepository (src/infrastructure/persistence/in_memory_data_repository.ts) — full implementation of the UnifiedDataRepository interface using Map-based storage, with a :memory: CatalogStore for search indexing
  • CompositeUnifiedDataRepository (src/domain/data/composite_data_repository.ts) — routes writes by lifetime === "ephemeral" to the in-memory repo, reads check in-memory first then fall back to the filesystem repo
  • CompositeDataQueryService (src/domain/data/composite_data_query_service.ts) — merges query results from both persistent and ephemeral catalogs so CEL queries work transparently

And modifications to existing wiring:

  • DefaultStepExecutor.buildDeps() — accept and wire ephemeral repo
  • WorkflowExecutionService.run() — create ephemeral store at workflow start, dispose in finally blocks
  • modelMethodRun() — same pattern for standalone method runs
  • executeWorkflowWithLocks() — serve-side creation/disposal
  • DataLifecycleService — remove the "not implemented" warning for ephemeral

A detailed design document is at design/ephemeral-data.md with exact file paths, line numbers, interface contracts, and a phased implementation plan.

02Bog Flow
OPENTRIAGEDIN PROGRESSSHIPPED

Open

6/16/2026, 9:00:20 PM

No activity in this phase yet.

03Sludge Pulse

Sign in to post a ripple.