#2111 serve: idle peer data.query remains stale when a remote datastore honors selective pulls

Summary

With two long-running swamp serve instances sharing one repository and remote datastore, server B does not discover runtime output written by server A through passive data.query requests when the datastore provider honors pullChanged({ subdirs }).

The output is available on A and recoverable by B after cold startup. B's background pollers refresh configuration and access-control records only. The query handler reads B's local cache without requesting a runtime refresh.

This is a missing runtime-refresh path for shared servers. It is not evidence of MongoDB losing data, and it is separate from managed-definition resolution in #2109 and managed-edit propagation in #2108.

Environment

• Swamp: 20260910.194957.0-sha.ae1ed149.
• Provider: published @jamesakeech/mongodb-datastore@2026.09.11.1.
• MongoDB: 8.0.29-noble, local single-member replica set.
• Two serve processes: same repository ID, database, tenant and namespace; independent repository directories, local extension indexes and SWAMP_HOME caches.
• Namespace: shared; hydration strategy: full; managed configuration enabled.
• Local test servers used loopback addresses, no authentication, and scheduling disabled. Configuration/access polling used the default 30-second interval.
• Provider advertises namespacedSync, configRefresh and controlPlane, and actually restricts pulls to requested subdirectories.

Reproduction

1. Prepare repository A with the published provider, a reachable MongoDB replica set, and a command/shell model named serve-compat. Configure its execute method to print a unique marker.
2. Configure the datastore with core namespace shared and migrate managed configuration.
3. Bootstrap repository B with the same repository marker and datastore settings. Give B its own cache and independently created extension index. Do not copy the host-local extension catalog between repository directories.
4. Start both servers and wait for /ready. Verify they resolve the same model and see both live instances.
5. Send model.method.run to A:

{"id":"run-a","type":"model.method.run","payload":{"modelIdOrName":"serve-compat","methodName":"execute","skipAllReports":true,"skipAllChecks":true}}

1. Wait for successful completion and datastore flush. Confirm A's data.query returns the output. Repeatedly send the same query to B without executing a model or restarting B:

{"id":"query-b","type":"data.query","payload":{"predicate":"modelName == 'serve-compat' && name == 'result'"}}

1. Continue querying for more than one background polling interval.
2. Stop B and restart the same repository with a new empty SWAMP_HOME. Query again after /ready.

Actual result

In the published-package run:

• A's query returned the successful output, version 1.
• B returned empty results for 40 consecutive queries across approximately 40 seconds. The freshness assertion timed out after 40.21 seconds.
• B's cold startup then hydrated and returned that same output.
• Shared heartbeats, active-run owner lookup, serialized execution, distinct concurrent output versions, and survivor recovery passed.

The same passive-read failure occurred in the local-source run. The complete published-package suite passed 10 of 13 assertions; the other two failures were the already-reported #2108 and #2109.

No longer observation window or network-partition test is claimed. The source paths below explain why these pollers do not eventually fetch unrelated runtime data.

Expected behavior / requested contract

A shared serve deployment should offer a supported way for an idle reader to discover another server's committed output without restarting or executing unrelated work: either a documented periodic runtime refresh or an explicit fresh-read/refresh operation.

The datastore design intentionally makes ordinary reads cache-only so they remain available during writes. This report does not claim a current immediate-consistency guarantee, nor request that every default read silently become a blocking remote operation. Maintainers should define the refresh interval or explicit freshness contract.

Source diagnosis at the tested revision

• src/serve/access_data_poller.ts:28,97: polls only data/swamp/grant, data/swamp/group, data/@swamp/grant, and data/@swamp/group.
• src/serve/config_poller.ts:81: requests only subdirs: ["config"].
• src/serve/handlers/data_handlers.ts:229: handleDataQuery delegates to the local query service without a datastore pull.
• src/domain/data/data_query_service.ts:315,336: an invalidated catalog is rebuilt from local data, not fetched from the remote datastore.
• src/infrastructure/persistence/catalog_store.ts:666: invalidation only clears the populated flag. A fix needs both remote refresh and catalog invalidation.
• src/domain/datastore/datastore_sync_service.ts:149: providers advertising configRefresh should honor requested subdirectory restrictions.
• design/enablers/datastores.md:394–450: documents cache-only read semantics and manual swamp datastore sync --pull.

The original @keeb/mongodb-datastore@2026.09.04.1 and tested @swamp/s3-datastore@2026.09.10.1 ignore subdirs. Their access-data polling incidentally refreshed all runtime data and masked this gap. Tests should use a provider or fake that actually honors the restriction.

Potential fixes and trade-offs

1. Dedicated runtime-data poller (recommended starting point). Add a configurable poller to serve that refreshes the current namespace's data/ subtree and then invalidates the local query catalog. This preserves fast cached reads and provides eventual visibility on idle peers. Costs include per-server background transfer, larger caches, and freshness limited by successful polling cycles. Keep configuration and access-control refresh independent. Validate additional runtime roots separately for other read APIs.

2. Explicit fresh-read or refresh operation. Let callers request a bounded refresh after authorization and before querying. This avoids background work when idle and lets callers choose when to pay for freshness. It adds network latency and failure modes. A failed refresh must not silently return stale data as fresh. Arbitrary CEL predicates may require refreshing the full current-namespace data subtree rather than one model.

3. Metadata-only refresh with asynchronous content hydration. This could reduce transfer of large outputs. It is not a drop-in change: some current query paths read content synchronously and do not call the asynchronous hydration hook. Missing content can affect attribute predicates and projections. Query-path changes and content-dependent regression tests are required.

4. Remote notifications or a change feed. This could reduce polling delay and repeated scans. It requires a cross-provider contract, durable change positions, reconnect/replay handling, and catalog refresh coordination. It is a larger follow-up than the first two options.

5. Unconditional refresh before every query. This directly refreshes query inputs but changes documented cached-read latency and availability, repeats work across requests, and competes with writes. Avoid making this the default without an explicit decision about read semantics.

6. Full background pulls as a workaround. Having providers ignore subdirs recreates the previous incidental visibility. It broadens every configuration/access refresh, can fetch unrelated large outputs, and makes correctness depend on provider behavior. This is a possible temporary workaround, not the preferred core fix.

Requirements shared by the proposed fixes

• Preserve the configured namespace and protect in-progress local writes. Serializing pulls alone does not protect files that a local model is writing. If dirty paths are skipped, retain their eligibility for a later refresh instead of advancing a watermark past them.
• Pull remote changes before invalidating the catalog. Invalidation alone only rebuilds from existing local files; pulling alone may leave a populated catalog stale.
• Coalesce overlapping refreshes without treating a configuration-only pull as a completed runtime refresh. Sequence catalog backfill/invalidation so an older rebuild cannot erase a newer invalidation.
• Define failure, timeout and cancellation behavior. A timeout does not prove a provider has stopped mutating its cache; the tested provider checks cancellation only on entry.
• Preserve normal cached-read availability. The datastore design explicitly uses cache-only reads during writes; polling provides eventual visibility, not a transactional snapshot or immediate consistency guarantee.

The likely implementation locations are a new runtime poller under src/serve/, lifecycle wiring in src/cli/commands/serve.ts, and catalog invalidation through CatalogStore.invalidate(). An explicit freshness option would also require serve protocol validation and changes to handleDataQuery.

These are investigated proposals and source-derived design constraints. No candidate core fix has been implemented or runtime-tested.

Focused regression for the selected fix

• Use two persistent server contexts with separate caches and a provider that honors subdirs.
• Prime B's query catalog, then publish an output from A and wait for its remote commit.
• Run the selected refresh path and assert B returns the new output without restart or local execution.
• Assert the correct namespace/runtime subtree is requested and catalog invalidation occurs after the pull.
• Keep config/access-only refresh restricted. Those paths must not be the incidental mechanism that makes this test pass.
• Verify a concurrent local dirty output remains intact and publishable; skipped paths remain eligible for later refresh.
• If metadata-only refresh is selected, test an attributes predicate and content projection with content initially absent.

Related, distinct reports: https://swamp-club.com/lab/2108 and https://swamp-club.com/lab/2109.

Version check before submission: swamp update --check reported the tested build as up to date.