Skip to main content
← Back to list
01Issue
BugOpenSwamp CLIPublic
AssigneesNone

Relationships

#2435 serve: boot hydration ignores `hydrationStrategy: lazy` and downloads every payload

Opened by swamp_lord · 9/23/2026

Summary

swamp serve does a full pull at startup even when the datastore is configured with hydrationStrategy: lazy. A lazy CLI setup against the same datastore downloads metadata only.

Environment

  • swamp built from main @ 1da85ec8 (20260923.175528.0-sha.1da85ec8), unmodified
  • @swamp/s3-datastore@2026.09.23.1 (MinIO), managedConfig: true, hydrationStrategy: lazy
  • Each instance starts with an empty HOME and a repo containing only .swamp.yaml (stateless pod)

Steps to reproduce

  1. Operator repo on the datastore with hydrationStrategy: lazy.
  2. swamp model create command/shell payload-maker, then run execute 40 times with a ~27 KB output, giving 160 raw payload objects in the bucket.
  3. Baseline: a new empty repo runs swamp datastore setup extension @swamp/s3-datastore --config '{…same prefix…}' --hydration-strategy lazy --skip-migration.
  4. Serve: a fresh instance runs swamp serve until /health is 200.
  5. Count data/**/metadata.yaml and data/**/raw on each instance's disk.

Expected

Serve's boot downloads metadata only, like the lazy setup.

Evidence

hydrationStrategy: lazy
raw payload objects in datastore: 160
fresh CLI (lazy setup)   metadata: 160   raw: 0
fresh serve after boot   metadata: 160   raw: 160
serve log: Pulled 365 file(s) from remote datastore

Full transcripts and logs are available on request.

Root cause (on main)

  • The CLI applies lazy hydration in requireInitializedRepo(): registerDatastoreSync({ … metadataOnly: datastoreConfig.hydrationStrategy === "lazy" }) (repo_context.ts ≈L746).
  • Serve uses requireInitializedRepoUnlocked() (L918), which does no initial pull, then calls hydrateLocalCache({ syncService, catalogInvalidate, signal, namespace }) (serve.ts L1961).
  • HydrateLocalCacheDeps (serve/boot_reconciliation.ts L58–63) has no metadataOnly field, so pullChanged always does a full pull.

Also: with managedConfig, a failed boot hydration is caught and only logged at WARN (boot_reconciliation.ts L87–91, Startup cache hydration failed), and serve starts without its config. It should fail startup.

Impact

On a real store (~48.5k objects), boot pulled everything and then exceeded the default ~1 GB V8 heap (Fatal JavaScript out of memory), crash-looping serve.

Suggested fix

Add metadataOnly to HydrateLocalCacheDeps, pass isCustomDatastoreConfig(datastoreConfig) && datastoreConfig.hydrationStrategy === "lazy" from serve, and make hydration failure fatal when managedConfig is on.

#2152 (open, lazy metadata-walk performance), #2418 (open, poller fast path after lazy hydration). Neither covers serve's boot pull.

#2404 (open): with a namespaced datastore, lazy hydrateFile writes to <ns>/<ns>/…, so reads fail with ENOENT. The fix here isn't blocked on it, but namespaced serve deployments will hit #2404 as soon as boot actually goes lazy.

Reproduction configuration

Test environment

swamp built from main @ 1da85ec847ef6a8a0ce0297563b2a9fab1f4b2b8, unmodified (20260923.175528.0-sha.1da85ec8)
Datastore @swamp/s3-datastore@2026.09.23.1 on local MinIO (http://127.0.0.1:19000, bucket swamp-repro)
Vault none configured; control-plane _token-secrets only
Instances each serve instance starts with an empty HOME and a repo containing only .swamp.yaml

.swamp.yaml:

swampVersion: 20260923.175528.0
initializedAt: "2026-09-23T18:30:07.855Z"
repoId: 29569fdd-0694-4e48-b05c-2c9bdc557383
tools:
  - claude
gitignoreManaged: true
datastore:
  type: "@swamp/s3-datastore"
  config:
    bucket: swamp-repro
    prefix: 03-serve-boot-ignores-lazy-20260923183002
    endpoint: "http://127.0.0.1:19000"
    forcePathStyle: true
    region: us-east-1
  hydrationStrategy: lazy
  managedConfig: true

serve.yaml (equivalent of the flags used; the run passed these on the command line):

host: 127.0.0.1
port: 19131
hot-reload: true
auth:
  mode: none

Serve command:

swamp serve --repo-dir <repo> --config serve.yaml
02Bog Flow
◉OPEN○TRIAGED○IN PROGRESS○SHIPPED

Open

9/23/2026, 7:34:34 PM

No activity in this phase yet.

03Sludge Pulse

Sign in to post a ripple.