Relationships
#2435 serve: boot hydration ignores `hydrationStrategy: lazy` and downloads every payload
Opened by swamp_lord · 9/23/2026
Summary
swamp serve does a full pull at startup even when the datastore is
configured with hydrationStrategy: lazy. A lazy CLI setup against the same
datastore downloads metadata only.
Environment
- swamp built from
main@1da85ec8(20260923.175528.0-sha.1da85ec8), unmodified @swamp/s3-datastore@2026.09.23.1(MinIO),managedConfig: true,hydrationStrategy: lazy- Each instance starts with an empty
HOMEand a repo containing only.swamp.yaml(stateless pod)
Steps to reproduce
- Operator repo on the datastore with
hydrationStrategy: lazy. swamp model create command/shell payload-maker, then runexecute40 times with a ~27 KB output, giving 160rawpayload objects in the bucket.- Baseline: a new empty repo runs
swamp datastore setup extension @swamp/s3-datastore --config '{…same prefix…}' --hydration-strategy lazy --skip-migration. - Serve: a fresh instance runs
swamp serveuntil/healthis 200. - Count
data/**/metadata.yamlanddata/**/rawon each instance's disk.
Expected
Serve's boot downloads metadata only, like the lazy setup.
Evidence
hydrationStrategy: lazy
raw payload objects in datastore: 160
fresh CLI (lazy setup) metadata: 160 raw: 0
fresh serve after boot metadata: 160 raw: 160
serve log: Pulled 365 file(s) from remote datastoreFull transcripts and logs are available on request.
Root cause (on main)
- The CLI applies lazy hydration in
requireInitializedRepo():registerDatastoreSync({ … metadataOnly: datastoreConfig.hydrationStrategy === "lazy" })(repo_context.ts≈L746). - Serve uses
requireInitializedRepoUnlocked()(L918), which does no initial pull, then callshydrateLocalCache({ syncService, catalogInvalidate, signal, namespace })(serve.tsL1961). HydrateLocalCacheDeps(serve/boot_reconciliation.tsL58–63) has nometadataOnlyfield, sopullChangedalways does a full pull.
Also: with managedConfig, a failed boot hydration is caught and only logged at WARN (boot_reconciliation.ts L87–91, Startup cache hydration failed), and serve starts without its config. It should fail startup.
Impact
On a real store (~48.5k objects), boot pulled everything and then exceeded the default ~1 GB V8 heap (Fatal JavaScript out of memory), crash-looping serve.
Suggested fix
Add metadataOnly to HydrateLocalCacheDeps, pass isCustomDatastoreConfig(datastoreConfig) && datastoreConfig.hydrationStrategy === "lazy" from serve, and make hydration failure fatal when managedConfig is on.
Related
#2152 (open, lazy metadata-walk performance), #2418 (open, poller fast path after lazy hydration). Neither covers serve's boot pull.
#2404 (open): with a namespaced datastore, lazy hydrateFile writes to <ns>/<ns>/…, so reads fail with ENOENT. The fix here isn't blocked on it, but namespaced serve deployments will hit #2404 as soon as boot actually goes lazy.
Reproduction configuration
Test environment
| swamp | built from main @ 1da85ec847ef6a8a0ce0297563b2a9fab1f4b2b8, unmodified (20260923.175528.0-sha.1da85ec8) |
| Datastore | @swamp/s3-datastore@2026.09.23.1 on local MinIO (http://127.0.0.1:19000, bucket swamp-repro) |
| Vault | none configured; control-plane _token-secrets only |
| Instances | each serve instance starts with an empty HOME and a repo containing only .swamp.yaml |
.swamp.yaml:
swampVersion: 20260923.175528.0
initializedAt: "2026-09-23T18:30:07.855Z"
repoId: 29569fdd-0694-4e48-b05c-2c9bdc557383
tools:
- claude
gitignoreManaged: true
datastore:
type: "@swamp/s3-datastore"
config:
bucket: swamp-repro
prefix: 03-serve-boot-ignores-lazy-20260923183002
endpoint: "http://127.0.0.1:19000"
forcePathStyle: true
region: us-east-1
hydrationStrategy: lazy
managedConfig: trueserve.yaml (equivalent of the flags used; the run passed these on the command line):
host: 127.0.0.1
port: 19131
hot-reload: true
auth:
mode: noneServe command:
swamp serve --repo-dir <repo> --config serve.yamlOpen
No activity in this phase yet.
Sign in to post a ripple.