#146

feat: add support for Nix via a flake

s12d ago

← Back to list

01Issue

FeatureOpenSwamp CLITeam

AssigneesNone

Relationships

#1020 swamp serve startup/shutdown is very slow on large repo and lacks diagnostics

Opened by evrardjp · 7/7/2026

Summary

swamp serve is extremely slow to start on my repository/data, consumes very high memory/CPU, and there does not appear to be a way to see what it is doing during startup.

I tried running with swamp serve -v, but it did not reveal the slow phase or explain where the time/memory goes.

Environment

swamp version: 20260706.220403.0-sha.72b7ed98
repository: /var/home/evrardjp/git/evrardjp/local-infra
.swamp size: ~9.1G
model count from swamp model search --json: 47 results

Observed behavior

Startup takes minutes before useful output appears. Example run:

❯ swamp serve
19:51:43.232   warning serve                auth-mode is 'none' — this mode is deprecated and will be removed in a future release. Use --auth-mode token for authenticated access. See https://swamp-club.com/manual/how-to/swamp-serve/set-up-token-auth
19:51:43.233   info    serve                Initializing repository at "/var/home/evrardjp/git/evrardjp/local-infra"
19:51:43.317   info    swamp·domain·access·policy-snapshot Loaded policy snapshot: 0 active grant(s), 0 group(s)
19:51:43.317   info    serve                Policy snapshot loaded (reload mode: "manual")
19:52:08.605   warning serve                Reaping orphaned workflow run "fc497990-5ed4-4066-b44f-10dd333c8f68" (workflow: "github-project-stale-classification-daily")




19:55:32.427   info    serve                Scheduled workflow "cluster-pool-sync" ("*/5 * * * *")

It takes more than 3 minutes to output the first scheduled-workflow line after the orphan reaping log. Regardless of the schedule, here it is 5 minutes, but it would happen if it is 1 minute.

Side info: My machine is starting to struggle, see CPU spiking:

4190633 evrardjp  20   0   22.4g   2.9g   4960 D  74.7   6.1   1:38.25 swamp .

Notes:

CPU was ~74% in this sample.
starting is very slow.
closing is also very slow.

Expected behavior

swamp serve should either start in a reasonable time on a large existing repo or clearly identify the slow startup phase.
Verbose logging should make it possible to diagnose what is happening: which startup step is running, how many records/files are being scanned, per-phase timing, maybe memory/heap metrics.
If this is expected for large datasets, there should be a documented way to profile/trace startup or identify the dataset/model/workflow causing the slowdown.

Request

Is there a way to track what swamp serve is doing during startup/shutdown today? If not, please add diagnostics/progress logging or a tracing/profiling mode for swamp serve startup/shutdown so users can understand and report actionable bottlenecks.

02Bog Flow

Open

7/7/2026, 8:06:06 PM

No activity in this phase yet.

03Sludge Pulse

evrardjp commented 7/7/2026, 8:49:53 PM(edited)

Interesting that I used my agent to generate this issue, but the format of this does NOT seem to match the usual issues expectations (swamp version, etc.).

Feel free to close if it's not correct.

evrardjp commented 7/7/2026, 8:51:57 PM

Oh, well, I guess that's it:


<--- Last few GCs --->

[10545:0x565482d8b000]   261167 ms: Mark-Compact (reduce) 4092.1 (4100.4) -> 4092.1 (4100.4) MB, pooled: 0.0 MB, 415.14 / 0.00 ms (average mu = 0.005, current mu = 0.000) last resort; GC in old space requested
[10545:0x565482d8b000]   261585 ms: Mark-Compact (reduce) 4092.1 (4100.4) -> 4092.1 (4100.4) MB, pooled: 0.0 MB, 417.26 / 0.00 ms (average mu = 0.002, current mu = 0.000) last resort; GC in old space requested



#
# Fatal JavaScript out of memory: Reached heap limit
#
==== C stack trace ===============================

    swamp(+0x23efc5e) [0x565448c54c5e]
    swamp(+0x23eec7b) [0x565448c53c7b]
    swamp(+0x23f0ec8) [0x565448c55ec8]
    swamp(+0x249fb75) [0x565448d04b75]
    swamp(+0x2656487) [0x565448ebb487]
    swamp(+0x2655bb4) [0x565448ebabb4]
    swamp(+0x265537d) [0x565448eba37d]
    swamp(+0x2648d9b) [0x565448eadd9b]
    swamp(+0x264873c) [0x565448ead73c]
    swamp(+0x2626f6c) [0x565448e8bf6c]
    swamp(+0x261d471) [0x565448e82471]
    swamp(+0x27876d3) [0x565448fec6d3]
    swamp(+0x278ad7f) [0x565448fefd7f]
    swamp(+0x278ad9b) [0x565448fefd9b]
    swamp(+0x278a906) [0x565448fef906]
    swamp(+0x278d580) [0x565448ff2580]
    swamp(+0x278ad8c) [0x565448fefd8c]
    swamp(+0x27872ff) [0x565448fec2ff]
    swamp(+0x2786fb0) [0x565448febfb0]
    swamp(+0x24ec505) [0x565448d51505]
    [0x7f075be81ef6]
Trace/breakpoint trap      (core dumped) swamp serve

Scorecard: beta/rc-only extensions show "not scored" despite successful scoring (GET /score resolves only latestVersion)

`swamp extension search --channel beta|rc|stable` ignores the filter — all channels + the no-flag default return an identical result set

Beta-channel extensions are never scored — perpetual 'Scoring is taking longer than expected'

feat: allow models to call other models mid-execution via context.runModel()

swamp-club: enable OAuth provider for swamp serve collective-based admission

extension update/install reports new version while on-disk pulled files stay old

command/shell: step with timeout + a surviving child process reports exit -1 despite command exiting 0

swamp serve startup/shutdown is very slow on large repo and lacks diagnostics

Vault push log overstates what left the machine: Pushed N vault file(s) to datastore counts every dirty file, not just vault files

serve-auth: mode: oauth — device grant client + collective-based admission

serve-auth: enable OIDC Provider + Device Authorization plugins in swamp-club

Orphaned data (migrated-away/deleted model) can't be GC'd or deleted

Docs: document swamp data prune (orphaned-data reclamation)

UAT: CLI coverage for swamp data prune (orphaned-data reclamation)

Session credential revocation scans entire credential map on every dispatch completion

Docs: update worker-commands.md for --all flag and JSON output change

datastore migrate-index aborts on hardcoded 30s S3 requestTimeout for large shards

swamp worker list: hide disconnected workers by default, add --all flag

swamp serve should survive bad extension failures

Worker fleets phase 7: how-to guides and explanation docs

swamp serve: catch unhandled promise rejections from extensions

[@adam/cfgmgmt] SSH helper can crash swamp serve via unhandled finally() rejection

Docs: add --server-token flag to worker connect reference

Add --server-token to worker connect for authenticated orchestrators

Docs: document --trusted-hosts flag for swamp serve

Document SWAMP_SERVE_EXTRA_HEADERS for reverse proxy/tunnel setups

Add --trusted-hosts flag to swamp serve for Docker/Kubernetes worker connections

Allow swamp clients to send custom headers when connecting to swamp serve

Docs: update serve-flags.md default queue timeout from 60s to 10m

Worker fleets phase 6: raise default queue timeout to 10 minutes

Worker fleets phase 4b: capacity slots and concurrent dispatch

MethodContext docs/comments mention context.putSecret but runtime exposes vaultService.put

Docs: add --concurrency flag and SLOTS column to worker-commands reference

install.sh still points users to systeminit/swamp

Support user systemd units for 'swamp serve daemon enable'

swamp serve scheduled execution runs steps in an isolated data scope: readResource sees empty state, writeResource never reaches the model-scope chain — stateful scheduled workflows duplicate instead of deduplicating

Support installing swamp via mise GitHub backend

Worker fleets phase 4a: dispatch runner extraction — child process per dispatch

Docs: add swamp worker daemon enable/disable/status to worker-commands reference

Worker fleets phase 3 PR2: swamp worker daemon enable

Docs: add drain, lifecycle policies, and exit-code contract to references

Worker fleets phase 3 PR1: drain state machine, lifecycle policies, signal handling

Leaderboard search excludes ghosts — a visible ghost can't be found by its alias

issue-lifecycle: plan lifecycle entry silently dropped when summary exceeds 1000 chars

Docs: add worker verify, --verify-on-enroll, and env-var config to references

Labs - Filters do not apply to Blocked status and Billing Tier category

Worker fleets phase 2 PR2: env-var config, swamp worker verify, --verify-on-enroll

Docs: add --type flag to issue edit reference and how-to

Docs: update manual with whole-field expression syntax for workflow task inputs/globalArgs

Allow changing an existing issue to security/confidential after submission

Allow dynamic object expressions for workflow task inputs/globalArgs

Docs: update enrollment token references for fleet tokens (--max-enrollments)

Worker fleets phase 2 PR1: fleet tokens (--max-enrollments, bindings, worker naming)

$identify events are never consumed — anonymous web history never binds to the account

Worker fleets phase 1 PR3: boot reconciliation and integration test

Login success still shows login tip

Support confidential visibility for non-security issues

Fine-grained scopes for swamp.club access tokens

Add associative info to user profiles: collective membership + collaborative contributions

Remove comments from /feed and redesign the page

feed_post_score_events queue grows unboundedly (no TTL, no delete-on-delivery)

Make paid subscribers visually stand out (badge, profile popup summary)

Relax the profanity filter

Giphy integration via /giphy command

@-mention autocomplete for users when writing lab ripples

Worker fleets phase 1 PR2: step_queued events, pending-dispatch model, queue introspection

Docs: add swamp worker queue to worker-commands reference

/lab shipped list is sorted by issue number, not ship date

Docs: update manual for ephemeral data and step-output dependency deferral

Docs: update remote execution references for elastic queueing

Support dynamic host discovery from external sources

Worker fleets phase 1: elastic queueing

catalog backfill should support scoped population for targeted lookups

Docs: document global skill sync behavior for custom tools in manual

swamp update syncs global skills to default agent directories instead of configured custom Pi skillsDir

coerceToSuffix does not sanitize path-unsafe characters, causing forEach items and vary dimensions with / to produce invalid data instance names

Badge for activity on other people's issues

swamp serve: add @keychain=<service>/<account> indirection for secret args on macOS

swamp serve: eliminate wrapper scripts via @vault= secret refs and native startup hooks

Automate mechanical frontmatter fixes in the fixing stage