Skip to main content
← Back to list
01Issue
FeatureShippedSwamp CLIPublic
Assigneeszack

Relationships

#4590 Lazy loading L0: startup benchmark and CLI parity scripts

Opened by zack · 11/10/2026· Shipped 11/10/2026

First step of the lazy command loading plan. Every later step has to show two things the same way: what it did to startup cost, and that it changed no CLI output. Doing either by hand per PR is slow and easy to get wrong.

Context

Measured on a prototype (compiled, warm), routing argv to the invoked command plus build-time bundling takes swamp version from 384 ms / 219 MB to ~111 ms / 109 MB, data get from 416 to ~132 ms, workflow run from 498 to ~358 ms and --help from 391 to ~377 ms. Every PR in that plan quotes this benchmark before and after, and runs this parity check against a binary built from its base commit.

Scope

  • scripts/bench_startup.ts, with a deno run bench-startup task. Given a binary path, it runs a fixed command set in a scratch repo it creates and removes:
    • version
    • --help
    • help data get
    • data get
    • behavior method run
    • workflow run (a one-step command/shell workflow)
    • the telemetry sender entry
  • It reports cold and warm medians of at least 5 runs: wall time and peak RSS. Cold means deleting $TMPDIR/deno-compile-<binary-name>.cache before each run. Peak RSS comes from wait4 rusage: spawn with posix_spawn and read ru_maxrss from os.wait4. The script can call python3 -I, or use a Deno FFI shim if a Deno-only tool is preferred.
  • scripts/cli_parity.ts, with a deno run cli-parity task. Given two binaries, it compares stdout, stderr and exit code for:
    • --help, help, help --include-hidden
    • <path> --help and help <path> for every command path in the first binary's help --include-hidden output
    • completions bash|zsh|fish
    • a mistyped word at the root and in every group (the suggestion list)
    • version --json
  • SWAMP_NO_TELEMETRY=1 and SWAMP_NO_UPDATE_CHECK=1 are set for every run of either script.
  • Document both in contributing/, including when to rerun them: every PR in the lazy-loading plan, and any Deno version bump once the bundled build ships.

Not in scope

CI gating on the numbers.

Acceptance

  • The benchmark prints a table for a given binary.
  • Two binaries with different file names do not share a code cache.
  • The parity script exits non-zero on a difference and prints the first differing invocation. Run against two builds of the same commit, it passes.
  • Unit tests cover argument parsing, median calculation and the diff report, with no subprocesses.

Release impact

Only scripts/ and contributing/ change, so affectedCommands is empty and nothing ships in the binary. Risk: none.

02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 5 MOREREVIEW+ 6 MOREPR_LINKED+ 2 MORESESSION_SUMMARIZED

Shipped

11/10/2026, 5:51:44 PM

Click a lifecycle step above to view its details.

03Sludge Pulse
zack assigned zack11/10/2026, 4:57:45 PM

Sign in to post a ripple.