#1955 serve boot: sweepStaleRecords does a full modelMethodRun per stale lease (~12.5s each), serializing boot reconciliation — 4m28s to sweep nothing

What happens

On swamp serve startup, the Boot: sweeping stale records phase expires stale step leases one at a time. Each expiry attempt takes a constant ~12.5s and then fails with Step lease '<id>' does not exist. Each attempt is a full modelMethodRun, not a lookup. With 19 orphaned leases the sweep took 268s, and total startup was 4m32s — 98.5% of it in this single phase. /ready does not serve 200 until it finishes.

Timeline from a real boot

21:22:33.577  Loaded serve config from "/workspace/.swamp/serve.yaml"
21:22:33.589  Initializing repository at "/workspace"
21:22:36.642  HA: detached runs, pending-run durability
21:22:36.819  Boot: reaping stale runs via tracker
21:22:36.912  Boot: reconciling workflow run state
21:22:37.355  Boot: sweeping stale records          <-- 268.2s
21:27:05.507  Boot reconciliation: swept 0 lease(s), 0 pending dispatch(es), 1 worker(s)
21:27:05.507  Boot: starting scheduler
21:27:05.718  WebSocket API server listening on "ws://127.0.0.1:9090"
21:27:05.833  Startup complete — /ready is now serving 200

Every other phase is sub-second. Repo init + extension bundling is 3.05s; the scheduler start, WS listen and ready flip together take 0.33s.

The 19 failed lease expiries, at a constant interval

21:23:16  (first)     21:24:56  +13.0s     21:26:23  +12.0s
21:23:28  +12.0s      21:25:08  +12.0s     21:26:36  +13.0s
21:23:41  +13.0s      21:25:21  +13.0s     21:26:48  +12.0s
21:23:53  +12.0s      21:25:33  +12.0s     21:27:01  +13.0s
21:24:06  +13.0s      21:25:46  +13.0s
21:24:19  +13.0s      21:25:58  +12.0s
21:24:31  +12.0s      21:26:11  +13.0s
21:24:43  +12.0s

Each emits:

Failed to expire stale lease "1627c6e3-f604-465a-8458-dd7914b5d2c4":
  "Step lease '1627c6e3-f604-465a-8458-dd7914b5d2c4' does not exist"

19/19 end in does not exist. Net result of the whole 4.5 minutes: swept 0 lease(s), 0 pending dispatch(es), 1 worker(s).

Root cause in the source

src/serve/boot_reconciliation.ts:139-172 — the sweep runs a full model method execution per lease, sequentially:

for (const { attrs, modelName } of await loadAttrsForType(repo, STEP_LEASE_MODEL_TYPE)) {
  if (attrs.state !== "active") continue;
  const leaseId = attrs.leaseId;
  if (typeof leaseId !== "string") continue;
  try {
    await transition({
      typeArg: STEP_LEASE_MODEL_TYPE.normalized,
      definitionName: modelName,
      methodName: "expire",
      inputs: { leaseId, error: "orchestrator restart" },
    });

transition defaults to defaultRunTransition (same file, :111-135), which calls createWorkerModelRunDeps(...) inside the function — so the deps are rebuilt on every iteration — and then drives the whole modelMethodRun pipeline: definition lookup, evaluated-definition load/save, model lock, writeResource, catalog update. That is the ~12.5s. It is not a timeout; it is a full method run being used to flip one record's state field, one at a time, with no batching or concurrency.

Second defect: the lease is readable one way and not the other

The sweep selects a lease by reading it off disk — repo.findAllForType() plus repo.getContent() in loadAttrsForType (:854-879) — and only proceeds when the parsed content says state === "active". The artifact demonstrably exists.

The method it then invokes fails to read that same lease (src/domain/models/worker/step_lease_model.ts:80-88):

async function readLease(context, leaseId) {
  const raw = await context.readResource!(leaseDataName(leaseId));
  if (raw === null) {
    throw new Error(`Step lease '${leaseId}' does not exist`);
  }

So readResource("lease-<leaseId>") returns null for a lease the sweep just read successfully. 19/19 fail this way. One candidate cause is ruled out: the modelName: data.tags["modelName"] ?? "" fallback at :873 is not firing — the catalog has zero rows with a blank model_name.

Third factor: the scan is O(all leases ever)

loadAttrsForType reads and JSON-parses every latest-version step-lease artifact to find the active ones. On this install:

type_normalized         rows     is_latest
swamp/step-lease        30196    10115
swamp/pending-dispatch     86       43
swamp/worker            20643       14

10,115 artifact reads to find 19 active leases, on every boot.

Not datastore-specific

sweepStaleRecords is called unconditionally from src/cli/commands/serve.ts:2669 with no datastore or control-plane branch, and reads through the local FileSystemUnifiedDataRepository. Any serve on any datastore configuration executes this path.

Impact

• serve startup scales linearly with orphaned lease count at ~12.5s each.
• The entire window is downtime: no scheduler, no WS API, /ready not serving.
• It gets worse over time — see below.

Contributing factor: the orphans cannot be cleared

swamp run doctor --fix reports reaped: 0, exit 0, empty stderr, against 25 stale runs. The cause is in src/infrastructure/persistence/run_tracker_store.ts:357-373 — reapStaleRuns, which run doctor --fix calls (src/cli/commands/run.ts:224):

const currentHostname = hostname();
for (const run of stale) {
  const shouldReap = run.hostname === currentHostname
    ? isProcessDead(run.pid)
    : true; // Cross-machine: rely on TTL alone

isProcessDead (src/infrastructure/runtime/process.ts:29-42) is Deno.kill(pid, "SIGCONT") and reports "alive" whenever the signal succeeds.

The hostname check assumes a hostname identifies a PID namespace. It does not. All three swamp containers on this host — serve, worker, serve-move — report the same Docker hostname, and each runs swamp at a low PID in its own namespace. Every leaked row carries pid: 7. So when serve evaluates a row written by the worker container, run.hostname === currentHostname is true, and it signals PID 7 in its own namespace — which is serve itself, alive. shouldReap is false for every row, forever.

The : true branch is the one that would work here, but the shared hostname prevents it from being reached.

The boot-time path reapDeadProcessRuns (:376-390) fails for a second, separate reason — it skips its own PID explicitly:

if (run.hostname !== currentHostname) continue;
if (run.pid === Deno.pid) continue;
if (!isProcessDead(run.pid)) continue;

At boot, serve is PID 7, so run.pid === Deno.pid skips every row before the liveness check runs at all.

swamp run gc cannot help either: it never deletes rows marked running.

There is consequently no supported way to clear these records, so they accumulate and each restart pays ~12.5s per accumulated lease. Our tracker held 19 at the boot above and holds 25 now.

Expected

• Expiring a stale lease does not require a full model-method run. A direct state write, or at minimum a batched one, is proportionate to the work.
• createWorkerModelRunDeps is hoisted out of the per-lease loop.
• readResource resolves a lease that loadAttrsForType just read, or the sweep stops selecting leases it cannot act on.
• The scan is indexed on state rather than reading every historical lease.
• Boot reconciliation does not gate /ready on a best-effort cleanup pass.
• Run liveness is keyed on something that actually identifies a PID namespace (instance id / boot id / container id), not on hostname(), so that run doctor --fix can reap runs whose owning process is genuinely gone.

Environment

swamp 20260902.023341.0-sha.820f22f0, linux/x86_64, Docker. Datastore @magistr/mongodb-datastore (control-plane: not available → Control-plane store: local filesystem fallback), serve mode local, --auth-mode=token.