Lab #3962: swamp serve on a v3 datastore never settles a workflow run left running by a stopped instance: boot reaper skips it as belonging to a remote instance

Summary

Found while triaging #3958. On a format 3 (v3) datastore, a workflow run left running by a stopped swamp serve is never settled by serve in another checkout, and a quick restart of the same checkout does not settle it either. A v3 serve keeps its control plane in the checkout (swamp-club#3534), so it writes no heartbeats to the bucket, and the boot reaper has nothing to judge another instance's run by.

Environment

swamp 20261026.014016.0-sha.7e1aad04 (Linux x86_64), @swamp/s3-datastore 2026.10.24.2, --format 3, the local fake S3 from #3958. No AWS.

Reproduction

Leave a run running in the replica: the clean shutdown with a slow bucket from the companion issue. Then do either of these:

Second checkout. Start serve in a copy of the repo without .swamp/ and with a fresh SWAMP_HOME. Read the run at boot and again 70 s later.
Same checkout. Restart the original serve within 90 s of its last tracker heartbeat, and read the run every 30 s for 4.5 minutes.

Expected

A run whose owning instance has stopped, and released the v3 writer lease, is settled as interrupted, and the settlement replicates.

Actual

Second checkout:

Restoring this checkout's copy … from replication lineage "000000000001"
Skipping workflow run "53bf95b3-…" (workflow: "slow") — belongs to remote instance "47d96377-b0a3-4468-918e-8c74b06b2828"

The run reads running at boot and 70 s later, and stays that way for as long as that serve runs.

Same checkout:
```
Recovering work an interrupted command left in this checkout…
Skipping workflow run "53bf95b3-…" (workflow: "slow") — tracker reports still running
```
The run reads running for the whole 4.5 minutes. The tracker row had been reopened as running. reapDeadProcessRuns skips it because the row carries the old instance id, and reapStaleRuns needs a heartbeat older than 90 s; serve 2 booted about 54 s after the last one. The reap runs only at boot.

The run healed only when the original checkout restarted more than 90 s after the last heartbeat (Reaping orphaned workflow run … "tracker confirmed stale"). That restart also logged that it dropped 4 commit(s) the replica never confirmed: another checkout wrote to the datastore first, so they are lost.

Analysis

reapOrphanedWorkflowRuns (src/domain/workflows/orphaned_run_reaper.ts) skips a run with a foreign instance id when no heartbeat lookup is given ("belongs to remote instance"). On v3, serve passes none, because hasRemoteControlPlane is false.
The bucket already records that a v3 writer stopped: its lease release (for example released-000000000002). The reaper could use that, or the stopped serve could leave a record that its runs are orphaned.
The same-checkout skip comes from a tracker row that shutdown never settles. #3958 tracks that for format 2.

Scheduled flaky-test detection is failing

Manual: the supported extension dependency categories, the canonical form and the lock rule, rewritten once against the final design

Registry audits v2 extension archives as written, and a range costs the loose point only without a valid lock

Extension consumers keep the publisher's bundle: pull, update and auto-resolve stop rebuilding (direction A for #3634)

Extension push: inputs that open up (workspace root lock, ancestor config refused by name, JSR dist-tags) and the skill docs rewritten once

Extension push ships the canonical form: every npm:/jsr: import pinned exact, and a v2 dependency-audit.json

stagecraft studio ticket columns: round-5 review lows from #3828 (source watch cost, relationText lookup, cross-instance claim)

A long-lived serve client for extensions, and a stable rate_limited error code

swamp serve on a v3 datastore never settles a workflow run left running by a stopped instance: boot reaper skips it as belonging to a remote instance

swamp serve on a v3 datastore: a run interrupted at a clean shutdown is never committed when the bucket is slow, and stays running for every other checkout

Support creating 1Password vaults for new clients

Filesystem datastore: an unclean shutdown leaves zero-byte data versions as the latest version, and `doctor datastores --repair` does not detect them

datastore_v3_replica_verify_test stall case flakes under the full integration run: restore.pid is read before the stub writes it

Generated state schemas require every child of nested objects, so normal resources warn on create/get

Add LinkedIn as a login provider

Showcase reward backfill can double-pay 5,000,000 points when replicas start together

Showcase editor: caption counter is not announced to screen readers

Profile showcase read: approved list sorts on review.reviewedAt in memory

Showcase countByStatus: $match live statuses before the $group

Showcase admin queues: index-serve the updatedAt sort on approved/rejected

Datastore rework Phase 5: content-level authorisation for forwarded definition and evaluated-file writes

Annotate and retain past workflow runs and data after the fact

Docs: --trusted-hosts accepts host:port for public addresses on non-default ports

A workflow step's method output is invisible to `data query` and `data list`, and the method summary's retrieval command returns an older run's data

Skill propulsion rule is silent when no transition is satisfied because stage work is outstanding

package.json mode: no loose-specifier warning for a range push ships as the exact installed version

extension fmt (and any command) creates or rewrites deno.lock when the extension lives in the repo's loaded extensions/ dir

serve shutdown does not wait for worker status writes, and its listener accepts workers after the gateway is disposed

Registry rescore: secret-safe-docs passes a secret README line on versions whose stored metadata predates the sensitive flag (follow-up to #3783)

stagecraft: rename work mode `method` to `behavior`, matching swamp's rename of models to behaviors

extension quality --json reports percentage but not the grade that extension info and the registry show

auth token create --json fields key and signinToken are undocumented; no token-mode service setup page

trigger.schedule is evaluated in the serve host's local time zone, with no zone field and no doc saying so

workflow validate/run say "Workflow not found" for a file that doctor workflows explains is misnamed

telemetry stats says "No telemetry data found" right after successful sends; say the spool is empty because it flushed

Docs: publish typed-secrets Phase 4.6 expression and delivery semantics

Extension registry page: a supported way to include media (images) in the README

CEL: data.latest(...).version is a double, but data.version() only accepts int, so "version - 1" fails at run time

workflow resume --run refuses workflows with the same step name in two jobs, though validate and run accept them

Update public OpenTelemetry manual for remote waterfalls and execution metrics (#3786)

Server-driven warning and info messages to CLI users, targeted by swamp version (exploration)

Serve: delegate namespace-scoped extension pulls and reload without granting administrator access

workflow cancel --server ignores --ca-cert and SWAMP_CA_CERT

Preserve active worker execution across serve restarts and upgrades; reconnect and reattach dispatches

v3: chunk durability barrier costs a full drive-cache flush per command on macOS; use an ordering barrier

vary/forEach object values with no key/name/id produce data names Windows refuses

issue-lifecycle start drops issue comments: context.comments is always empty

swamp core: a non-string sensitive value reads back as a JSON string

Generated vault keys carry raw data-name and vary characters that strict vault providers reject

bundleExtension 'bundles on different caches do not wait for each other' test fails under CPU load

Relationships

#3962 swamp serve on a v3 datastore never settles a workflow run left running by a stopped instance: boot reaper skips it as belonging to a remote instance