Lab #3109: Settle expired signal waits from swamp serve

Problem

Split out of swamp-club#3094. A wait_for_signal deadline is noticed only when something looks: workflow signal, workflow waits and workflow resume each settle an overdue wait as timed_out (outcomeAt in src/domain/workflows/signal_wait_cleanup.ts). Under swamp serve nothing looks, so an expired wait stays suspended and its failed handlers do not run until someone resumes the run.

Proposal

The continuation sweep of swamp-club#3108 also lists registered waits and, for each one past its deadline with no outcome, creates a timed_out outcome and attempts continuation. Every instance may sweep; the create decides between a timeout and a signal arriving at the same moment.
After downtime, overdue waits are caught up on the first sweep. The deadline is absolute; how soon the handlers run depends on serve being up.
A maximum timeout configured on serve. Today one year is hard-coded in the task schema.

Open decisions

The maximum timeout: where it is enforced (when the workflow is validated, or when the wait opens) and what happens to a wait already open with a longer one.
Whether an expired approval gate should be failed by the sweep. Today it stays suspended (evaluateApprovalTimeout is checked only on approve, reject and listing); changing it alters existing workflows.
Whether the sweep settles waits on a custom datastore, where this host may not have the run record. Settling needs only the registration; continuing needs the run.

Acceptance criteria

A wait past its deadline fails its step with wait_timeout, and its failed handler runs, with no client action.
Waits that expired while serve was down are settled on the first sweep after boot.
A signal accepted before the deadline still holds when the sweep runs after it.

Tests

Integration tests with a fake clock and ordered creates, never sleeps: signal against timeout, deadline catch-up after downtime, two instances sweeping the same wait.

Depends on

swamp-club#3108 (continuation claim and sweep). Without it a timed-out wait would be settled but its run not resumed.

References

swamp-club#3094, swamp-club#3093, the Timing out paragraph of the Wait for Signal section in design/primitives/workflows.md.

A nested structural swamp still waits on its run's lock when the lock holder is not a same-host ancestor (cross-host worker, --server into a non-ancestor serve)

A nested structural swamp that outlives the run that started it keeps skipping that run's lock while the holder writes

Show signal waits to remote clients and the dashboard

Settle expired signal waits from swamp serve

Resume signalled workflow runs automatically under swamp serve

Deliver workflow signals through a write-once outcome record instead of writing the run record

Forward held-lock identity over --server so a loopback nested swamp does not wait on its caller's lock

A nested structural swamp on a same-host remote worker waits on its own step's lock held by swamp serve

Fail fast when nested structural commands under parallel runs wait on each other's locks

Docs: remove followUpActions from the MethodResult reference in the extension model manual

signal_test: already-settled test fails by chance when a random UUID spells the sender name

Add a wait_for_signal workflow step that pauses a run for a JSON message

Deliver workflow signals through swamp serve

run doctor through serve interrupts a live run of another serve instance when no heartbeats are recorded

Flaky test: WalSink replayed segments are delivered before events written after replay leaves one WAL segment

Remove followUpActions: no model or extension can produce them

Workflow schedule watcher and workflow edit symlink lookup ignore the managed config workflows dir

main is red: repository_dirty_coverage_test fails for UnifiedData.collectGarbage(orphaned deferred write)

isProcessGone always reports alive on Linux under the test task permissions, failing 9 tests on main

Flaky test: WalSink replayed segments are delivered before events written after replay

serve: relative tls cert-file/key-file and --config resolve against the working directory, not the repository

findBySpec/findByTag lose a workflow step's output after a version is deleted, pruned or rolled back

serve HA: worker enrollment on a replica that lacks the enrollment token's definition may create a second definition (unverified)

serve HA: pull a token's definition on an auth miss so a peer accepts a new token at once

Run tracker keeps some interrupted workflow rows forever: retention waits for markSettled, which several paths never call

run doctor reads every run record on each run to rebuild the run indexes

Serve audit: two sinks with the same name share one emitter cursor, so the second never receives events

Serve audit: a durable sink write that times out is retried while the first call is still pending, so the store can hold duplicate sequences

Flaky test: ModelResolver data accessors for a renamed model return the earlier id's record from findBySpec

data query --limit returns fewer live records than exist and reports limited: false when stale catalog rows fall inside the limit

data get: help example passes --run latest, which is not a recognised run id

workflow reject leaves sibling gates waiting_approval and dependents pending in a failed, retryable run

Show expired approval gates as expired, with a Cancel action in the dashboard

digitalocean codegen leaves an orphaned security_secret.ts model file after its endpoint left the spec

workflow cancel cannot cancel a suspended run whose workflow file was deleted, and cancel --all silently skips it

A job alone in its level starts after the abort, and its unstarted step is recorded as a real failure instead of settledByAbort

workflow approve racing workflow cancel loses the cancel: the run returns to suspended after cancel reported cancelled

data query: does not follow data rename forwarding that data get follows

Unit tests share fixed /tmp catalog paths, so concurrent runs corrupt them and fail every later run

data query: renamed data items are not found by their old name

data query: model lookup by id and orphan recovery parity with data get

workflow approve on a run that is not suspended prints a fatal error with a stack trace

data query: no way to target a workflow's latest run

data query: binary content lacks contentEncoding/base64 parity with data get

data query: no fallback from spec name to data instance name

data query: no definitionHash or garbageCollection fields in query results

A nested structural swamp spawned by swamp serve skips every lock serve holds, including other runs' locks

data query: confirm catalog parity with data get for custom datastores

data query: single-result --json mode for scripts migrating from data get

Docs: document data query --single and fix the stale --limit default in reference/data.md

Relationships

#3109 Settle expired signal waits from swamp serve