Lab #2379: Let a method inside serve start a workflow run: context.runWorkflow

Problem Statement

Split out of #2351 (reported by bixu). An extension method running inside swamp serve has no supported way to start a workflow run. MethodContext offers runModel, but nothing that queues a workflow through serve so that placed steps land on workers and the caller gets a run id back.

#2351 itself is being scoped to the evaluation bug that blocked the reporter's workaround (a nested workflow step's workflowIdOrName reading data.latest is evaluated at run start). That fix lets one dynamic step per stage replace the 46-step guarded matrix, but a driver method still cannot start the next stage directly.

Proposed Solution

context.runWorkflow({ workflowIdOrName, inputs }) on MethodContext. It queues a run through serve's run path and returns the run id.

Design questions to settle first

Authorization. The gate precedent (context.approveWorkflowGate / rejectWorkflowGate) was unwired in GitHub PR swamp-club/swamp#1960 (issue #1397) because model code bypassed serve's principal authorization. A method-started run must be authorized against the initiating principal's run grant on workflow:<name>, but MethodContext carries no Principal today, only an initiatedBy string inside tagOverrides.
Serve port. Serve starts runs only inside handleWorkflowRun (src/serve/handlers/workflow_handlers.ts), which registers the run in the ActiveRunRegistry (concurrency caps, cancel, run.attach), writes the control-plane active-run record, and pushes sync after the run. A runWorkflow port would need to reuse that path rather than build an in-process WorkflowExecutionService, which would dispatch placed steps (the dispatcher is process-global) but skip all of the above.
Extension rules. runModel gates cross-extension calls on the caller's manifest dependencies. Workflows carry no extension provenance: extension workflows are identified only by name, and a local workflow can shadow an extension workflow of the same name (CompositeWorkflowRepository).
Run id timing. The run id is minted by WorkflowRun.create and emitted in the started event after lookup, input validation and evaluation, so returning it synchronously means waiting for started or pre-minting it.
Remote execution. On a worker, remote_method_context.ts stubs runModel and the gate calls; runWorkflow would need a stub or a new worker-to-orchestrator verb.
CLI (no serve). Decide whether runWorkflow runs in-process outside serve, where placed steps fail at step time with no worker dispatcher.

Alternatives Considered

In-process nested run from the method: gets worker placement for free but skips the registry, HA reconciliation, authorization and audit.
A loopback webhook the pod signs to itself: rejected by the reporter (moves an internal call onto the network and needs a dedicated secret).

Additional Context

Related: #2351, #1397, GitHub PR swamp-club/swamp#1960
Reporter deployment: k3s pod, swamp serve, managedConfig, S3 datastore, placed steps on a worker fleet

🤖

Reporter-side answers to the design questions, from the harness that would call runWorkflow.

Authorization. Our driver method runs inside serve for the principal that started the parent run: a webhook, a schedule, or an operator. A check of that principal's run grant on workflow:<name> fits us. A webhook-started parent should carry the webhook's own principal. Then a route cannot launch workflows its grants do not cover.

Serve port. Reuse handleWorkflowRun. We rely on the active-run registry for cancel and run.attach, and on the post-run sync. An in-process run that skips those would break our dashboards and our retry path.

Extension rules. Name lookup is enough for us. We call only workflows our own extension bundles. A manifest dependencies check for extension workflows would help, but it does not block us.

Run id timing. We need the run id to record which run owns a stage, not to wait on it. A pre-minted id returned at once suits us best. A wait for started also works.

Remote execution. Our driver never runs on a worker. A stub that throws a clear error on workers is enough.

CLI without serve. A clear error outside serve is fine. We never drive propulsion from a local CLI.

Today's workaround: #2351's step-time evaluation lets us run one unrolled loop of advance steps and dynamic stage steps. It works, but a human or a webhook still starts each segment after a gate. With runWorkflow, the approval itself could start the next segment.

A nested structural swamp still waits on its run's lock when the lock holder is not a same-host ancestor (cross-host worker, --server into a non-ancestor serve)

A nested structural swamp that outlives the run that started it keeps skipping that run's lock while the holder writes

Show signal waits to remote clients and the dashboard

Settle expired signal waits from swamp serve

Resume signalled workflow runs automatically under swamp serve

Deliver workflow signals through a write-once outcome record instead of writing the run record

Forward held-lock identity over --server so a loopback nested swamp does not wait on its caller's lock

A nested structural swamp on a same-host remote worker waits on its own step's lock held by swamp serve

Fail fast when nested structural commands under parallel runs wait on each other's locks

Docs: remove followUpActions from the MethodResult reference in the extension model manual

signal_test: already-settled test fails by chance when a random UUID spells the sender name

Add a wait_for_signal workflow step that pauses a run for a JSON message

Deliver workflow signals through swamp serve

run doctor through serve interrupts a live run of another serve instance when no heartbeats are recorded

Flaky test: WalSink replayed segments are delivered before events written after replay leaves one WAL segment

Remove followUpActions: no model or extension can produce them

Workflow schedule watcher and workflow edit symlink lookup ignore the managed config workflows dir

main is red: repository_dirty_coverage_test fails for UnifiedData.collectGarbage(orphaned deferred write)

isProcessGone always reports alive on Linux under the test task permissions, failing 9 tests on main

Flaky test: WalSink replayed segments are delivered before events written after replay

serve: relative tls cert-file/key-file and --config resolve against the working directory, not the repository

findBySpec/findByTag lose a workflow step's output after a version is deleted, pruned or rolled back

serve HA: worker enrollment on a replica that lacks the enrollment token's definition may create a second definition (unverified)

serve HA: pull a token's definition on an auth miss so a peer accepts a new token at once

Run tracker keeps some interrupted workflow rows forever: retention waits for markSettled, which several paths never call

run doctor reads every run record on each run to rebuild the run indexes

Serve audit: two sinks with the same name share one emitter cursor, so the second never receives events

Serve audit: a durable sink write that times out is retried while the first call is still pending, so the store can hold duplicate sequences

Flaky test: ModelResolver data accessors for a renamed model return the earlier id's record from findBySpec

data query --limit returns fewer live records than exist and reports limited: false when stale catalog rows fall inside the limit

data get: help example passes --run latest, which is not a recognised run id

workflow reject leaves sibling gates waiting_approval and dependents pending in a failed, retryable run

Show expired approval gates as expired, with a Cancel action in the dashboard

digitalocean codegen leaves an orphaned security_secret.ts model file after its endpoint left the spec

workflow cancel cannot cancel a suspended run whose workflow file was deleted, and cancel --all silently skips it

A job alone in its level starts after the abort, and its unstarted step is recorded as a real failure instead of settledByAbort

workflow approve racing workflow cancel loses the cancel: the run returns to suspended after cancel reported cancelled

data query: does not follow data rename forwarding that data get follows

Unit tests share fixed /tmp catalog paths, so concurrent runs corrupt them and fail every later run

data query: renamed data items are not found by their old name

data query: model lookup by id and orphan recovery parity with data get

workflow approve on a run that is not suspended prints a fatal error with a stack trace

data query: no way to target a workflow's latest run

data query: binary content lacks contentEncoding/base64 parity with data get

data query: no fallback from spec name to data instance name

data query: no definitionHash or garbageCollection fields in query results

A nested structural swamp spawned by swamp serve skips every lock serve holds, including other runs' locks

data query: confirm catalog parity with data get for custom datastores

data query: single-result --json mode for scripts migrating from data get

Docs: document data query --single and fix the stale --limit default in reference/data.md

data get --workflow silently returns an arbitrary step's data when several steps share a data name

swamp doctor workflows misses nested and .yml extension workflows that the loader reads

auth whoami says the stored API key is no longer valid when auth.json only holds an env-key identity cache

Docs: swamp skill says cancelling a parent run cancels its nested child runs; the child is left suspended

Namespace advice in the slow-lock warning is wrong for a single repo on a local datastore

Method-run records left running by a dead owner are never reaped

Flaky: usecase_sync_characterization_data_test 'data gc (serve)' asserts markDirty order that varies under parallel load

Worker dispatch rebuilds the definition with Definition.create, so a legacy-named model cannot run on a worker

A cancelled workflow exits before its steps stop, leaving their method-run records at running

Nested swamp structural command in a shell step times out on its parent's per-model lock (SWAMP_LOCK_HOLDER_PID is stripped)

Auth gate blocks nested swamp in workflow shell steps when the credential is in SWAMP_API_KEY

swamp workflow create crashes with an [FTL] ZodError stack trace on an invalid (uppercase) workflow name

swamp model create crashes with an [FTL] ZodError stack trace on an invalid (uppercase) model name

model cancel SIGTERMs a running swamp serve process when the method run belongs to a serve-run workflow

Follow-up action retries ignore the run's abort signal and can outlast a cancel

model cancel --all on a running workflow leaves its method-run records at running, or records them failed instead of cancelled

model cancel SIGKILLs the owner 2 s after SIGTERM, before the step executor's own 3 s kill grace, so the run never records its own cancellation

workflow cancel and supersede of a suspended run leave its jobs running in the cancelled record

Tell timeouts apart from cancels in method-run records, and review the hidden 30 s fallback timer for step-called model methods

A force-exited workflow run stays running with a dead pid: resume, recover and run doctor --fix cannot clear it

A nested workflow that suspends on manual approval forwards its suspended event into the parent run's stream

workflow cancel SIGKILLs the run 2 s after SIGTERM, cutting off always/completed cleanup jobs mid-step

Warn when datastore setup keeps managedConfig but the new datastore has no config tier

Docs: run doctor --fix and workflow recover settle runs left running by a force-exited process

serve: a relative grants-dir resolves against the working directory, not the repository

workflow resume cleanup mode can run an always-gated teardown before a suspended job's approved work

Docs: serve-flags.md should state that relative grants-dir and grants-file resolve against the repository

managedConfig: extension writes update the shared lockfile from a stale cache and overwrite peers' entries

managedConfig: model/vault create and other bulk-push commands re-upload a stale extension lockfile and erase peers' entries

Cancel, reject and supersede of a parent run should settle its suspended nested child runs