#3679 Preserve active worker execution across serve restarts and upgrades; reconnect and reattach dispatches

Problem

Operators should be able to replace or upgrade swamp serve while remote workers are doing long-running work. Workers should keep their existing execution alive, reconnect when serve returns, and deliver results into the original workflow run. Today we must wait for full release/UAT workflows to finish before upgrading ops, even though the work is running on separate GCE/GKE workers.

This is a core swamp runtime feature, not an extension change.

Current behavior and evidence

Inspected deployed source at 148237ef: src/worker/connect.ts closes its RPC channel on control-socket closure; src/domain/remote/rpc_channel.ts::close rejects pending calls and aborts in-flight inbound handlers. Worker enrollment reconnects, but execution does not survive.

design/enablers/remote-execution.md, Failure, reconnection, and retry (lines 1306–1368 in the inspected checkout), explicitly describes abort plus redispatch for no-write work, failure for write-bearing work, and defers mid-step recovery pending write idempotency.

Operational case: ops runs native build workers, a GKE general fleet, and 36-shard ephemeral UAT fleets. A serve binary upgrade should not require abandoning or repeating an otherwise healthy test/build process. This request comes from code/design inspection during upgrade preparation, not a destructive restart experiment.

Requested behavior

• Decouple worker execution lifetime from the control connection. On a transient disconnect or planned serve restart, keep the existing worker subprocess and dispatch alive rather than aborting or restarting it.
• Persist enough orchestrator state to recover the original workflow/nested-workflow graph, dispatch identity, ownership and completion state after process replacement. Re-enrollment must reconcile existing running/completed dispatches before scheduling replacements.
• Reattach the same execution to the restored orchestrator. Prevent duplicate dispatch or duplicate completion through durable ownership/fencing and acknowledgment semantics.
• Handle unavailable data-plane requests explicitly: bounded buffering or backpressure, authenticated credential refresh, and idempotent write/result delivery with reconciliation for lost acknowledgments. A write-bearing step must not simply be blindly replayed.
• Preserve run IDs, logs, artifacts and trace correlation through reconnect, and expose reconnecting/reattaching state in the dashboard.
• Define the supported outage window and serve/worker protocol-version compatibility for rolling upgrades; unsupported transitions should have actionable diagnostics.
• Keep explicit cancellation, credential revocation, worker process death and lease expiry distinct from a transient serve disconnect. Do not promise transparent survival if the worker VM/process itself is replaced.

Acceptance coverage

Start long-running worker work, including subprocess execution, writes and a nested workflow; restart/upgrade serve while it is active; verify the worker subprocess was not restarted, the original run completes, and results/data versions are not duplicated. Cover completion while serve is down, an acknowledged-write response lost in transit, repeated disconnects, cancellation/revocation, and bounded outage expiry. Include supported mixed-version upgrade coverage.

Existing work / alternatives

Related #2153 provides durable step-boundary recovery, not preservation of an already-running worker execution. Related #2484 provides shutdown draining, which waits for work rather than allowing serve replacement while it continues. #1514 covers client reattachment. These do not satisfy the worker-execution requirement above.

The current workaround is to drain/wait for a quiet window before restarting serve, or recover/retry interrupted work afterward. That blocks timely runtime upgrades during continuous workloads. Searched Lab for reconnect, restart and in-flight feature requests and reattach; no matching open feature was found.