Lab #3157: A cancelled shell step can fail with a lock timeout instead of reporting as cancelled

Follow-up to swamp-club#3111 (PR 2910), from the adversarial review of that change. Traced by reading the code; not reproduced.

What happens

Since swamp-club#3111 a shell step ends its lock hand-off after its command returns and before it writes: it re-keys the locks held for the run and waits for a structural command (for example swamp data gc) that is still working under the old nonce, up to SWAMP_LOCK_TIMEOUT_MS.

When the step is cancelled, executeProcess throws, the catch in src/domain/models/command/shell/shell_model.ts turns that into exit code -1, and the method still awaits lockHandOff.end(). If a structural command the step started is still working, that wait can run to the lock timeout and throw LockTimeoutError. The step then fails with a lock timeout instead of ending as cancelled, and the cancel takes up to the lock timeout (60 seconds by default) longer.

Nothing is written in this case, so it is safe. It is slower and mislabelled.

Why it is inconsistent

The dispatch path already has this exception. In DispatchService dispatchOnce (src/serve/dispatch_service.ts) an attempt that failed on a cancelled step logs the reclaim failure and rethrows the original AbortError, so the step stays cancelled. The local shell path has no such rule, so the two disagree.

Possible directions

Mirror the dispatch rule in the shell model: when context.signal is aborted and the hand-off end rejects, log it and keep the cancellation outcome. The shell model must then write nothing, since the locks were not taken back.
Pass the abort signal into the reclaim wait so a cancel ends the wait early in both paths. A cancelled run writes nothing to the model, so it has no need to wait out the structural command.
At least document the difference in design/enablers/datastores.md, Hand-offs and re-keying.

A reclaim failure after a lost worker is logged with the cancelled-step wording.
A drain whose skipped set never settles ends in a LockTimeoutError that says it waited on held locks when none were held.

flaky: LitestreamReplicator 'a failed launch releases its lock probe, so the next acquire starts a sidecar' fails by timing in verify-build

typed-secrets 4.4: persistence and resume hold references: rehydration stops reading the vault, every persisted copy uses the persisted form, and run events are masked

data_writer_property_test: a secret holding a newline and a literal backslash-n escapes the persisted-content property (seed 1648929202)

typed-secrets: the accessor adapter reads a legacy row stamped with the empty namespace as foreign, so its tagged fields stay reference text on the typed path

typed-secrets query view: route a row's JSON content by whether the adapter adapted the record, not by the service's namespace rule

flaky: for_each_name_property_test lets the text around a sentinel equal the secret's value, so the untyped value search replaces both and the property fails by seed

typed-secrets 4.3: the expression-facing query view registers Secret in DataQueryService

CEL duration arithmetic is off by a second when a duration has negative nanos (cel-js 7.6.1)

typed-secrets 4.2: live and reconstructed step outputs and child outputs return references through the accessor adapter

A coded UserError a model method throws reaches a workflow step without its code, because InProcessExecutor keeps only the message

Dashboard Workflows page takes ~19s on a serve with many extension workflows: workflow.search resolves each workflow's owner with a full extension-workflow scan

codegen: BROKEN_REGEXES override for the @swamp/aws/invoicing InvoiceUnit name lacks the u flag, so it rejects ordinary names

dashboard: run page polling stacks history.get and history.logs requests while serve is slow

serve: add child spans and attributes to workflow.history.get (and run_summary/run_search) to diagnose slow dashboard loads

serve: webhook-triggered workflow runs can't be cancelled — cancel can't find them, so it reports serve's own pid as "the process running it is still alive"

A workflow step fails on a static validation error in a method it does not invoke, where swamp behavior run of the same method succeeds

serve: a run orphaned by a container restart can't be cancelled — the restarted serve reuses the dead owner's pid, so cancel reports the owner still alive

typed-secrets 4.6: turn the typed path on for every entry point, remove the internal option, and delete the regular-expression rewrite, the value search and the plaintext resolution of tagged fields

typed-secrets 4.5: the inline resolutions refuse or mask: placement, task selectors, child references and deferred targets refuse secrets, assert messages mask them, and the deprecated resolvers go

typed-secrets 4.1: workflow passes bind sentinels as Secrets: typed guards and asserts, forEach items and identities, child bindings, webhook inputs, and the synchronous authorization step

typed-secrets 3.5: local entry points, validation diagnostics, context.runModel and model.method delivery

model.method in a workflow guard fails with 'Do not know how to serialize a BigInt' when its inputs hold an int

data query: filter on data lifecycle so deleted resources can be excluded

CEL expression results that hold a timestamp, duration or uint become an empty object

Catalog rebuild restores pre-rename versions of a name renamed away and written again

Remote resolveSecret allowlist is always empty for workflow steps that use vault.get

swamp issue redacts git commit SHAs as secrets and fast-check replay paths as IP addresses

typed-secrets 3.4: destination refusals, identifier stand-ins, shell safety and masking

Remote readResource does not apply the legacy schema fallback for untagged sensitive records

Flaky: datastore_v3_outbox_model_property_test throws MayLandLaterError on a random seed (aged commit, failed discard-and-mark checkpoint, power cut)

v3 outbox model property test fails: a commit dropped after a failed append-discard-and-mark reports MayLandLaterError instead of the expected outcome

Workflow step dataOutputOverrides vaultName is accepted but silently dropped

typed-secrets 3.3: sensitive-field accessors, data-origin sentinels and reference validation

Generated vault keys carry raw data-name and vary characters that strict vault providers reject

typed-secrets 3.2: vault.get as a CEL function, bare-token rewriting, AST extraction and up-front existence checks

typed-secrets 3.1: the production Secret, its refusals, the fault context, the container rule and the boundary conversion, behind the internal option

Flaky test: extension_pull_dependency_conflict_test fails with 'failed to copy esbuild binary' during deno bundle

Extension tarball analyzer ignores webhook entry points

Registry drops contentMetadata.extensions (method grafts) at confirm-push

digitalocean codegen leaves an orphaned security_secret.ts model file after its endpoint left the spec

A --repo-dir or SWAMP_REPO_DIR through a symlink fails with 'unable to open database file'

@swamp/1password: vault delete of a missing item probably fails instead of being a no-op

@swamp/1password: item or vault names containing 'authorization' are misreported as auth failures

typed-secrets 2.2: VaultSecretBag entries hold references, one registry per run, 128-bit sentinels

Show wait keys and pending signals to remote clients and the dashboard

Flaky: LitestreamReplicator.acquire 'concurrent acquires in one process share one sidecar and one hold' fails 'no hold leaked' under full-suite load

VaultService.requireExistenceProvider labels the vault instance name as its type

VaultService capability errors label the vault instance name as its type

typed-secrets 2.1: VaultExistenceProvider, its conformance suite, and has on the built-in providers

MockVaultProvider.get and delete throw synchronously instead of rejecting

typed-secrets 1.2: pin the Secret Proxy, refusal set and container rule on cel-js 7.6.1

issue-lifecycle: pr_merged and ship report success and advance the phase when swamp-club is unreachable, writing nothing upstream

serve check-config crashes with a TypeError when a list-valued auth key is a string

access grant list --server shows user subjects as user:user:<name>

Flaky: process_bundle_prober_test 'stopping a host that has already exited lists no children under its pid' fails under full-suite load

typed-secrets 0.4: characterize worker secret authorization and control-plane secret traffic

issue-lifecycle: an approved plan cannot be sent back to planning

typed-secrets 1.2: minimal opaque Secret, CelVaultNamespace receiver, policy inventory and evaluation frame

typed-secrets 1.1: commit the cel-js 7.6.1 probe harness as pinned behavior tests

typed-secrets 0.1: land the typed-secrets plan as a design enabler doc

swamp update: a slicker, colorful download progress bar that ends with a newline

gcs-datastore: a failed lock read is reported as "unlocked" — #2298 in the GCS backend

A manual worker verify racing an enrollment probe can leave a healthy worker unverified until it reconnects

bundle freshness: a one-off deno info failure leaves a source on the in-directory fallback for good

serve shutdown waits for an unsignalled run index rebuild: the continuation sweep's boot pass and request-led rebuilds cannot be stopped by SIGTERM

serve: handlers that sanitize an error for the client never log the original, so path-bearing failures are undiagnosable

literal() output in a step's inputs logs repeated Refusing to evaluate expression warnings

Read non-string sensitive resource fields back with their type

serve and swamp dashboard shutdown waits seconds for the run index warm-up: the plain index parse is not abortable, and the warm-up waits on request-led loads it joined (regression from #3105)

Run index snapshot cache keeps serving a stale run status: ctime is only second-resolution so in-place same-size rewrites alias the stamp, and snapshots are cached even when repair could not write the index (regression from #3105)

Flaky: serve_v3_extension_lockfile_test 'serve extension.pull commits the new entry' fails under full-suite load with extension_pull_failed

Run index repair-by-stat misses an in-place, same-size record rewrite that restores mtime within the same second as its entry: ctime is reported in whole seconds

Dashboard: an idle, visible tab keeps counting runs that have aged out of a preset time range

Dashboard: Models and a model's page re-send model.output.search every minute on a preset time range

Concurrent run writers leave finished runs stuck as running in the run index: repairIndex returns early once counts agree, so lost updates are never healed (regression from #3105)

Run index updates race across processes: concurrent writer processes can lose each other's index entry updates

Dashboard: a live run page polls a full workflow.run.search every 2.5 s, and time-range presets refetch the run summary every minute

Dashboard Workflows, workflow and run pages are slow with large history: workflow.run.summary and workflow.run.search read every run index of every workflow on each request

README, design docs and agent-constraints examples still teach swamp model commands

Decide and add behavior spellings for --model flags, enum values and SWAMP_MODELS_DIR outside the behavior tree