A nested structural swamp still waits on its run's lock when the lock holder is not a same-host ancestor (cross-host worker, --server into a non-ancestor serve)

h13h ago

#3111

A nested structural swamp that outlives the run that started it keeps skipping that run's lock while the holder writes

14h ago

#3110

Deliver workflow signals through a write-once outcome record instead of writing the run record

h14h ago

#2982

Forward held-lock identity over --server so a loopback nested swamp does not wait on its caller's lock

h16h ago

#2983

A nested structural swamp on a same-host remote worker waits on its own step's lock held by swamp serve

h17h ago

#2981

Fail fast when nested structural commands under parallel runs wait on each other's locks

h17h ago

#3079

Docs: remove followUpActions from the MethodResult reference in the extension model manual

h17h ago

#3092

signal_test: already-settled test fails by chance when a random UUID spells the sender name

17h ago

#3068

Add a wait_for_signal workflow step that pauses a run for a JSON message

h17h ago

#3094

Deliver workflow signals through swamp serve

h12h ago

#3059

run doctor through serve interrupts a live run of another serve instance when no heartbeats are recorded

h18h ago

#3081

Flaky test: WalSink replayed segments are delivered before events written after replay leaves one WAL segment

h19h ago

#3078

Remove followUpActions: no model or extension can produce them

h19h ago

#2946

Workflow schedule watcher and workflow edit symlink lookup ignore the managed config workflows dir

h19h ago

#3084

main is red: repository_dirty_coverage_test fails for UnifiedData.collectGarbage(orphaned deferred write)

19h ago

#3082

isProcessGone always reports alive on Linux under the test task permissions, failing 9 tests on main

h19h ago

#3083

Flaky test: WalSink replayed segments are delivered before events written after replay

17h ago

#2893

serve: relative tls cert-file/key-file and --config resolve against the working directory, not the repository

h20h ago

#2975

findBySpec/findByTag lose a workflow step's output after a version is deleted, pruned or rolled back

s20h ago

#3074

serve HA: worker enrollment on a replica that lacks the enrollment token's definition may create a second definition (unverified)

20h ago

#3073

serve HA: pull a token's definition on an auth miss so a peer accepts a new token at once

20h ago

#2917

Run tracker keeps some interrupted workflow rows forever: retention waits for markSettled, which several paths never call

21h ago

#3051

run doctor reads every run record on each run to rebuild the run indexes

hs21h ago

#3037

Serve audit: two sinks with the same name share one emitter cursor, so the second never receives events

1d ago

#3039

Serve audit: a durable sink write that times out is retried while the first call is still pending, so the store can hold duplicate sequences

s1d ago

#3043

Flaky test: ModelResolver data accessors for a renamed model return the earlier id's record from findBySpec

s1d ago

#2985

data query --limit returns fewer live records than exist and reports limited: false when stale catalog rows fall inside the limit

s1d ago

#2987

data get: help example passes --run latest, which is not a recognised run id

s1d ago

#2905

workflow reject leaves sibling gates waiting_approval and dependents pending in a failed, retryable run

h1d ago

#3045

Show expired approval gates as expired, with a Cancel action in the dashboard

1d ago

#3041

digitalocean codegen leaves an orphaned security_secret.ts model file after its endpoint left the spec

1d ago

#2920

workflow cancel cannot cancel a suspended run whose workflow file was deleted, and cancel --all silently skips it

h1d ago

#2898

A job alone in its level starts after the abort, and its unstarted step is recorded as a real failure instead of settledByAbort

h1d ago

#2919

workflow approve racing workflow cancel loses the cancel: the run returns to suspended after cancel reported cancelled

h1d ago

#2972

data query: does not follow data rename forwarding that data get follows

h1d ago

#2994

Unit tests share fixed /tmp catalog paths, so concurrent runs corrupt them and fail every later run

h1d ago

#2968

data query: renamed data items are not found by their old name

h4d ago

#2960

data query: model lookup by id and orphan recovery parity with data get

h4d ago

#2899

workflow approve on a run that is not suspended prints a fatal error with a stack trace

h4d ago

#2957

data query: no way to target a workflow's latest run

h4d ago

#2959

data query: binary content lacks contentEncoding/base64 parity with data get

h4d ago

#2958

data query: no fallback from spec name to data instance name

h4d ago

#2967

data query: no definitionHash or garbageCollection fields in query results

h4d ago

#2955

A nested structural swamp spawned by swamp serve skips every lock serve holds, including other runs' locks

h4d ago

#2962

data query: confirm catalog parity with data get for custom datastores

h4d ago

#2961

data query: single-result --json mode for scripts migrating from data get

h4d ago

#2973

Docs: document data query --single and fix the stale --limit default in reference/data.md

1d ago

#2948

data get --workflow silently returns an arbitrary step's data when several steps share a data name

h4d ago

#2942

swamp doctor workflows misses nested and .yml extension workflows that the loader reads

h4d ago

#2928

auth whoami says the stored API key is no longer valid when auth.json only holds an env-key identity cache

s4d ago

#2921

Docs: swamp skill says cancelling a parent run cancels its nested child runs; the child is left suspended

s4d ago

#2941

Namespace advice in the slow-lock warning is wrong for a single repo on a local datastore

h4d ago

#2930

Method-run records left running by a dead owner are never reaped

s5d ago

#2909

Flaky: usecase_sync_characterization_data_test 'data gc (serve)' asserts markDirty order that varies under parallel load

s5d ago

#2926

Worker dispatch rebuilds the definition with Definition.create, so a legacy-named model cannot run on a worker

h5d ago

#2918

A cancelled workflow exits before its steps stop, leaving their method-run records at running

h5d ago

#2927

Nested swamp structural command in a shell step times out on its parent's per-model lock (SWAMP_LOCK_HOLDER_PID is stripped)

5d ago

#2924

Auth gate blocks nested swamp in workflow shell steps when the credential is in SWAMP_API_KEY

h5d ago

#2925

swamp workflow create crashes with an [FTL] ZodError stack trace on an invalid (uppercase) workflow name

5d ago

#2923

swamp model create crashes with an [FTL] ZodError stack trace on an invalid (uppercase) model name

h5d ago

#2914

model cancel SIGTERMs a running swamp serve process when the method run belongs to a serve-run workflow

h5d ago

#2929

Follow-up action retries ignore the run's abort signal and can outlast a cancel

h20h ago

#2915

model cancel --all on a running workflow leaves its method-run records at running, or records them failed instead of cancelled

5d ago

#2910

model cancel SIGKILLs the owner 2 s after SIGTERM, before the step executor's own 3 s kill grace, so the run never records its own cancellation

h5d ago

#2895

workflow cancel and supersede of a suspended run leave its jobs running in the cancelled record

h5d ago

#2922

Tell timeouts apart from cancels in method-run records, and review the hidden 30 s fallback timer for step-called model methods

5d ago

#2896

A force-exited workflow run stays running with a dead pid: resume, recover and run doctor --fix cannot clear it

h5d ago

#2736

A nested workflow that suspends on manual approval forwards its suspended event into the parent run's stream

h5d ago

#2897

workflow cancel SIGKILLs the run 2 s after SIGTERM, cutting off always/completed cleanup jobs mid-step

h5d ago

#2845

Warn when datastore setup keeps managedConfig but the new datastore has no config tier

h5d ago

#2903

Docs: run doctor --fix and workflow recover settle runs left running by a force-exited process

5d ago

#2812

serve: a relative grants-dir resolves against the working directory, not the repository

h5d ago

#2597

workflow resume cleanup mode can run an always-gated teardown before a suspended job's approved work

h5d ago

#2894

Docs: serve-flags.md should state that relative grants-dir and grants-file resolve against the repository

5d ago

#2838

managedConfig: extension writes update the shared lockfile from a stale cache and overwrite peers' entries

h5d ago

#2868

managedConfig: model/vault create and other bulk-push commands re-upload a stale extension lockfile and erase peers' entries

h5d ago

#2867

Cancel, reject and supersede of a parent run should settle its suspended nested child runs

6d ago

#2837

datastore setup extension overwrites the shared config tier with the repo-local copy and drops managedConfig from .swamp.yaml

h6d ago

#2844

datastore setup can overwrite an existing remote config tier when it moves an in-repo tier into an extension datastore

6d ago

#2829

serve daemon enable writes a unit that crash-loops when serve args are invalid (e.g. missing --admins)

h6d ago

#2833

Concurrent embedded deno extraction races: stale temp-file cleanup deletes another process's in-flight .deno.tmp file (chmod ENOENT)

h6d ago

#2836

Docs: manual approval gates inside nested workflows (swamp-club#2736)

5d ago

#2802

Telemetry bridge drops a parent step's invocation when a nested workflow step has the same job and step names

h6d ago

#2824

serve daemon: unit sets SWAMP_HOME, which relocates the config dir — token/oauth daemons can't find auth.json and crash-loop

h6d ago

#2825

Workflow step that times out on the model lock leaves its output and run-tracker row stuck at 'running'

h6d ago

#2827

Docs: serve daemon enable pins SWAMP_CONFIG_DIR; document the SWAMP_CONFIG_DIR override

6d ago

#2724

An extension upgrade that hits a type collision is rolled back to no version: roll back to the prior version with a commit/rollback install handle

h6d ago

#2735

Telemetry bridge attributes a parent run's later method invocations to its nested child run

h6d ago

#2814

sensitive_data_delivery_test fails where /bin/sh is bash: the final ps is exec'd and reports its own argv

h6d ago

#2815

build-attestation rejects every run: evaluated workflows now carry sensitiveFormat, which the provenance check treats as an uncommitted field

h6d ago

#2762

DuplicateTypeError names the incoming extension as the existing claimant and reports a false ghost row when it sorts first

h6d ago

#2723

Install extensions by stage and swap, with a journal and crash recovery

h7d ago

#2593

workflow resume ignores SWAMP_MAX_CONCURRENT_STEPS for job-level concurrency

h7d ago

#2651

Tidy up tests and helpers from the serve suspended-run cancel

h7d ago

#2709

Concurrent extension installs and removals can interleave in one checkout: add a pulled-extensions lock and LockfileRepository.refresh()

h7d ago

#2722

deno run audit prints No description available for every advisory

h7d ago

#2492

Add a literal() CEL function so one string can mix swamp expressions with another service template syntax

h7d ago

#2707

Extension archives have no size cap: cap the compressed and decompressed size on pull and push

h7d ago

#2616

withGeneratorSpan does not forward return() to the wrapped generator, so its finally blocks are skipped on early exit

h7d ago

#2713

forEach over a direct step with a shared modelName runs every iteration with one iteration's globalArgs on first creation

h7d ago

#2708

Split installExtension into an unlocked prepare and a repo-changing apply

h7d ago

#2650

workflow cancel --server shows raw JSON errors and drops --reason

h7d ago

#2702

Two pulled extensions providing one type + a rebuilt catalog brick every command (I-Repo-1 at startup reconcile, even rm/--repair)

h7d ago

#2490

Extension catalog keeps stale rows after install migration and rm; model type describe crashes with ENOENT

h7d ago

#2698

Flaky test: CollectiveRefreshService fallback getAccessToken test counts calls after a fixed 200ms sleep

sh7d ago

#2701

Docs: worker-fleet guides start swamp serve without --admins and mint tokens with a removed command

7d ago

#2493

model validate log output misaligns multi-entry Expression paths errors

h7d ago

#2447

Retry refusals print unquoted forEach template names in --from hints, which bash rejects

h7d ago

#2580

workflow run log output: a forEach step with a templated name labels its iterations with the raw expression, e.g. deploy-${{ self.env }}[0]

h7d ago

#2694

Docs: update the model validate Expression paths example in cel-expressions.md

7d ago

#2652

Docker image runs swamp as PID 1 without an init, so orphaned step processes are never reaped

h7d ago

#2617

swamp.cli.bootstrap, configure_extension_loaders and teardown spans each start their own trace

h7d ago

#2691

Docs: run swamp under an init in the worker-fleet guides

h7d ago

#2646

Extension restore leaves a parent installed and its dependency permanently missing when the dependency install fails

h7d ago

#2426

serve: vault edit over --server opens an editor on the server host and ignores the managedConfig vaults dir

s7d ago

#2514

A suspended run started by swamp serve cannot be cancelled

h8d ago

#2634

command/shell: aborting a step kills only sh, leaving the command's child processes orphaned

h8d ago

#2639

Extension restore drops the lockfile entry's channel and leaks the parent's expectedChecksum into dependency installs

h8d ago

#2429

managedConfig: serve extension handlers disagree on the pulled-extensions root, and lockfile files[] are repo-relative

h8d ago

#2589

swamp.model.method spans are parented to the workflow run span instead of their swamp.workflow.step span

h8d ago

#2314

Platform certificate load failure surfaces as an uncaught error on network commands

h8d ago

#2417

serve: server-login can't be traced: no request spans, and gcs-datastore push is unparented with an 81s gap in child spans

h8d ago

#2420

workflow resume keeps the original run's process identity (run record pid and instanceId)

h8d ago

#2612

managedConfig: store pulled extension archives in the config tier and converge each checkout to the shared lockfile

h7d ago

#2547

workflow run --json: guarded-skip lines land on stdout or stderr depending on timing

h8d ago

#2360

workflow run and model method run reject --ca-cert, but their TLS error tells users to pass it

h8d ago

#2549

After a cancel or --timeout, jobs that share a level are abandoned mid-flight: their steps stay running and their own cleanup steps never run

h8d ago

#2359

SWAMP_CA_CERT overrides the --ca-cert flag, contrary to the documented flag-wins precedence

h8d ago

#2594

workflow resume --timeout records cancel_reason 'The signal has been aborted' instead of a timeout reason

h8d ago

← Back to list

01Issue

BugShippedSwamp CLIPublic

Assigneesstack72 hammz

Relationships

#2698 Flaky test: CollectiveRefreshService fallback getAccessToken test counts calls after a fixed 200ms sleep

Opened by hammz · 9/29/2026· Shipped 9/29/2026

src/serve/collective_refresh_service_test.ts:272 (works with fallback getAccessToken) starts the service, sleeps a fixed 200ms with setTimeout, disposes, then asserts accessTokenCalls equals 1. Under the full parallel test run in verify-build it observed 2 (run 896b24e4-8d6d-4e9d-a1e4-d13c0da561d4, commit f591d840, branch issue-2493, which does not touch src/serve). It passes 5 of 5 in isolation. AGENTS.md says never wait with a fixed sleep for async work and to poll with waitFor from swamp-testing; the test should wait for the first refresh to complete (or assert at least one call and the updated collectives) rather than depending on how many refresh cycles fit in 200ms.

02Bog Flow

Shipped

9/29/2026, 4:39:55 PM

Click a lifecycle step above to view its details.

03Sludge Pulse

stack72 assigned stack729/29/2026, 4:06:13 PM

hammz assigned hammz9/29/2026, 4:14:39 PM

A nested structural swamp still waits on its run's lock when the lock holder is not a same-host ancestor (cross-host worker, --server into a non-ancestor serve)

A nested structural swamp that outlives the run that started it keeps skipping that run's lock while the holder writes

Show signal waits to remote clients and the dashboard

Settle expired signal waits from swamp serve

Resume signalled workflow runs automatically under swamp serve

Deliver workflow signals through a write-once outcome record instead of writing the run record

Forward held-lock identity over --server so a loopback nested swamp does not wait on its caller's lock

A nested structural swamp on a same-host remote worker waits on its own step's lock held by swamp serve

Fail fast when nested structural commands under parallel runs wait on each other's locks

Docs: remove followUpActions from the MethodResult reference in the extension model manual

signal_test: already-settled test fails by chance when a random UUID spells the sender name

Add a wait_for_signal workflow step that pauses a run for a JSON message

Deliver workflow signals through swamp serve

run doctor through serve interrupts a live run of another serve instance when no heartbeats are recorded

Flaky test: WalSink replayed segments are delivered before events written after replay leaves one WAL segment

Remove followUpActions: no model or extension can produce them

Workflow schedule watcher and workflow edit symlink lookup ignore the managed config workflows dir

main is red: repository_dirty_coverage_test fails for UnifiedData.collectGarbage(orphaned deferred write)

isProcessGone always reports alive on Linux under the test task permissions, failing 9 tests on main

Flaky test: WalSink replayed segments are delivered before events written after replay

serve: relative tls cert-file/key-file and --config resolve against the working directory, not the repository

findBySpec/findByTag lose a workflow step's output after a version is deleted, pruned or rolled back

serve HA: worker enrollment on a replica that lacks the enrollment token's definition may create a second definition (unverified)

serve HA: pull a token's definition on an auth miss so a peer accepts a new token at once

Run tracker keeps some interrupted workflow rows forever: retention waits for markSettled, which several paths never call

run doctor reads every run record on each run to rebuild the run indexes

Serve audit: two sinks with the same name share one emitter cursor, so the second never receives events

Serve audit: a durable sink write that times out is retried while the first call is still pending, so the store can hold duplicate sequences

Flaky test: ModelResolver data accessors for a renamed model return the earlier id's record from findBySpec

data query --limit returns fewer live records than exist and reports limited: false when stale catalog rows fall inside the limit

data get: help example passes --run latest, which is not a recognised run id

workflow reject leaves sibling gates waiting_approval and dependents pending in a failed, retryable run

Show expired approval gates as expired, with a Cancel action in the dashboard

digitalocean codegen leaves an orphaned security_secret.ts model file after its endpoint left the spec

workflow cancel cannot cancel a suspended run whose workflow file was deleted, and cancel --all silently skips it

A job alone in its level starts after the abort, and its unstarted step is recorded as a real failure instead of settledByAbort

workflow approve racing workflow cancel loses the cancel: the run returns to suspended after cancel reported cancelled

data query: does not follow data rename forwarding that data get follows

Unit tests share fixed /tmp catalog paths, so concurrent runs corrupt them and fail every later run

data query: renamed data items are not found by their old name

data query: model lookup by id and orphan recovery parity with data get

workflow approve on a run that is not suspended prints a fatal error with a stack trace

data query: no way to target a workflow's latest run

data query: binary content lacks contentEncoding/base64 parity with data get

data query: no fallback from spec name to data instance name

data query: no definitionHash or garbageCollection fields in query results

A nested structural swamp spawned by swamp serve skips every lock serve holds, including other runs' locks

data query: confirm catalog parity with data get for custom datastores

data query: single-result --json mode for scripts migrating from data get

Docs: document data query --single and fix the stale --limit default in reference/data.md

data get --workflow silently returns an arbitrary step's data when several steps share a data name

swamp doctor workflows misses nested and .yml extension workflows that the loader reads

auth whoami says the stored API key is no longer valid when auth.json only holds an env-key identity cache

Docs: swamp skill says cancelling a parent run cancels its nested child runs; the child is left suspended

Namespace advice in the slow-lock warning is wrong for a single repo on a local datastore

Method-run records left running by a dead owner are never reaped

Flaky: usecase_sync_characterization_data_test 'data gc (serve)' asserts markDirty order that varies under parallel load

Worker dispatch rebuilds the definition with Definition.create, so a legacy-named model cannot run on a worker

A cancelled workflow exits before its steps stop, leaving their method-run records at running

Nested swamp structural command in a shell step times out on its parent's per-model lock (SWAMP_LOCK_HOLDER_PID is stripped)

Auth gate blocks nested swamp in workflow shell steps when the credential is in SWAMP_API_KEY

swamp workflow create crashes with an [FTL] ZodError stack trace on an invalid (uppercase) workflow name

swamp model create crashes with an [FTL] ZodError stack trace on an invalid (uppercase) model name

model cancel SIGTERMs a running swamp serve process when the method run belongs to a serve-run workflow

Follow-up action retries ignore the run's abort signal and can outlast a cancel

model cancel --all on a running workflow leaves its method-run records at running, or records them failed instead of cancelled

model cancel SIGKILLs the owner 2 s after SIGTERM, before the step executor's own 3 s kill grace, so the run never records its own cancellation

workflow cancel and supersede of a suspended run leave its jobs running in the cancelled record

Tell timeouts apart from cancels in method-run records, and review the hidden 30 s fallback timer for step-called model methods

A force-exited workflow run stays running with a dead pid: resume, recover and run doctor --fix cannot clear it

A nested workflow that suspends on manual approval forwards its suspended event into the parent run's stream

workflow cancel SIGKILLs the run 2 s after SIGTERM, cutting off always/completed cleanup jobs mid-step

Warn when datastore setup keeps managedConfig but the new datastore has no config tier

Docs: run doctor --fix and workflow recover settle runs left running by a force-exited process

serve: a relative grants-dir resolves against the working directory, not the repository

workflow resume cleanup mode can run an always-gated teardown before a suspended job's approved work

Docs: serve-flags.md should state that relative grants-dir and grants-file resolve against the repository

managedConfig: extension writes update the shared lockfile from a stale cache and overwrite peers' entries

managedConfig: model/vault create and other bulk-push commands re-upload a stale extension lockfile and erase peers' entries

Cancel, reject and supersede of a parent run should settle its suspended nested child runs