swamp serve is a single point of failure. Every piece of in-flight state — queued webhook runs, queued cron fires, active workflow executions, and worker dispatch bookkeeping — lives in memory. A process crash silently loses accepted work that callers already received a 200 queued for. On restart, a boot sweep tombstones interrupted workflows rather than recovering them. Running a second instance is actively destructive: the new instance's startup sweep destroys the first instance's live state.

This makes swamp serve unsuitable for any deployment where uptime matters — Kubernetes pods get replaced, ASG instances die, processes OOM, and hosts reboot. Today, every one of those events loses work with no recourse.

Desired Outcome

swamp serve can be deployed as one or more instances behind a load balancer with no accepted work lost on process death, no unsafe overlap during instance replacement, and no new infrastructure dependencies.

Specifically:

No accepted work is ever memory-only. Every webhook acknowledgement and cron fire is backed by a durable record before the caller is told it succeeded.
Process death is recoverable. Interrupted work is re-enqueued or resumed at startup according to a per-workflow policy, not silently discarded.
Instance replacement is safe. A new instance starting while an old one is still running cannot corrupt or destroy the old instance's in-flight state.
Multiple instances can share work. N instances can run against the same datastore, dividing work by per-item claims, with distributed deduplication for cron.
No new services. All coordination primitives (leases, epochs, claims, fire records) are records in the datastore the repo already has. The deployment unit remains a single swamp binary.

The full design brief and phase breakdown live in the artifact linked from this session. The implementation will be tracked as child issues of this one.

02Bog Flow

Open

7/29/2026, 3:58:22 AM

No activity in this phase yet.

03Sludge Pulse

stack72 commented 7/28/2026, 11:23:32 PM

First PR - https://github.com/swamp-club/swamp/pull/2005

swamp serve ties a run's life to the WebSocket that started it. Socket dies, run dies. That makes HA impossible — there's nothing to reconnect to.

This PR adds --detach-runs. With it, runs execute independently of the client connection. A disconnected client can re-attach by run ID and pick up the event stream where it left off. Without the flag, behavior is identical to today.

this is the foundation for durable jobs, crash recovery and multi-instance so clients can reconnect

model validate: transport-reachable preflight sends unresolved vault CEL expression as credential

Add `amp` (Amp Code) as a built-in --tool for repo init/upgrade/audit

Add rollbackOnFailure for method-level write atomicity

Expose resource attributes in DataHandle for workflow report stepExecutions

Make swamp serve durable and highly available

Docs: document guard and resume --from in workflows reference

.swamp.yaml rewritten with quote style change + lastSkillMigrationWarning on read-only operations

Add --from flag to workflow resume for DAG re-entry on failed runs

data delete removes data locally but push reports 'no changes' — next datastore sync restores everything

Empty report returns persist 0-byte versions that mask the real report — and `swamp data gc` then deletes the real version, keeping the empties

Docs: document guard field for workflow steps

First-class scheduled garbage collection policy per datastore namespace

Add guard field to workflow steps for idempotent execution

--quiet mode does not suppress in-process extension logger output

tailscale transport hangs indefinitely on Tailscale SSH check-mode auth; never surfaces the login URL

Concurrent @type auto-creation races: N simultaneous runs create N definitions sharing one name, and only one is reachable

Worker queries need a safe connected-worker filter for fleet fan-out

Extension docs still ship the Driver extension type, which the binary refuses (rejectRemovedDriverFields) after #535

Docs: document vault selection for sensitive fields (defaultVault, per-resource vaultName)

azure-kv error wrapping loses original error type — ERROR_TYPE always 'Error' in traces

writeResource commits per-call: a method that fails midway leaves its earlier outputs persisted, with no marker that the run failed

Docs: document workflow-scope ReportContext fields in extension report reference

Cloudflare: expose GraphQL Analytics API (httpRequestsAdaptiveGroups / client-IP request analytics)

aws-sm deleteAnnotation removes all non-aws: tags, not just swamp-specific ones

No way to select which vault receives a sensitive-marked field's value

aws-sm: namespace annotation labels with swamp:label: prefix

workflow assert: a step whose CEL expression fails to evaluate is omitted from --junit output, so the XML reports a clean pass

Documentation for workflow assert steps and JUnit output

Expose workflow inputs in WorkflowReportContext

Teach swamp skill how to write assert workflow steps

Include a workflow's declared `inputs` in `workflow get --json` (and a `hasInputs` hint in `workflow search --json`)

Docs: add how-to guide for running swamp serve/worker in daemon mode

worker daemon enable: relative/missing --cache-dir fails launchd EX_CONFIG while daemon status reports running

swamp doctor extensions: nondeterministic BundleBuildFailed for local extensions on an unchanged tree

Docs: document datastore directory relocation during setup

vault extensions: emit OTel spans for vault operations (AWS SM, Azure KV, 1Password)

Issue redactor corrupts reports: dotted code identifiers masked as hostnames, loopback and RFC 5737 documentation IPs masked, angle-bracket placeholders half-eaten

data prune resolves auto-definitions at the local path, not the datastore path — deletes live models' data (verified: destroyed a working server token and an active grant)

workflow/job-lifetime data is never reclaimed — expiry check requires an ownerDefinition.workflowId that step execution never sets

runModel breadth limit (MAX_INVOCATION_BREADTH=100) is never enforced — tracking object is never written back to the caller context; verified 150/150 calls succeeded

serve daemon enable is ungated — it reports success, then the account gate fires inside the detached daemon, which crash-loops invisibly

Purely local operations require a swamp-club account: local-token serve is gated while deprecated --auth-mode none is not, third-party OIDC is gated, and 'Local_Encryption' is paywalled by capitalization

A collective token stored in auth.json (rather than SWAMP_API_KEY) is permanently scope-less — resolution only runs for the env var, and scopes are never persisted for collective tokens

An empty stored apiKey counts as authenticated — saveIdentityCache writes apiKey:'' on a virgin machine, permanently satisfying the mandatory-account gate with no credential

Account/scope gates fire before argument, type and repo validation — 'swamp vault create env my-vault' (the command's own example, an invalid type) returns a paywall instead of 'Unknown vault type'

auth token create advertises a 'collective:write' scope that does not exist, and following its advice hits 'collective tokens cannot create other collective tokens'

requireScope does exact string matching — fine-grained scopes fail gates that demand the literal wildcard, and minted scopes are never validated or normalized

Collective-token scope cache never expires, is never refreshed by whoami, and cannot be cleared via the CLI — scope changes and revocations never take effect

Collective-token scope cache is keyed on a 12-char token prefix — only 2 chars distinguish org tokens, so a rotated token silently inherits another token's scopes

Misleading 'Your collective token lacks the X scope' when scope resolution failed — startup whoami errors are silently swallowed

Community points / reputation: award points for upvoting extensions, feature requests, comments, and feed posts

globalArgs fields leak into method arguments when a method's arguments schema is a bare z.record()

swamp serve --auth-mode oauth: crash during first-run admin resolution discards the OAuth token, forcing full device-flow re-registration

Docs: document extension runtime permissions and device I/O workaround in user manual

datastore setup extension drops 'namespace' from --config, making extensions that require it impossible to configure

Kiro CLI v3 permissions: no capability covers disclose_context (skill loading)

CLI: swamp invite <email> to drive the platform-invite API

api-key-scoping explanation doc contradicts shipped fine-grained scopes

Fine-grained scopes for swamp.club access tokens

End-to-end UAT for fine-grained collective token scopes (#960)

Docs: document swamp auth token create command for collective tokens

open method fails for password-auth hosts: SSHPASS env set but ssh call never wrapped in sshpass

docs: update swamp skills for setup extension --namespace

apps sync/lookup fail with 'partial() cannot be used on object schemas containing refinements'

Docs: update autoGc documentation to reflect write-time version pruning

model search/list doesn't enumerate auto-created definitions under .swamp/auto-definitions/

CLI command to create scoped collective tokens (phase 8 of #960)

docs: document auth gates for team features (datastores, vaults, serve)

`datastore setup filesystem` never transfers content when leaving a sync-based (remote) datastore — copies only the catalog and hardcodes `filesPulled: 0`

Workflow/CLI output uses red text for non-error/informational lines, contradicting standard error-color convention

Support end-user timezones on https://swamp-club.com/u/shelson/activity

Document doctor datastores --repair for namespace contamination cleanup

version-drift check (from #236) is a raw string-inequality, false-positives on multi-model manifests and metadata-only bumps

Pre-flight checks receive unresolved vault.get() expression text, not the resolved secret

Invite people who have never used swamp to a collective (email invite flow)

docs: update giga-swamp manual pages for setup extension --namespace

Docs: update extension push output examples to include channel and visibility

Collective management API for extension-based automation

Remote --server wss:// fails through an HTTP/2 TLS reverse proxy (WS client ALPN is h2-only)

uat: add namespace isolation and shard resilience test coverage