Lab #2437: control-plane vault init failures are swallowed; `access token reveal` says "Vault '

Summary

When the control-plane vault can't initialise (for example, because of bad datastore credentials), the real error is logged at WARN and discarded. swamp access token reveal then fails with "Vault '_token-secrets' not found. No vaults are configured", plus advice to swamp vault create <type> _token-secrets. That hides the actual problem and points to the wrong fix.

Environment

swamp built from main @ 1da85ec8 (20260923.175528.0-sha.1da85ec8), unmodified
@swamp/s3-datastore@2026.09.23.1 (MinIO), managedConfig: true
Also seen with CLI 20260917.193606.0 / 20260923.001003.0 against @swamp/gcs-datastore@2026.09.23.1, where the cause was expired GCP application-default credentials

Steps to reproduce

A repo on a remote datastore with managed config.
swamp access token mint probe-token --principal user:probe --duration 1d. The secret is stored in the remote _token-secrets.
Control: swamp access token reveal probe-token --yes → the token is revealed.
AWS_SECRET_ACCESS_KEY=wrong swamp access token reveal probe-token --yes → see below.
Repeat step 4 with --log-level debug.

Expected

The command fails with the initialisation error, e.g. S3 headBucket failed HTTP 403 (check AWS credentials …).

Actual

Failed to read secret for token 'probe-token' from vault '_token-secrets': Vault '_token-secrets' not found. No vaults are configured.
Note: Vaults are NOT configured in .swamp.yaml. Create a vault using:
  swamp vault create <type> _token-secrets

Only visible at debug level:

[WRN] vaults·control-plane-init: Failed to initialize control-plane vault: "S3 headBucket failed HTTP 403 (check AWS credentials — …)". Token secrets will be stored in the user vault.

Full transcripts and logs are available on request.

Root cause (on `main`)

src/domain/vaults/control_plane_vault_init.ts: initializeControlPlaneVault wraps provider.initialize() in try/catch, logs a WARN and returns null, so _token-secrets is never registered via VaultService.registerGlobalProvider.
src/cli/commands/access_token_reveal.ts (≈L90–96) ignores the null, builds VaultService.fromRepository, and the lookup fails as "not found".

Suggested fix

When the control plane is remote (hasRemote), propagate the initialisation error, or have commands that need _token-secrets fail with it instead of "not found". Don't suggest vault create for the reserved _token-secrets name.

#2324 (open): the control-plane encryption key is stored beside the ciphertext. #1859 (shipped): reveal docs.

Reproduction configuration

Test environment


swamp	built from `main` @ `1da85ec847ef6a8a0ce0297563b2a9fab1f4b2b8`, unmodified (`20260923.175528.0-sha.1da85ec8`)
Datastore	`@swamp/s3-datastore@2026.09.23.1` on local MinIO (`http://127.0.0.1:19000`, bucket `swamp-repro`)
Vault	none configured; control-plane `_token-secrets` only
Instances	each serve instance starts with an empty `HOME` and a repo containing only `.swamp.yaml`

.swamp.yaml:

swampVersion: 20260923.175528.0
initializedAt: "2026-09-23T18:37:26.223Z"
repoId: 80c3b52b-367c-49d9-9dbb-8b922c911e0b
tools:
  - claude
gitignoreManaged: true
datastore:
  type: "@swamp/s3-datastore"
  config:
    bucket: swamp-repro
    prefix: 08-control-plane-init-error-swallowed-20260923183720
    endpoint: "http://127.0.0.1:19000"
    forcePathStyle: true
    region: us-east-1
  hydrationStrategy: lazy
  managedConfig: true

Package everything on ops.swamp-club.com; stop direct server edits

artifacts.swamp-club.com unreachable in Spain during LaLiga matches (ISP blocking of Cloudflare IPs)

Shared dispatch bookkeeping queue delays completed worker results by up to 58 seconds

Serve globally scans definitions twice per new direct-type step, delaying worker dispatch by minutes

Serve: execute independent webhook-triggered workflows concurrently

Update public OpenTelemetry manual for remote waterfalls and execution metrics (#3786)

serve/fleet: close OpenTelemetry attribution gaps across dispatch, data plane and worker lifecycle

serve: multi-minute write_resource latency during concurrent worker readiness

Installers intermittently fail resolving stable release: add bounded retries and actionable diagnostics

Serve: delegate namespace-scoped extension pulls and reload without granting administrator access

README.md needs updating with current language and content

Preserve active worker execution across serve restarts and upgrades; reconnect and reattach dispatches

Flaky test: lock hand-off timeout test times out in its own setup under load

Dashboard: a hidden tab keeps polling and re-rendering (health stream, live run page)

Registry client: retry transient upstream failures with bounded backoff and visible attempt progress

Pause and resume a scheduled workflow without editing or deleting its definition

Remote worker step marked failed ('aborted due to timeout') when the result report times out, though the method succeeded and its data was saved

Retention: one timeframe, enforced automatically, after which nothing older remains anywhere in the datastore

Workflow run cancellation on concurrency-key conflict

serve keeps a worker whose enroll finished after the worker timed out, then refuses the real worker as "already connected" and dispatches into the dead session

workflow run search / history search parse every run record on every call, so their cost grows with all history ever kept

Configurable storage tiers: runs, outputs, data versions and audit move through an ordered list of user-defined tiers, and stay queryable across them

Fleet visibility: live worker host map grouped by label, with slot-level lease state

Docs: dashboard reference needs the Workers view (/dashboard/workers)

serve: the per-IP WebSocket burst limit can't be configured, so workers behind one NAT IP throttle each other

swamp access check run locally ignores the named resource's stored type and tags

Activity projector fails with ZooKeeper "Bad version" error on swamp.score_grants during keeper instability

extension push: workflows listed with a shared directory prefix collapse into one workflows.yaml; all but the last are silently dropped

CLI tells signed-in callers to sign in when the registry rate-limits them

Add a subprocessors page at /subprocessors

Make Compute Routes list usable without create-only global arguments

Cloud Asset: complete metadata-only project search snapshot for lifecycle planning

Dashboard: deep links to a model's data item, data version and report

swamp report get --version N returns "Report not found" for any version older than the latest

serve: `serve reload` drops `export const extension` methods from a pulled package, while other extensions on the same type survive

serve: one unresolvable username in `auth.admins` stops the server from starting

control-plane vault init failures are swallowed; `access token reveal` says "Vault '_token-secrets' not found"

serve: boot token-secret migration deletes the source vault item

serve: boot hydration ignores `hydrationStrategy: lazy` and downloads every payload

Security Center: complete location-scoped finding snapshots for current-run reporting

IAM service accounts: verified deterministic-account lifecycle method with safe 404/409 handling

Secret Manager: complete metadata-only secret/version hygiene snapshot across projects

extension push: allow explicit private visibility in the manifest and at publish time, including public collectives

Swamp Club deno serve stays alive after SIGTERM because background loops are not stopped

Verification reviews infer pass from Claude usage-limit errors

Swamp Club fresh-database startup aborts migrations on missing deviceCode and concurrent replica markers

Logging sinks: convergent audit sink setup with typed writer identity and destination verification

Org Policy: typed batch effective-policy snapshots for baseline reports

Recommender: complete multi-scope recommendation snapshot for cost reports

Billing Budgets: convergent ensure-by-ID-or-unique-name method with etag and notification verification

Publishing manual contradicts registry omission defaults and supported visibility values

Service Usage: persist a complete paginated enabled-services snapshot for drift reports

Resource Manager: recursive caller-visible organisation hierarchy inventory with full pagination

Registry API: opt-in private visibility at publish time with fully backward-compatible defaults (prerequisite for CLI #2198)

BigQuery jobs: add a typed query method that polls, paginates and persists query results

Generic OTLP endpoint appends signal path after query or fragment

Cloud Build create supplies project instead of required projectId path parameter

Make official Docker image OCI-compliant

Issue lifecycle start drops top-level issue comments

Global Secret Manager methods build an invalid regional parent

No way to store code-review results — verdicts, findings, and cost are not persisted for any repo

Relationships

#2437 control-plane vault init failures are swallowed; `access token reveal` says "Vault '_token-secrets' not found"