Skip to main content
← Back to list
01Issue
BugShippedExtensionsPublic
Assigneesstack72

Relationships

#1805 @swamp/s3-datastore disables IMDS on plain EC2, so an attached instance profile cannot resolve

Opened by keeb · 8/24/2026· Shipped 8/25/2026

Found while documenting AWS credential resolution for #1248.

Problem

datastore/s3/extensions/datastores/_lib/s3_client.ts:347-353 disables the AWS SDK instance metadata credential lookup:

if (
  !Deno.env.get("AWS_EC2_METADATA_DISABLED") &&
  !Deno.env.get("AWS_CONTAINER_CREDENTIALS_RELATIVE_URI") &&
  !Deno.env.get("AWS_CONTAINER_CREDENTIALS_FULL_URI")
) {
  Deno.env.set("AWS_EC2_METADATA_DISABLED", "true");
}

The comment above it says "When not running in a container environment (ECS/EKS)" and "Off-cloud, the metadata endpoint is unreachable". Avoiding the IMDS stall off-cloud is the right goal; it is half of the #803 fix.

But there is no EC2 detection. The only things that suppress the disable are the two ECS/EKS container-credential variables. On a plain EC2 instance drawing credentials from an attached instance profile, neither is set, so the datastore disables the exact credential source it is meant to use. IMDS is where instance-profile credentials come from on EC2; ECS and EKS use the container endpoint instead, which is why only those two cases are covered.

Expected outcome on such a host: credentials do not resolve, and the #803 preflight fails in about 3 seconds with either a CredentialsProviderError classified as session-expired, or:

Credential preflight timed out after 3000ms — verify that AWS credentials are configured (AWS_ACCESS_KEY_ID, AWS_PROFILE, or attached IAM role) and that the credential source is responsive

which names the attached IAM role as a thing to check, having just disabled it.

This is from reading the code, not from a reproduction on EC2. Worth confirming on a real instance before fixing, because a careless change here re-introduces the off-cloud stall that #803 removed.

Why it has stayed quiet

Anyone running swamp on EC2 with a profile or static env credentials is unaffected, because those providers resolve earlier in the chain than IMDS. It only bites a host whose only credential source is the instance profile.

Documented behaviour disagrees

Both the extension description and content/manual/reference/datastore-configuration.md list "IAM role attached to the instance or task" as a supported credential source. On plain EC2 that is currently false without an explicit opt-out.

Workaround

Set AWS_EC2_METADATA_DISABLED=false. The guard tests truthiness of the string, so any value prevents swamp from setting it to true, and the SDK reads "false" as not-disabled. This is now written down at /manual/how-to/authenticate-to-aws-with-sso and in both extension descriptions, but it should not be necessary.

Suggested fix

Probe IMDS with a short timeout rather than disabling it outright, or gate the disable on an actual EC2 check. Whichever is chosen belongs in the shared credential module proposed in #1804 rather than only in the s3 client, since the same off-cloud stall affects the vault and the generated model family, neither of which carries this code at all.

Upstream repository: https://github.com/systeminit/swamp-extensions

02Bog Flow
✓OPEN✓TRIAGED✓IN PROGRESS✓SHIPPED+ 1 MOREASSIGNED+ 4 MOREREVIEW+ 5 MOREPR_MERGED+ 2 MORESESSION_SUMMARIZED

Shipped

8/25/2026, 3:41:04 AM

Click a lifecycle step above to view its details.

03Sludge Pulse
stack72 assigned stack728/25/2026, 2:15:27 AM

Sign in to post a ripple.