The Credential Precedence Mistake That Shows Up Two Weeks Later in an Audit

Working extensively with AWS credentials in Kubernetes this quarter revealed how often credential precedence causes configuration issues. While the AWS SDK’s credential chain is well-designed, understanding the priority order is crucial for production deployments. Here’s what I’ve learned. The Problem Nobody Talks About A recent incident illustrated this well: We configured IRSA for a microservice, validated it in staging, and deployed to production successfully. Two weeks later, an audit revealed the service was using broader IAM permissions than expected. The cause was an AWS_ACCESS_KEY_ID environment variable in a Secret that was taking precedence over the IRSA configuration. The SDK found credentials and stopped looking. It never even checked IRSA. The SDK found credentials and stopped looking. It never even checked IRSA. This is the #1 source of credential-related incidents I’ve seen in Kubernetes environments. The credential chain uses “first match wins” logic, and understanding this precedence is critical. The Credential Chain: Priority Order In most AWS SDKs, the default credential chain generally evaluates credentials in the following order, stopping at the first valid credentials: Key Insight: The SDK doesn’t validate permissions or check if credentials are appropriate—it just uses the first valid credentials it finds. Key Insight: Why Precedence Matters: The Shadow Effect When multiple credential sources exist, higher-priority sources “shadow” lower-priority ones: The Four Credential Providers 1. Environment Variables (Highest Priority) 🔴 Environment Variables: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN Environment Variables When to use: Local development, CI/CD pipelines where you control the environment completely. When to use: When NOT to use: Production Kubernetes—too easy to accidentally commit or misconfigure. When NOT to use: Go Example: Go Example: // SDK automatically picks these up cfg, err := config.LoadDefaultConfig(ctx) // Will use env vars if present, regardless of IRSA/Pod Identity configuration! // SDK automatically picks these up cfg, err := config.LoadDefaultConfig(ctx) // Will use env vars if present, regardless of IRSA/Pod Identity configuration! Kubernetes Manifest (Anti-pattern): Kubernetes Manifest (Anti-pattern): spec: containers: - name: app env: - name: AWS_ACCESS_KEY_ID value: "AKIAI..." # ⚠️ This shadows everything else! spec: containers: - name: app env: - name: AWS_ACCESS_KEY_ID value: "AKIAI..." # ⚠️ This shadows everything else! The Shadow Problem: If these are set anywhere—in a ConfigMap, Secret, or Dockerfile—they will override all other credential sources. The Shadow Problem: 2. Web Identity Token: IRSA vs EKS Pod Identity (Recommended) ✅ AWS provides two modern approaches for pod-level credentials in EKS. Both use the Web Identity Token provider in the credential chain, but they work differently under the hood. Understanding the Two Approaches IRSA (IAM Roles for Service Accounts) – The Original How it works: How it works: Setup Requirements: Setup Requirements: OIDC provider configured in IAM Service Account annotation IAM role with trust policy referencing OIDC provider OIDC provider configured in IAM Service Account annotation IAM role with trust policy referencing OIDC provider Configuration: Configuration: apiVersion: v1 kind: ServiceAccount metadata: name: my-app-sa annotations: eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/my-app-role --- apiVersion: v1 kind: Pod metadata: name: my-app spec: serviceAccountName: my-app-sa containers: - name: app image: myapp:latest apiVersion: v1 kind: ServiceAccount metadata: name: my-app-sa annotations: eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/my-app-role --- apiVersion: v1 kind: Pod metadata: name: my-app spec: serviceAccountName: my-app-sa containers: - name: app image: myapp:latest What gets injected: What gets injected: # Environment variables AWS_ROLE_ARN=arn:aws:iam::123456789012:role/my-app-role AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token # Volume mount /var/run/secrets/eks.amazonaws.com/serviceaccount/token (JWT, auto-refreshed) # Environment variables AWS_ROLE_ARN=arn:aws:iam::123456789012:role/my-app-role AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token # Volume mount /var/run/secrets/eks.amazonaws.com/serviceaccount/token (JWT, auto-refreshed) Go Code: Go Code: // SDK automatically detects IRSA configuration cfg, err := config.LoadDefaultConfig(ctx) // SDK reads token and exchanges it transparently // SDK automatically detects IRSA configuration cfg, err := config.LoadDefaultConfig(ctx) // SDK reads token and exchanges it transparently Pros: Pros: ✅ Works across AWS accounts (cross-account assume role) ✅ OIDC standard, portable to other Kubernetes environments ✅ Fine-grained control with IAM trust policies ✅ Works across AWS accounts (cross-account assume role) ✅ OIDC standard, portable to other Kubernetes environments ✅ Fine-grained control with IAM trust policies Cons: Cons: ⚠️ Requires OIDC provider setup (one-time per cluster) ⚠️ Trust policy can be complex for multi-tenant scenarios ⚠️ Token validation happens during credential refresh cycles, not on every AWS API call. ⚠️ Requires OIDC provider setup (one-time per cluster) ⚠️ Trust policy can be complex for multi-tenant scenarios ⚠️ Token validation happens during credential refresh cycles, not on every AWS API call. EKS Pod Identity – The New Standard Introduced in late 2023, EKS Pod Identity simplifies credential management with a cluster add-on. Introduced in late 2023 How it works: How it works: Setup Requirements: Setup Requirements: EKS Pod Identity add-on installed on cluster Pod Identity association created (links ServiceAccount to IAM role) No customer-managed OIDC provider configuration is required. EKS Pod Identity add-on installed on cluster Pod Identity association created (links ServiceAccount to IAM role) No customer-managed OIDC provider configuration is required. Configuration: Configuration: # Create IAM role (standard role, no special trust policy needed) aws iam create-role --role-name my-app-role --assume-role-policy-document '{ "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": {"Service": "pods.eks.amazonaws.com"}, "Action": ["sts:AssumeRole", "sts:TagSession"] }] }' # Create Pod Identity association aws eks create-pod-identity-association \ --cluster-name my-cluster \ --namespace default \ --service-account my-app-sa \ --role-arn arn:aws:iam::123456789012:role/my-app-role # NEW (June 2025): Native cross-account support # Specify both source and target role ARNs for cross-account access aws eks create-pod-identity-association \ --cluster-name my-cluster \ --namespace default \ --service-account my-app-sa \ --role-arn arn:aws:iam::111111111111:role/source-account-role \ --target-role-arn arn:aws:iam::222222222222:role/target-account-role # Create IAM role (standard role, no special trust policy needed) aws iam create-role --role-name my-app-role --assume-role-policy-document '{ "Version": "2012-10-17", "Statement": [{ "Effect": "Allow", "Principal": {"Service": "pods.eks.amazonaws.com"}, "Action": ["sts:AssumeRole", "sts:TagSession"] }] }' # Create Pod Identity association aws eks create-pod-identity-association \ --cluster-name my-cluster \ --namespace default \ --service-account my-app-sa \ --role-arn arn:aws:iam::123456789012:role/my-app-role # NEW (June 2025): Native cross-account support # Specify both source and target role ARNs for cross-account access aws eks create-pod-identity-association \ --cluster-name my-cluster \ --namespace default \ --service-account my-app-sa \ --role-arn arn:aws:iam::111111111111:role/source-account-role \ --target-role-arn arn:aws:iam::222222222222:role/target-account-role Kubernetes manifest (simpler!): Kubernetes manifest (simpler!): apiVersion: v1 kind: ServiceAccount metadata: name: my-app-sa # No annotations needed! --- apiVersion: v1 kind: Pod metadata: name: my-app spec: serviceAccountName: my-app-sa containers: - name: app image: myapp:latest apiVersion: v1 kind: ServiceAccount metadata: name: my-app-sa # No annotations needed! --- apiVersion: v1 kind: Pod metadata: name: my-app spec: serviceAccountName: my-app-sa containers: - name: app image: myapp:latest What gets injected: What gets injected: # Environment variables (different from IRSA!) AWS_CONTAINER_CREDENTIALS_FULL_URI=http://169.254.170.23/v1/credentials AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE=/var/run/secrets/pods.eks.amazonaws.com/serviceaccount/eks-pod-identity-token # Volume mount /var/run/secrets/pods.eks.amazonaws.com/serviceaccount/eks-pod-identity-token # Environment variables (different from IRSA!) AWS_CONTAINER_CREDENTIALS_FULL_URI=http://169.254.170.23/v1/credentials AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE=/var/run/secrets/pods.eks.amazonaws.com/serviceaccount/eks-pod-identity-token # Volume mount /var/run/secrets/pods.eks.amazonaws.com/serviceaccount/eks-pod-identity-token Go Code (identical to IRSA): Go Code (identical to IRSA): // SDK automatically detects Pod Identity configuration cfg, err := config.LoadDefaultConfig(ctx) // SDK calls the Pod Identity agent transparently // SDK automatically detects Pod Identity configuration cfg, err := config.LoadDefaultConfig(ctx) // SDK calls the Pod Identity agent transparently Pros: Pros: ✅ Simpler setup (No customer-managed OIDC provider configuration is required) ✅ Pod Identity often results in lower latency because the SDK talks to a local agent, which handles STS interactions and caching on behalf of the pod. ✅ Better multi-tenant isolation ✅ Centralized association management ✅ Works with EKS versions 1.24+ ✅ Native cross-account support (added June 2025) – automatic IAM role chaining with external ID for security ✅ Simpler setup (No customer-managed OIDC provider configuration is required) ✅ Pod Identity often results in lower latency because the SDK talks to a local agent, which handles STS interactions and caching on behalf of the pod. ✅ Better multi-tenant isolation ✅ Centralized association management ✅ Works with EKS versions 1.24+ ✅ Native cross-account support (added June 2025) – automatic IAM role chaining with external ID for security Native cross-account support (added June 2025) Cons: Cons: ⚠️ EKS-specific (not portable to other Kubernetes) ⚠️ Requires cluster add-on installation ⚠️ EKS-specific (not portable to other Kubernetes) ⚠️ Requires cluster add-on installation IRSA vs Pod Identity: When to Use Which? Decision Matrix: Decision Matrix: Criteria IRSA Pod Identity Setup Complexity Medium (OIDC provider) Low (add-on) Cross-Account Access ✅ Yes ✅ Yes (native as of June 2025) Performance Good (STS call) Better (local agent) EKS Version Any 1.24+ Portability High (OIDC standard) Low (EKS only) Multi-Tenancy Manual (trust policy) Built-in (associations) Credential Refresh STS via internet Local agent Criteria IRSA Pod Identity Setup Complexity Medium (OIDC provider) Low (add-on) Cross-Account Access ✅ Yes ✅ Yes (native as of June 2025) Performance Good (STS call) Better (local agent) EKS Version Any 1.24+ Portability High (OIDC standard) Low (EKS only) Multi-Tenancy Manual (trust policy) Built-in (associations) Credential Refresh STS via internet Local agent Criteria IRSA Pod Identity Criteria Criteria IRSA IRSA Pod Identity Pod Identity Setup Complexity Medium (OIDC provider) Low (add-on) Setup Complexity Setup Complexity Medium (OIDC provider) Medium (OIDC provider) Low (add-on) Low (add-on) Cross-Account Access ✅ Yes ✅ Yes (native as of June 2025) Cross-Account Access Cross-Account Access ✅ Yes ✅ Yes ✅ Yes (native as of June 2025) ✅ Yes (native as of June 2025) Performance Good (STS call) Better (local agent) Performance Performance Good (STS call) Good (STS call) Better (local agent) Better (local agent) EKS Version Any 1.24+ EKS Version EKS Version Any Any 1.24+ 1.24+ Portability High (OIDC standard) Low (EKS only) Portability Portability High (OIDC standard) High (OIDC standard) Low (EKS only) Low (EKS only) Multi-Tenancy Manual (trust policy) Built-in (associations) Multi-Tenancy Multi-Tenancy Manual (trust policy) Manual (trust policy) Built-in (associations) Built-in (associations) Credential Refresh STS via internet Local agent Credential Refresh Credential Refresh STS via internet STS via internet Local agent Local agent My Recommendation: My Recommendation: New EKS clusters (1.24+): Start with Pod Identity for simplicity and performance Existing IRSA deployments: No rush to migrate unless you hit issues Cross-account scenarios: Both IRSA and Pod Identity now support native cross-account access (Pod Identity added this in June 2025) High-traffic applications: Pod Identity for better performance New EKS clusters (1.24+): Start with Pod Identity for simplicity and performance New EKS clusters (1.24+) Pod Identity Existing IRSA deployments: No rush to migrate unless you hit issues Existing IRSA deployments Cross-account scenarios: Both IRSA and Pod Identity now support native cross-account access (Pod Identity added this in June 2025) Cross-account scenarios IRSA Pod Identity High-traffic applications: Pod Identity for better performance High-traffic applications Pod Identity SDK Behavior with Both Approaches The beauty is that from your application’s perspective, both are transparent: package main import ( "context" "fmt" "os" "github.com/aws/aws-sdk-go-v2/config" "github.com/aws/aws-sdk-go-v2/service/s3" ) func main() { ctx := context.Background() // SDK automatically detects either IRSA or Pod Identity cfg, err := config.LoadDefaultConfig(ctx) if err != nil { panic(err) } // Check which mechanism is being used (for debugging) creds, _ := cfg.Credentials.Retrieve(ctx) fmt.Printf("Credential Source: %s\n", creds.Source) // For IRSA: WebIdentityTokenProvider // Pod Identity: ContainerCredentialsProvider (SDK v2) // But different env vars under the hood if os.Getenv("AWS_CONTAINER_CREDENTIALS_FULL_URI") != "" { fmt.Println("Using: EKS Pod Identity") } else if os.Getenv("AWS_ROLE_ARN") != "" { fmt.Println("Using: IRSA") } // Use AWS services normally s3Client := s3.NewFromConfig(cfg) output, _ := s3Client.ListBuckets(ctx, &s3.ListBucketsInput{}) fmt.Printf("Found %d buckets\n", len(output.Buckets)) } package main import ( "context" "fmt" "os" "github.com/aws/aws-sdk-go-v2/config" "github.com/aws/aws-sdk-go-v2/service/s3" ) func main() { ctx := context.Background() // SDK automatically detects either IRSA or Pod Identity cfg, err := config.LoadDefaultConfig(ctx) if err != nil { panic(err) } // Check which mechanism is being used (for debugging) creds, _ := cfg.Credentials.Retrieve(ctx) fmt.Printf("Credential Source: %s\n", creds.Source) // For IRSA: WebIdentityTokenProvider // Pod Identity: ContainerCredentialsProvider (SDK v2) // But different env vars under the hood if os.Getenv("AWS_CONTAINER_CREDENTIALS_FULL_URI") != "" { fmt.Println("Using: EKS Pod Identity") } else if os.Getenv("AWS_ROLE_ARN") != "" { fmt.Println("Using: IRSA") } // Use AWS services normally s3Client := s3.NewFromConfig(cfg) output, _ := s3Client.ListBuckets(ctx, &s3.ListBucketsInput{}) fmt.Printf("Found %d buckets\n", len(output.Buckets)) } 3. Shared Credentials File Location: ~/.aws/credentials (or AWS_SHARED_CREDENTIALS_FILE) Location: ~/.aws/credentials Format: Format: [default] aws_access_key_id = AKIAI... aws_secret_access_key = ... [production] aws_access_key_id = AKIAI... aws_secret_access_key = ... [default] aws_access_key_id = AKIAI... aws_secret_access_key = ... [production] aws_access_key_id = AKIAI... aws_secret_access_key = ... Use case: Multi-account scenarios, legacy migrations Use case: Kubernetes pattern: Kubernetes pattern: # Mount credentials file from ConfigMap/Secret volumes: - name: aws-creds secret: secretName: aws-credentials volumeMounts: - name: aws-creds mountPath: /root/.aws readOnly: true # Mount credentials file from ConfigMap/Secret volumes: - name: aws-creds secret: secretName: aws-credentials volumeMounts: - name: aws-creds mountPath: /root/.aws readOnly: true Rarely needed in modern Kubernetes deployments where IRSA or Pod Identity is available. Rarely needed 4. EC2 Instance Metadata (IMDS) (Lowest Priority) How it works: SDK queries the instance metadata service at http://169.254.169.254/latest/meta-data/ How it works: http://169.254.169.254/latest/meta-data/ In Kubernetes context: Returns the EKS node’s IAM role, not pod-specific credentials. In Kubernetes context: EKS node’s IAM role The Problem: The Problem: All pods on the node inherit the same permissions—violates least privilege. All pods on the node inherit the same permissions Disable IMDS when using IRSA/Pod Identity: Disable IMDS when using IRSA/Pod Identity: env: - name: AWS_EC2_METADATA_DISABLED value: "true" env: - name: AWS_EC2_METADATA_DISABLED value: "true" Or use IMDSv2 with hop limit to prevent pod access (node-level configuration). Or Common Precedence Mistakes Mistake #1: The Silent Shadow Setup: Setup: # You configure IRSA or Pod Identity (good) apiVersion: v1 kind: ServiceAccount metadata: name: my-app-sa annotations: eks.amazonaws.com/role-arn: arn:aws:iam::123:role/restricted-role But your ConfigMap has this (bad) # You configure IRSA or Pod Identity (good) apiVersion: v1 kind: ServiceAccount metadata: name: my-app-sa annotations: eks.amazonaws.com/role-arn: arn:aws:iam::123:role/restricted-role --- # But your ConfigMap has this (bad) apiVersion: v1 kind: ConfigMap metadata: name: app-config data: AWS_ACCESS_KEY_ID: "AKIAI..." # ⚠️ Left over from testing # You configure IRSA or Pod Identity (good) apiVersion: v1 kind: ServiceAccount metadata: name: my-app-sa annotations: eks.amazonaws.com/role-arn: arn:aws:iam::123:role/restricted-role --- # But your ConfigMap has this (bad) apiVersion: v1 kind: ConfigMap metadata: name: app-config data: AWS_ACCESS_KEY_ID: "AKIAI..." # ⚠️ Left over from testing Result: App uses the ConfigMap credentials (full admin!), not IRSA/Pod Identity (restricted). No errors, no warnings—silent security violation. Result: Detection: Detection: // Add this to your app initialization creds, _ := cfg.Credentials.Retrieve(ctx) if creds.Source != "WebIdentityTokenProvider" { log.Warnf("Expected Web Identity but got: %s", creds.Source) } // Or check the specific mechanism if os.Getenv("AWS_ACCESS_KEY_ID") != "" { log.Error("Environment credentials are shadowing IRSA/Pod Identity!") } // Add this to your app initialization creds, _ := cfg.Credentials.Retrieve(ctx) if creds.Source != "WebIdentityTokenProvider" { log.Warnf("Expected Web Identity but got: %s", creds.Source) } // Or check the specific mechanism if os.Getenv("AWS_ACCESS_KEY_ID") != "" { log.Error("Environment credentials are shadowing IRSA/Pod Identity!") } Mistake #2: Mixing IRSA and Pod Identity Setup: Setup: # ServiceAccount has IRSA annotation apiVersion: v1 kind: ServiceAccount metadata: annotations: eks.amazonaws.com/role-arn: arn:aws:iam::123:role/irsa-role --- # But you also created a Pod Identity association via CLI # aws eks create-pod-identity-association --service-account my-app-sa ... # ServiceAccount has IRSA annotation apiVersion: v1 kind: ServiceAccount metadata: annotations: eks.amazonaws.com/role-arn: arn:aws:iam::123:role/irsa-role --- # But you also created a Pod Identity association via CLI # aws eks create-pod-identity-association --service-account my-app-sa ... What happens: Mixing IRSA and Pod Identity leads to undefined and SDK-dependent behavior and should be avoided. What happens: Result: Confusion in debugging, potential permission mismatches. Result: Fix: Choose one mechanism per ServiceAccount and stick with it. Fix: Mistake #3: The Typo Fallback Setup: Setup: apiVersion: v1 kind: ServiceAccount metadata: annotations: eks.amazonaws.com/role-arn: arn:aws:iam::123:role/my-rol # Missing 'e'! apiVersion: v1 kind: ServiceAccount metadata: annotations: eks.amazonaws.com/role-arn: arn:aws:iam::123:role/my-rol # Missing 'e'! What happens: What happens: Environment variables: ❌ Not set Web Identity (IRSA): ❌ Invalid role ARN, STS call fails Shared credentials: ❌ No file IMDS: Depending on SDK behavior and error handling, a failed Web Identity exchange may result in either an immediate failure or a fallback to the next provider (such as IMDS). Environment variables: ❌ Not set Web Identity (IRSA): ❌ Invalid role ARN, STS call fails Shared credentials: ❌ No file IMDS: Depending on SDK behavior and error handling, a failed Web Identity exchange may result in either an immediate failure or a fallback to the next provider (such as IMDS). Depending on SDK behavior and error handling, a failed Web Identity exchange may result in either an immediate failure or a fallback to the next provider (such as IMDS). Result: App works but with wrong (usually over-privileged) permissions. Result: Mistake #4: Docker Image Pollution # Dockerfile (bad practice) FROM golang:1.21 # Someone added these during testing... ENV AWS_ACCESS_KEY_ID=AKIAI... ENV AWS_SECRET_ACCESS_KEY=... COPY . . RUN go build -o app CMD ["./app"] # Dockerfile (bad practice) FROM golang:1.21 # Someone added these during testing... ENV AWS_ACCESS_KEY_ID=AKIAI... ENV AWS_SECRET_ACCESS_KEY=... COPY . . RUN go build -o app CMD ["./app"] Result: Every pod using this image ignores IRSA/Pod Identity and uses hardcoded credentials. Result: Better approach: Better approach: FROM golang:1.21 COPY . . RUN go build -o app # No AWS credentials in image! # Let Kubernetes inject them via IRSA/Pod Identity CMD ["./app"] FROM golang:1.21 COPY . . RUN go build -o app # No AWS credentials in image! # Let Kubernetes inject them via IRSA/Pod Identity CMD ["./app"] Debugging Credential Chain Issues Enhanced Diagnostic Tool package main import ( "context" "fmt" "os" "github.com/aws/aws-sdk-go-v2/config" "github.com/aws/aws-sdk-go-v2/service/sts" ) func main() { ctx := context.Background() fmt.Println("=== Credential Chain Status ===") // Check Priority 1: Environment Variables fmt.Println("\n1. Environment Variables:") if os.Getenv("AWS_ACCESS_KEY_ID") != "" { fmt.Println(" ⚠️ AWS_ACCESS_KEY_ID is set (shadows everything!)") } else { fmt.Println(" ✓ Not set") } // Check Priority 2: Web Identity (IRSA vs Pod Identity) fmt.Println("\n2. Web Identity Token:") if roleArn := os.Getenv("AWS_ROLE_ARN"); roleArn != "" { fmt.Printf(" ✓ IRSA configured\n") fmt.Printf(" Role: %s\n", roleArn) fmt.Printf(" Token: %s\n", os.Getenv("AWS_WEB_IDENTITY_TOKEN_FILE")) } else if credsUri := os.Getenv("AWS_CONTAINER_CREDENTIALS_FULL_URI"); credsUri != "" { fmt.Printf(" ✓ EKS Pod Identity configured\n") fmt.Printf(" URI: %s\n", credsUri) fmt.Printf(" Token: %s\n", os.Getenv("AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE")) } else { fmt.Println(" ✗ Not configured") } // Check Priority 3: Shared Credentials fmt.Println("\n3. Shared Credentials File:") credsFile := os.Getenv("AWS_SHARED_CREDENTIALS_FILE") if credsFile == "" { credsFile = os.ExpandEnv("$HOME/.aws/credentials") } if _, err := os.Stat(credsFile); err == nil { fmt.Printf(" ⚠️ Found: %s\n", credsFile) } else { fmt.Println(" ✓ Not found") } // Check Priority 4: IMDS fmt.Println("\n4. EC2 Instance Metadata:") if os.Getenv("AWS_EC2_METADATA_DISABLED") == "true" { fmt.Println(" ✓ Disabled") } else { fmt.Println(" ⚠️ Enabled (may fallback to node credentials)") } // Load config and see what's actually used fmt.Println("\n=== Active Credentials ===") cfg, err := config.LoadDefaultConfig(ctx) if err != nil { fmt.Printf("❌ Error: %v\n", err) return } creds, _ := cfg.Credentials.Retrieve(ctx) fmt.Printf("🎯 Source: %s\n", creds.Source) // Verify identity stsClient := sts.NewFromConfig(cfg) identity, _ := stsClient.GetCallerIdentity(ctx, &sts.GetCallerIdentityInput{}) fmt.Printf("Identity ARN: %s\n", *identity.Arn) // Provide recommendations fmt.Println("\n=== Recommendations ===") if creds.Source != "WebIdentityTokenProvider" && os.Getenv("ENVIRONMENT") == "production" { fmt.Println("⚠️ WARNING: Not using Web Identity (IRSA/Pod Identity) in production!") fmt.Println(" Consider configuring IRSA or Pod Identity for better security") } else if creds.Source == "WebIdentityTokenProvider" { if os.Getenv("AWS_CONTAINER_CREDENTIALS_FULL_URI") != "" { fmt.Println("✅ Using EKS Pod Identity - optimal setup!") } else { fmt.Println("✅ Using IRSA - good setup!") } } } package main import ( "context" "fmt" "os" "github.com/aws/aws-sdk-go-v2/config" "github.com/aws/aws-sdk-go-v2/service/sts" ) func main() { ctx := context.Background() fmt.Println("=== Credential Chain Status ===") // Check Priority 1: Environment Variables fmt.Println("\n1. Environment Variables:") if os.Getenv("AWS_ACCESS_KEY_ID") != "" { fmt.Println(" ⚠️ AWS_ACCESS_KEY_ID is set (shadows everything!)") } else { fmt.Println(" ✓ Not set") } // Check Priority 2: Web Identity (IRSA vs Pod Identity) fmt.Println("\n2. Web Identity Token:") if roleArn := os.Getenv("AWS_ROLE_ARN"); roleArn != "" { fmt.Printf(" ✓ IRSA configured\n") fmt.Printf(" Role: %s\n", roleArn) fmt.Printf(" Token: %s\n", os.Getenv("AWS_WEB_IDENTITY_TOKEN_FILE")) } else if credsUri := os.Getenv("AWS_CONTAINER_CREDENTIALS_FULL_URI"); credsUri != "" { fmt.Printf(" ✓ EKS Pod Identity configured\n") fmt.Printf(" URI: %s\n", credsUri) fmt.Printf(" Token: %s\n", os.Getenv("AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE")) } else { fmt.Println(" ✗ Not configured") } // Check Priority 3: Shared Credentials fmt.Println("\n3. Shared Credentials File:") credsFile := os.Getenv("AWS_SHARED_CREDENTIALS_FILE") if credsFile == "" { credsFile = os.ExpandEnv("$HOME/.aws/credentials") } if _, err := os.Stat(credsFile); err == nil { fmt.Printf(" ⚠️ Found: %s\n", credsFile) } else { fmt.Println(" ✓ Not found") } // Check Priority 4: IMDS fmt.Println("\n4. EC2 Instance Metadata:") if os.Getenv("AWS_EC2_METADATA_DISABLED") == "true" { fmt.Println(" ✓ Disabled") } else { fmt.Println(" ⚠️ Enabled (may fallback to node credentials)") } // Load config and see what's actually used fmt.Println("\n=== Active Credentials ===") cfg, err := config.LoadDefaultConfig(ctx) if err != nil { fmt.Printf("❌ Error: %v\n", err) return } creds, _ := cfg.Credentials.Retrieve(ctx) fmt.Printf("🎯 Source: %s\n", creds.Source) // Verify identity stsClient := sts.NewFromConfig(cfg) identity, _ := stsClient.GetCallerIdentity(ctx, &sts.GetCallerIdentityInput{}) fmt.Printf("Identity ARN: %s\n", *identity.Arn) // Provide recommendations fmt.Println("\n=== Recommendations ===") if creds.Source != "WebIdentityTokenProvider" && os.Getenv("ENVIRONMENT") == "production" { fmt.Println("⚠️ WARNING: Not using Web Identity (IRSA/Pod Identity) in production!") fmt.Println(" Consider configuring IRSA or Pod Identity for better security") } else if creds.Source == "WebIdentityTokenProvider" { if os.Getenv("AWS_CONTAINER_CREDENTIALS_FULL_URI") != "" { fmt.Println("✅ Using EKS Pod Identity - optimal setup!") } else { fmt.Println("✅ Using IRSA - good setup!") } } } The Complete Precedence Decision Tree Best Practices for Production 1. Choose Your Web Identity Mechanism For new deployments on EKS 1.24+: For new deployments on EKS 1.24+: # Install Pod Identity add-on eksctl create addon --name eks-pod-identity-agent --cluster my-cluster Create association # Install Pod Identity add-on eksctl create addon --name eks-pod-identity-agent --cluster my-cluster # Create association aws eks create-pod-identity-association \ --cluster-name my-cluster \ --namespace production \ --service-account my-app-sa \ --role-arn arn:aws:iam::123456789012:role/my-app-role # Install Pod Identity add-on eksctl create addon --name eks-pod-identity-agent --cluster my-cluster # Create association aws eks create-pod-identity-association \ --cluster-name my-cluster \ --namespace production \ --service-account my-app-sa \ --role-arn arn:aws:iam::123456789012:role/my-app-role For cross-account or multi-cloud: For cross-account or multi-cloud: # Use IRSA with OIDC provider eksctl utils associate-iam-oidc-provider --cluster my-cluster --approve # Create role with trust policy # Then annotate ServiceAccount # Use IRSA with OIDC provider eksctl utils associate-iam-oidc-provider --cluster my-cluster --approve # Create role with trust policy # Then annotate ServiceAccount 2. Enforce with Admission Control # Pseudocode for admission webhook function validatePod(pod): hasEnvCreds = pod has AWS_ACCESS_KEY_ID env var hasWebIdentity = pod.serviceAccount has role-arn annotation OR pod identity association exists if hasEnvCreds and hasWebIdentity: return DENY: "Cannot mix env credentials with Web Identity" if production namespace and not hasWebIdentity: return DENY: "Production pods must use IRSA or Pod Identity" return ALLOW # Pseudocode for admission webhook function validatePod(pod): hasEnvCreds = pod has AWS_ACCESS_KEY_ID env var hasWebIdentity = pod.serviceAccount has role-arn annotation OR pod identity association exists if hasEnvCreds and hasWebIdentity: return DENY: "Cannot mix env credentials with Web Identity" if production namespace and not hasWebIdentity: return DENY: "Production pods must use IRSA or Pod Identity" return ALLOW 3. Clean Dockerfile Hygiene FROM golang:1.21 as builder WORKDIR /app COPY . . RUN go build -o myapp FROM gcr.io/distroless/base-debian12 # CRITICAL: No AWS credentials in ENV # CRITICAL: No .aws directories in image COPY --from=builder /app/myapp / # Disable IMDS fallback (optional but recommended) ENV AWS_EC2_METADATA_DISABLED=true ENTRYPOINT ["/myapp"] FROM golang:1.21 as builder WORKDIR /app COPY . . RUN go build -o myapp FROM gcr.io/distroless/base-debian12 # CRITICAL: No AWS credentials in ENV # CRITICAL: No .aws directories in image COPY --from=builder /app/myapp / # Disable IMDS fallback (optional but recommended) ENV AWS_EC2_METADATA_DISABLED=true ENTRYPOINT ["/myapp"] 4. Application-Level Validation func initAWSClient(ctx context.Context) (*aws.Config, error) { cfg, err := config.LoadDefaultConfig(ctx) if err != nil { return nil, err } // Verify we're using expected credentials creds, err := cfg.Credentials.Retrieve(ctx) if err != nil { return nil, err } // In production, enforce Web Identity if os.Getenv("ENV") == "production" { if creds.Source != "WebIdentityTokenProvider" { return nil, fmt.Errorf( "production requires Web Identity (IRSA/Pod Identity), got: %s", creds.Source, ) } // Log which mechanism is being used if os.Getenv("AWS_CONTAINER_CREDENTIALS_FULL_URI") != "" { log.Info("Using EKS Pod Identity") } else { log.Info("Using IRSA") } } return &cfg, nil } func initAWSClient(ctx context.Context) (*aws.Config, error) { cfg, err := config.LoadDefaultConfig(ctx) if err != nil { return nil, err } // Verify we're using expected credentials creds, err := cfg.Credentials.Retrieve(ctx) if err != nil { return nil, err } // In production, enforce Web Identity if os.Getenv("ENV") == "production" { if creds.Source != "WebIdentityTokenProvider" { return nil, fmt.Errorf( "production requires Web Identity (IRSA/Pod Identity), got: %s", creds.Source, ) } // Log which mechanism is being used if os.Getenv("AWS_CONTAINER_CREDENTIALS_FULL_URI") != "" { log.Info("Using EKS Pod Identity") } else { log.Info("Using IRSA") } } return &cfg, nil } 5. Monitor with CloudWatch Set up alerts for unexpected credential usage: -- CloudWatch Logs Insights Query fields @timestamp, userIdentity.arn, sourceIPAddress | filter userIdentity.arn not like /expected-role-name/ | filter eventSource = "s3.amazonaws.com" | stats count() by userIdentity.arn -- CloudWatch Logs Insights Query fields @timestamp, userIdentity.arn, sourceIPAddress | filter userIdentity.arn not like /expected-role-name/ | filter eventSource = "s3.amazonaws.com" | stats count() by userIdentity.arn Real-World Architecture Pattern Here’s how we structure credentials across different environments: Our migration strategy: Our migration strategy: New services: Start with Pod Identity (EKS 1.24+) Existing IRSA: Keep as-is, migrate opportunistically Legacy IMDS: Migrate to Pod Identity with tight timelines Dev environments: Allow IMDS with minimal permissions New services: Start with Pod Identity (EKS 1.24+) New services Existing IRSA: Keep as-is, migrate opportunistically Existing IRSA Legacy IMDS: Migrate to Pod Identity with tight timelines Legacy IMDS Dev environments: Allow IMDS with minimal permissions Dev environments Troubleshooting Flowchart Summary: The Modern Precedence Pyramid Remember the credential chain as a pyramid—the SDK checks from top to bottom and stops at the first layer it finds: Golden Rules In production, use Web Identity exclusively (IRSA or Pod Identity) Never set AWS_ACCESS_KEY_ID in production — it shadows everything Choose Pod Identity for new EKS 1.24+ deployments — simpler and faster Use IRSA when you need cross-account access — more flexible trust policies Explicitly disable IMDS when using Web Identity to prevent fallback Validate credentials at app startup — fail fast if not using expected source Monitor CloudTrail for unexpected IAM ARNs making API calls Don’t mix IRSA and Pod Identity on the same ServiceAccount In production, use Web Identity exclusively (IRSA or Pod Identity) In production, use Web Identity exclusively Never set AWS_ACCESS_KEY_ID in production — it shadows everything Never set AWS_ACCESS_KEY_ID in production Choose Pod Identity for new EKS 1.24+ deployments — simpler and faster Choose Pod Identity for new EKS 1.24+ deployments Use IRSA when you need cross-account access — more flexible trust policies Use IRSA when you need cross-account access Explicitly disable IMDS when using Web Identity to prevent fallback Explicitly disable IMDS Validate credentials at app startup — fail fast if not using expected source Validate credentials at app startup Monitor CloudTrail for unexpected IAM ARNs making API calls Monitor CloudTrail Don’t mix IRSA and Pod Identity on the same ServiceAccount Don’t mix IRSA and Pod Identity IRSA vs Pod Identity: Quick Reference Feature IRSA Pod Identity EKS Version Any 1.24+ Setup OIDC provider + annotation Add-on + association Configuration ServiceAccount annotation AWS CLI/API association Token Location /var/run/secrets/eks.../token /var/run/secrets/pods.eks.../token Credential Flow Pod → STS Pod → Agent → EKS Pod Identity Service Performance Good (STS roundtrip) Better (local agent) Cross-Account ✅ Built-in ✅ Native (June 2025) Portability ✅ Any K8s with OIDC ❌ EKS only Complexity Medium Low Best For Multi-cloud, portability needs EKS-native deployments, simplicity Feature IRSA Pod Identity EKS Version Any 1.24+ Setup OIDC provider + annotation Add-on + association Configuration ServiceAccount annotation AWS CLI/API association Token Location /var/run/secrets/eks.../token /var/run/secrets/pods.eks.../token Credential Flow Pod → STS Pod → Agent → EKS Pod Identity Service Performance Good (STS roundtrip) Better (local agent) Cross-Account ✅ Built-in ✅ Native (June 2025) Portability ✅ Any K8s with OIDC ❌ EKS only Complexity Medium Low Best For Multi-cloud, portability needs EKS-native deployments, simplicity Feature IRSA Pod Identity Feature Feature IRSA IRSA Pod Identity Pod Identity EKS Version Any 1.24+ EKS Version EKS Version EKS Version Any Any 1.24+ 1.24+ Setup OIDC provider + annotation Add-on + association Setup Setup Setup OIDC provider + annotation OIDC provider + annotation Add-on + association Add-on + association Configuration ServiceAccount annotation AWS CLI/API association Configuration Configuration Configuration ServiceAccount annotation ServiceAccount annotation AWS CLI/API association AWS CLI/API association Token Location /var/run/secrets/eks.../token /var/run/secrets/pods.eks.../token Token Location Token Location Token Location /var/run/secrets/eks.../token /var/run/secrets/eks.../token /var/run/secrets/eks.../token /var/run/secrets/pods.eks.../token /var/run/secrets/pods.eks.../token /var/run/secrets/pods.eks.../token Credential Flow Pod → STS Pod → Agent → EKS Pod Identity Service Credential Flow Credential Flow Credential Flow Pod → STS Pod → STS Pod → Agent → EKS Pod Identity Service Pod → Agent → EKS Pod Identity Service Performance Good (STS roundtrip) Better (local agent) Performance Performance Performance Good (STS roundtrip) Good (STS roundtrip) Better (local agent) Better (local agent) Cross-Account ✅ Built-in ✅ Native (June 2025) Cross-Account Cross-Account Cross-Account ✅ Built-in ✅ Built-in ✅ Native (June 2025) ✅ Native (June 2025) Portability ✅ Any K8s with OIDC ❌ EKS only Portability Portability Portability ✅ Any K8s with OIDC ✅ Any K8s with OIDC ❌ EKS only ❌ EKS only Complexity Medium Low Complexity Complexity Complexity Medium Medium Low Low Best For Multi-cloud, portability needs EKS-native deployments, simplicity Best For Best For Best For Multi-cloud, portability needs Multi-cloud, portability needs EKS-native deployments, simplicity EKS-native deployments, simplicity The credential chain is powerful but unforgiving. Understanding precedence isn’t just about making things work—it’s about preventing silent security violations that only show up in your audit logs weeks later. With the addition of Pod Identity, you now have more options than ever, but the fundamental principle remains: first match wins, and environment variables always win first. first match wins, and environment variables always win first What’s your biggest credential challenge? Are you using IRSA, Pod Identity, or planning a migration? I’m happy to review specific scenarios in the comments. What’s your biggest credential challenge?