When it comes to managing access control within AWS's Elastic Kubernetes Service (EKS), IAM Roles for Service Accounts (IRSA) plays a crucial role. In this post, we will not only understand what IRSA is and how it operates but also walk through practical examples to demonstrate its implementation. As a bonus we will also look at the example where we setup the IRSA cross AWS Accounts i.e to talk to AWS Services in Account B from the Pod Running in EKS Cluster in Account A. Prerequisites The following are the prerequisites for this post to follow along - : We have explained in detail in the linked blog. Please follow along and deploy the EKS Cluster, you can also directly use this to deploy the EKS Cluster. EKS Cluster with OIDC Provider How to create an EKS Cluster using Terraform README to retrieve EKS Cluster Kubeconfig AWS CLI Install compatible with the EKS version you are installing. kubectl Basic understanding of and AWS IAM EKS Most importantly you need to have basic understanding of Kubernetes What is IRSA? is an EKS feature that permits you to associate an role with a Kubernetes service account. The Kubernetes service account then equips the pods that utilize it with AWS permissions. As such, these pods can access AWS services securely. IAM Roles for Service Accounts (IRSA) AWS Identity and Access Management (IAM) Before IRSA, developers were compelled to directly store AWS credentials in the application or use environment variables linked to Kubernetes Secrets. Both these methods could potentially lead to security risks. IRSA, however, allows developers to adopt the security best practice of granting . least privilege access How Does IRSA Work? Here is a brief workflow of how IRSA operates: An IAM Role gets associated with a Kubernetes Service Account via an IAM OIDC provider that the EKS cluster trusts. When your application intends to make an AWS API call, it leverages an AWS SDK. The SDK identifies the OIDC token and calls the AWS STS operation with this token. AssumeRoleWithWebIdentity AWS STS responds with temporary role credentials, which the SDK uses to call the AWS service. If the IAM Role has the necessary permissions, the AWS service processes the request. This entire process ensures that your application does not have to deal with AWS credentials directly, and no IAM roles are shared among applications. Implementing IRSA within same AWS Account Let's now look at an example where we set up an IRSA in an AWS EKS environment such that the pods can talk to AWS Services like S3 within the same AWS Account. Step 0: Create EKS Cluster with OIDC Provider Assuming you have followed our earlier blog on and have the EKS Cluster up and running. How to create an EKS Cluster using Terraform Step 1: Retrieve OIDC Provider ID from your EKS Cluster Firstly, retrieve the OIDC provider ID from your EKS cluster. This can be achieved using the AWS CLI command: aws eks describe-cluster --name mgmt --region us-east-1 --query "cluster.identity.oidc.issuer" | awk -F'/' '{print $NF}' | tr -d '"' This command will retrieve the OIDC issuer ID, which is required in the steps below to configure the IAM Role. Step 2: Create an IAM Role and Trust Relationship Create an IAM Role using the following AWS CLI command: aws iam create-role --role-name <IAM_ROLE_NAME> --assume-role-policy-document file://trust.json In the above command, is the trust relationship policy document that contains the OIDC provider as a trusted entity. It may look like this: trust.json {
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::<AWS_ACCOUNT_ID>:oidc-provider/oidc.eks.<REGION>.amazonaws.com/id/<EKS_OIDC_ID>"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.<REGION>.amazonaws.com/id/<EKS_OIDC_ID>:sub": "system:serviceaccount:<NAMESPACE>:<SERVICE_ACCOUNT_NAME>"
        }
      }
    }
  ]
} Replace with the AWS Account Number where your EKS Cluster is running. AWS_ACCOUNT_ID Replace with AWS region where your EKS Cluster is running. REGION Replace with OIDC Issuer ID retrieved in Step 1 above. EKS_OIDC_ID Replace with namespace where your pod/service account is. NAMESPACE Replace with the name of the service account you want to use associate with a pod. SERVICE_ACCOUNT_NAME Attach permissions to the IAM Role - AmazonS3FullAccess aws iam attach-role-policy --role-name <IAM_ROLE_NAME> --policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess Step 3: Associate the IAM Role with Kubernetes Service Account You can create a new service account or modify an existing one using the command, with the annotation pointing to the IAM role: kubectl eks.amazonaws.com/role-arn kubectl annotate serviceaccount -n <NAMESPACE> <SERVICE_ACCOUNT_NAME> eks.amazonaws.com/role-arn=arn:aws:iam::<AWS_ACCOUNT_ID>:role/<IAM_ROLE_NAME> Make sure the ServiceAccount you create or annotate is the one you configured for in above. Step 2 Step 4: Update Your Application to Use AWS SDK Assuming you already have an application pod configured with AWS SDK. Now you just need to make sure that your Pod or Deployment is using the ServiceAccount configured earlier. We have shown an example of sample Deployment which you can observe must be deployed in the same as your which you configured with IRSA in the steps above. NAMESPACE ServiceAccount apiVersion: apps/v1
kind: Deployment
metadata:
  name: example-deployment
  namespace: <NAMESPACE>
  ...
spec:
  selector:
    ...
  template:
    metadata:
      ...
    spec:
      serviceAccountName: <SERVICE_ACCOUNT_NAME>
      containers:
      - name: example-container
        image: example/image:latest
        ports:
        - containerPort: 8080 Running Example for IRSA within same account Assuming you have the EKS Cluster running and your AWS CLI is configured to talk to the AWS Account where your EKS Cluster is running. If not please follow the our earlier blog on and have the EKS Cluster up and running OR you can also directly use this to deploy the EKS Cluster. How to create an EKS Cluster using Terraform README Retrieve Kubeconfig and configure your terminal to talk to AWS EKS Cluster as follows, this should update the current kubeconfig context to point to the cluster - export EKS_CLUSTER_NAME=<your eks cluster name>
export EKS_AWS_REGION=<aws region where you created eks cluster>

aws eks update-kubeconfig --region ${EKS_AWS_REGION} --name ${EKS_CLUSTER_NAME}

# validate kubecontext as below, should point to your cluster
kubectl config current-context Create a namespace and service account in that namespace named as follows - irsa-test irsa-test # create namespace
kubectl create namespace irsa-test

# create serviceaccount
kubectl create serviceaccount --namespace irsa-test irsa-test Retrieve OIDC Issuer Id from the EKS Cluster export EKS_CLUSTER_NAME=<your eks cluster name>
export EKS_AWS_REGION=<aws region where you created eks cluster>

export OIDC_ISSUER_ID=$(aws eks describe-cluster --name ${EKS_CLUSTER_NAME} --region ${EKS_AWS_REGION} --query "cluster.identity.oidc.issuer" | awk -F'/' '{print $NF}' | tr -d '"') Create IAM Role with attached AWS managed policy and configure for ServiceAccount name in namespace as follows - AmazonS3FullAccess TrustRelationship irsa-test irsa-test export AWS_ACCOUNT_ID="$(aws sts get-caller-identity --query Account --output text)"
export EKS_AWS_REGION="<replace with aws region where you created eks cluster>"
export EKS_OIDC_ID=$(echo $OIDC_ISSUER_ID)
export NAMESPACE="irsa-test"
export SERVICE_ACCOUNT_NAME="irsa-test"

cat > trust.json << EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/oidc.eks.${REGION}.amazonaws.com/id/${EKS_OIDC_ID}"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.${REGION}.amazonaws.com/id/${EKS_OIDC_ID}:sub": "system:serviceaccount:${NAMESPACE}:${SERVICE_ACCOUNT_NAME}"
        }
      }
    }
  ]
}
EOF

# create IAM role and attach trust policy
aws iam create-role --role-name irsa-test --assume-role-policy-document file://trust.json

# remove trust.json file
rm trust.json

# attach AmazonS3FullAccess Permissions policy to the iam role
aws iam attach-role-policy --role-name irsa-test --policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess Annotate ServiceAccount in namespace with IAM Role as follows - irsa-test irsa-test export AWS_ACCOUNT_ID="$(aws sts get-caller-identity --query Account --output text)"
export NAMESPACE="irsa-test"
export SERVICE_ACCOUNT_NAME="irsa-test"

# annotate service account
kubectl annotate serviceaccount --namespace ${NAMESPACE} ${SERVICE_ACCOUNT_NAME} eks.amazonaws.com/role-arn=arn:aws:iam::${AWS_ACCOUNT_ID}:role/irsa-test Deploy a pod using the service account in namespace as follows - irsa-test irsa-test kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  name: irsa-test
  namespace: irsa-test
  labels:
    app: aws-cli
spec:
  selector:
    matchLabels:
      app: aws-cli
  template:
    metadata:
      labels:
        app: aws-cli
    spec:
      serviceAccountName: irsa-test
      containers:
      - name: aws-cli
        image: amazon/aws-cli
        command: [ "/bin/sh", "-c", "--" ]
        args: [ "while true; do sleep 39000; done;" ]
EOF Make sure the pod is running, using irsa-test serviceAccount and irsa-test serviceAccount is annotated with IAM Role you created above as follows - # check pod is running
kubectl get po -n irsa-test irsa-test

# check pod is deployed with irsa-test serviceaccount
kubectl get deploy -n irsa-test irsa-test -o jsonpath='{.spec.template.spec.serviceAccountName}'

# check service account is annotated with IAM Role
kubectl get sa -n irsa-test irsa-test -o jsonpath='{.metadata.annotations}' Exec into the pod deployed above and create an s3 bucket and validate s3 bucket is created successfully as follows, this will prove that your pod is configured for IRSA successfully - # exec into the pod
export POD_NAME=$(kubectl get po -n irsa-test | grep irsa-test | awk -F ' ' '{print $1}')
kubectl exec -it -n irsa-test ${POD_NAME} -- bash

# run following commands inside the pod
export BUCKET_NAME="irsa-test-sample-$(date +%s)"

# create s3 bucket
aws s3api create-bucket --bucket ${BUCKET_NAME} --region us-west-2 --create-bucket-configuration LocationConstraint=us-west-2

# validate s3 bucket is created, there shouldn't be any error message on stdout
aws s3api head-bucket --bucket ${BUCKET_NAME} --region us-west-2

# delete s3 bucket, there shouldn't be any errors on stdout
aws s3 rm s3://${BUCKET_NAME} --region us-west-2 --recursive
aws s3api delete-bucket --bucket ${BUCKET_NAME} --region us-west-2

# validate s3 bucket is deleted, you should see 404 error message on stdout
aws s3api head-bucket --bucket ${BUCKET_NAME} --region us-west-2 Implementing IRSA cross AWS Accounts Let's now look at an example where we set up an IRSA in an AWS EKS environment such that the pods can talk to AWS Services like S3 in different AWS Account than the one where EKS Cluster is running. We already saw how to setup IRSA within the same AWS Account in the section above, this section will take it forward from there. Let's call the AWS Account we tested above as and the Account2 will be another AWS Account we now want to create resources in. Account1 Step 0: AWS CLI is configured to talk to Account 2 Make sure your AWS CLI is now configured to talk to Account 2, this is needed to perform the steps below. Step 1: Create IAM Role In Account 2 with AmazonS3FullAccess and permissions to be Assumed by IAM Role created in Account 1 earlier Create IAM Role in AWS Account 2, make sure - aws iam create-role --role-name <IAM_ROLE_NAME> --assume-role-policy-document file://trust.json In the above command, is the trust relationship policy document that allows IAM Role created in Account 1 in section above to assume IAM Role in Account1. trust.json {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<ACC1_NUMBER>:role/<ACC1_IAM_ROLE_NAME>"
            },
            "Action": "sts:AssumeRole"
        }
    ]
} Attach permissions to the Account2 IAM Role - AmazonS3FullAccess aws iam attach-role-policy --role-name <IAM_ROLE_NAME> --policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess Step 2: Edit TrustRelationship in Account1 IAM Role to AssumeRole of Account2 Make sure your AWS CLI is now configured to talk to Account 1. Add an inline policy with STS permissions in IAM Role created earlier in Account 1 for IRSA such that it can Assume IAM Role created above in Account 2, as follows - {
    "Effect": "Allow",
    "Principal": {
        "AWS": "arn:aws:iam::<ACC2_NUMBER>:role/<ACC2_IAM_ROLE_NAME>"
    },
    "Action": "sts:AssumeRole"
} In summary, in Step1 you allowed Account2's IAM Role to be assumed by Account1's IRSA IAM Role and in Step2 you allowed Account1's IAM Role to have permissions to assume IAM Role in Account2. That's all you need. Now the pod running in your EKS Cluster should be able to assume IAM Role in Account2 and as per permissions it should be able to perform any s3 operations. Please see the running example below to validate the Cross Account IRSA setup. Running Example Cross Account IRSA Assuming you have the setup from the section earlier. If not, please read and follow the section earlier before proceeding with the examples below - Running Example for IRSA within same account Make sure your AWS CLI is now configured to talk to the AWS Account2 where you want your pod running in AWS Account1 to create resources. Create IAM Role with permissions in and to allow this IAM Role to be assumed by IAM Role created earlier as follows - AmazonS3FullAccess Account2 TrustRelationship irsa-test export AWS_ACCOUNT1_ID="AWS Account1 Number"

cat > trust.json << EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::${AWS_ACCOUNT1_ID}$:role/irsa-test"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}
EOF

# create IAM role and attach trust policy
aws iam create-role --role-name irsa-test --assume-role-policy-document file://trust.json

# remove trust.json file
rm trust.json

# attach AmazonS3FullAccess Permissions policy to the iam role
aws iam attach-role-policy --role-name irsa-test --policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess Login to AWS Account1, go to IAM Role and create an policy with permissions to allow Account1 to talk to Account2 as below - irsa-test inline {
	"Effect": "Allow",
	"Principal": {
		"AWS": "arn:aws:iam::<ACC2_NUMBER>:role/irsa-test"
	},
	"Action": "sts:AssumeRole"
} Replace with AWS Account2's number. ACC2_NUMBER Assuming your kubeconfig is still pointing to the EKS Cluster created earlier, also the pod is still running. Make sure to exit from the interactive terminal of pod, if it's still active from earlier session. irsa-test irsa-test Exec into the pod and assume the IAM Role in Account2 and retrieve the credentials, configure AWS CLI to now talk to Account2. Create s3 bucket and validate that it gets created in Account2 as follows - irsa-test # exec into the irsa-test pod
export POD_NAME=$(kubectl get po -n irsa-test | grep irsa-test | awk -F ' ' '{print $1}')
kubectl exec -it -n irsa-test ${POD_NAME} -- bash

# Account2's number
export AWS_ACCOUNT2_NUMBER=<replace with Account2 number>

# retrieve Account2's credentials
aws sts assume-role --role-arn "arn:aws:iam::${AWS_ACCOUNT2_NUMBER}:role/irsa-test" --role-session-name "create-bucket-session"

# copy AccessKeyId, SecretAccessKey and SessionToken from the output and set following environment variables
ASSUME_ROLE_OUTPUT=$(aws sts assume-role --role-arn "arn:aws:iam::${AWS_ACCOUNT2_NUMBER}:role/irsa-test" --role-session-name "create-bucket-session")

export AWS_ACCESS_KEY_ID=$(echo $ASSUME_ROLE_OUTPUT | grep -o '"AccessKeyId": "[^"]*"' | cut -d'"' -f4)
export AWS_SECRET_ACCESS_KEY=$(echo $ASSUME_ROLE_OUTPUT | grep -o '"SecretAccessKey": "[^"]*"' | cut -d'"' -f4)
export AWS_SESSION_TOKEN=$(echo $ASSUME_ROLE_OUTPUT | grep -o '"SessionToken": "[^"]*"' | cut -d'"' -f4)

# set bucket name
BUCKET_NAME="cross-irsa-test-$(date +%s)"

# make sure you are logged into AWS Account2 now
aws sts get-caller-identity --query Account --output text

# create s3 bucket
aws s3api create-bucket --bucket ${BUCKET_NAME} --region us-west-2 --create-bucket-configuration LocationConstraint=us-west-2

# validate s3 bucket is created, there shouldn't be any error message on stdout
aws s3api head-bucket --bucket ${BUCKET_NAME} --region us-west-2

# delete s3 bucket, there shouldn't be any errors on stdout
aws s3 rm s3://${BUCKET_NAME} --region us-west-2 --recursive
aws s3api delete-bucket --bucket ${BUCKET_NAME} --region us-west-2

# validate s3 bucket is deleted, you should see 404 error message on stdout
aws s3api head-bucket --bucket ${BUCKET_NAME} --region us-west-2 This proves that now Pod running inside an EKS Cluster in Account1 can talk AWS Services in Account 2. Cleanup . DO NOT FORGET TO DELETE THE INFRASTRUCTURE YOU ARE CREATING PLEASE FOLLOW THE CLEANUP STEPS TO CLEAN EKS CLUSTER IN THE BLOG LINKED ABOVE OR FOLLOW THIS . README To clean the IAM Role created simply log into the AWS Console of Account1 and Account2 and delete the IAM Roles you created in sections earlier. irsa-test if you don't cleanup the infrastructure you created and don't need anymore, you will have unexpected cloud costs in your AWS bill. NOTE: Conclusion IAM Roles for Service Accounts (IRSA) is a significant advancement for AWS EKS, offering a more secure way to access AWS resources from your applications. By integrating this feature into your applications, you can maintain granular control over access to AWS services and better adhere to the security best practices of least privilege and role segregation. Remember, each new AWS feature such as IRSA offers opportunities to enhance the security and capability of your applications, allowing you to deliver robust, scalable, and secure applications on the AWS platform. Author Notes Feel free to reach out with any concerns or questions you have. I will make every effort to address your inquiries and provide resolutions. Stay tuned for the upcoming blog in this series dedicated to Platformwale (Engineers who work on Infrastructure Platform teams). Originally at on Aug 2, 2023. published https://platformwale.blog

The code in this story is for educational purposes. The readers are solely responsible for whatever they build with it.

Walkthroughs, tutorials, guides, and tips. This story will teach you how to do something new or how to do something better.

IAM Roles for Service Accounts (IRSA) in AWS EKS Within and Cross AWS Accounts

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

Creating a Google Kubernetes Engine (GKE) Cluster with Terraform in a Custom VPC

101 Stories To Learn About Cloud Infrastructure

10 Things in Engineering We Don't Spend Enough Time On

10 Things I Did To Increase CloudTrail Logs Security

10 reasons to give cloud computing a go

Creating a Google Kubernetes Engine (GKE) Cluster with Terraform in a Custom VPC

101 Stories To Learn About Cloud Infrastructure

10 Things in Engineering We Don't Spend Enough Time On

10 Things I Did To Increase CloudTrail Logs Security

10 reasons to give cloud computing a go

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps