It is not a rare case when an application running on Google Kubernetes Engine ( ) needs to access Amazon Web Services ( ) APIs. Any application has needs. Maybe it needs to run an analytics query on Amazon Redshift, access data stored in Amazon S3 bucket, convert text to speech with Amazon Polly or use any other AWS service. This multi-cloud scenario is common nowadays, as companies are working with multiple cloud providers. GKE AWS Cross-cloud access introduces a new challenge; how to manage cloud credentials, required to access from one cloud provider to services running in the other. The naive approach, distributing and saving cloud provider secrets is not the most secure approach; distributing long-term credentials to each service, that needs to access AWS services, is challenging to manage and a potential security risk. Current Solutions Each cloud provides it’s own unique solution to overcome this challenge, and if you are working with a single cloud provider, it’s more than enough. Google Cloud announced a , the recommended way for GKE applications to authenticate to and consume other Google Cloud services. Workload Identity works by binding Kubernetes service accounts and Cloud IAM service accounts, so you can use Kubernetes-native concepts to define which workloads run as which identities, and permit your workloads to automatically access other Google Cloud services, all without having to manage Kubernetes secrets or IAM service account keys! Read DoiT blog post. Workload Identity Kubernetes GKE Workload Identity Amazon Web Services supports a similar functionality with feature. With IAM roles for service accounts on Amazon EKS clusters, you can associate an IAM role with a Kubernetes service account. This service account can then provide AWS permissions to the containers in any pod that uses that service account. With this feature, you no longer need to provide extended permissions to the worker node IAM role so that pods on that node can call AWS APIs. IAM Roles for Service Accounts But what if you are running your application workload on GKE cluster and would like to access AWS services without compromising on security? Use Case Definition Let’s assume that you already have an AWS account, and a GKE cluster and your company has decided to run a microservice-based application on GKE cluster, but still wants to use resources in the AWS account (Amazon S3 and SNS services) to integrate with other systems deployed on AWS. For example, the (deployed as a Kubernetes Job) is running inside a GKE cluster and needs to upload a data file into a S3 bucket and send a message to an Amazon SNS topic. The equivalent command-line might be: orchestration job aws s3 cp data.csv s3://my-data-bucket/datagram_12345.csv
aws sns publish --topic-arn arn:aws:sns:us-west-2:123456789012:my-data-topic --message "datagram_12345.csv apply geo-filter" Pretty simple example. In order for these commands to succeed, the must have AWS credentials available to it, and those credentials must be able to make the relevant API calls. orchestration job The Naive (and non-secure) Approach: IAM long-term credentials Export AWS Access Key and Secret Key for some AWS IAM User, and inject AWS credentials into the , either as a credentials file or environment variables. Probably not doing this directly, but using resource protected with . orchestration job Kubernetes Secrets RBAC authorization policy The risk here is that these credentials never expire. They have to be transferred somehow from the AWS environment to the GCP environment, and in most cases, people want them to be stored somewhere so that they can be used to re-create the later if required. orchestration job When using long-term AWS credentials, there are multiple ways that your AWS account can be compromised; unintentionally committing AWS credentials into a GitHub repository, keeping them in a Wiki system, reusing credentials for different services and applications, allowing non-restricted access and, . so on While it’s possible to design a proper credentials management solution for issued IAM User credentials, it won’t be required if you will never create these long-term credentials in the first place. The Proposed Approach The basic idea is to assign to GKE Pod, similarly to and cloud-specific features. AWS IAM Role Workload Identity EKS IAM Roles for Service Accounts Luckily for us, AWS allows to create an IAM role for OpenID Connect Federation identity providers instead of IAM users. On the other hand, Google implements OIDC provider and integrates it tightly with GKE through feature. Providing a valid OIDC token to GKE pod, running under Kubernetes Service Account linked to a Google Cloud Service Account. All these may come in handy to implement GKE-to-AWS secure access. OIDC Workload Identity Exchanging OIDC access token to ID token There is one thing missing, required to complete the puzzle. With properly setup GKE Pod gets an OIDC that allows access to Google Cloud services. In order to get temporary AWS credentials from AWS Security Token Service ( ), you need to provide a valid OIDC . Workflow Identity access token STS ID token AWS SDK (and tool) will automatically request temporary AWS credentials from STS service, when the following environment variables are properly setup: aws-cli - the path to the web identity token file (OIDC ID token) AWS_WEB_IDENTITY_TOKEN_FILE - the of the role to assume by Pod containers AWS_ROLE_ARN ARN - the name applied to this assume-role session AWS_ROLE_SESSION_NAME This may sound a bit complex, but I will provide a step-by-step guide and supporting open source project to simplify the setup. dointl/gtoken gtoken-webhook Kubernetes Mutating Admission webhook The is a Kubernetes mutating admission webhook, that mutates any K8s Pod running under specially annotated Kubernetes Service Account (see details below). gtoken-webhook gtoken-webhook mutation flow The injects a into a target Pod and an additional sidekick container (to refresh an OIDC ID token a moment before the expiration), mounts token volume and injects three AWS-specific environment variables. The container generates a valid GCP OIDC ID Token and writes it to the token volume. It also injects required AWS environment variables. gtoken-webhook gtoken initContainer gtoken gtoken The AWS SDK will automatically make the corresponding calls to AWS STS on your behalf. It will handle in-memory caching as well as refreshing credentials as needed. AssumeRoleWithWebIdentity The Configuration Flow Guide Deploy gtoken-webhook To deploy the gtoken-webhook server, we need to create a webhook service and a deployment in our Kubernetes cluster. It’s pretty straightforward except one thing, which is the server’s TLS configuration. If you’d care to examine the file, you’ll find that the certificate and corresponding private key files are read from command line arguments and that the path to these files comes from a volume mount that points to a Kubernetes secret: deployment.yaml [...] args: [...] - --tls-cert-file=/etc/webhook/certs/cert.pem - --tls-private-key-file=/etc/webhook/certs/key.pem volumeMounts: - name: webhook-certs mountPath: /etc/webhook/certs readOnly: true [...] volumes: - name: webhook-certs secret: secretName: gtoken-webhook-certs The most important thing to remember is to set the corresponding CA certificate later in the webhook configuration, so the apiserver will know that it should be accepted. For now, we’ll reuse the script originally written by the Istio team to generate a certificate signing request. Then we’ll send the request to the Kubernetes API, fetch the certificate, and create the required secret from the result. First, run the script and check if the secret holding the certificate and key has been created: webhook-create-signed-cert.sh ./deployment/webhook-create-signed-cert.sh

creating certs in tmpdir /var/folders/vl/gxsw2kf13jsf7s8xrqzcybb00000gp/T/tmp.xsatrckI71
Generating RSA private key, 2048 bit long modulus
.........................+++
....................+++
e is 65537 (0x10001)
certificatesigningrequest.certificates.k8s.io/gtoken-webhook-svc.default created
NAME                         AGE   REQUESTOR              CONDITION
gtoken-webhook-svc.default   1s    alexei@doit-intl.com   Pending
certificatesigningrequest.certificates.k8s.io/gtoken-webhook-svc.default approved
secret/gtoken-webhook-certs configured Once the secret is created, we can create a deployment and service. These are standard Kubernetes deployment and service resources. Up until this point we’ve produced nothing but an HTTP server that’s accepting requests through service on port : 443 kubectl create -f deployment/deployment.yaml

kubectl create -f deployment/service.yaml Configure Mutating Admission webhook Now that our webhook server is running, it can accept requests from the . However, we should create some configuration resources in Kubernetes first. Let’s start with our validating webhook, then we’ll configure the mutating webhook later. If you take a look at the , you’ll notice that it contains a placeholder for : apiserver webhook configuration CA_BUNDLE [...] service: name: gtoken-webhook-svc namespace: default path: "/pods" caBundle: ${CA_BUNDLE} [...] There is a that substitutes the placeholder in the configuration with this CA. Run this command before creating the validating webhook configuration: small script CA_BUNDLE cat ./deployment/mutatingwebhook.yaml | ./deployment/webhook-patch-ca-bundle.sh > ./deployment/mutatingwebhook-bundle.yaml Create a mutating webhook configuration: kubectl create -f deployment/mutatingwebhook-bundle.yaml Configure RBAC for gtoken-webhook Create Kubernetes Service Account to be used with : gtoken-webhook kubectl create -f deployment/service-account.yaml Define RBAC permission for webhook service account: kubectl create -f deployment/clusterrole.yaml kubectl create 0f deployment/clusterrolebinding.yaml # create a cluster role # define a cluster role binding Flow Variables Some of the following variables should be provided by the user, others will be automatically generated and reused in the following steps. - GCP project ID (provided by the user) PROJECT_ID - GKE cluster name (provided by the user) CLUSTER_NAME - Google Cloud Service Account name (provided by the user) GSA_NAME - Google Cloud Service Account unique ID (generated by Google) GSA_ID - Kubernetes Service Account name (provided by the user) KSA_NAME - Kubernetes namespace (provided by the user) KSA_NAMESPACE - AWS IAM role name (provided by the user) AWS_ROLE_NAME - an AWS IAM policy to assign to IAM role (provided by the user) AWS_POLICY_NAME - AWS IAM Role ARN identifier (generated by AWS) AWS_ROLE_ARN Google Cloud: Enable GKE Workload Identity Create a new GKE cluster with enabled: Workload Identity gcloud beta container clusters create ${CLUSTER_NAME} --identity-namespace=${PROJECT_ID}.svc.id.goog or update an existing cluster: gcloud beta container clusters update ${CLUSTER_NAME} --identity-namespace=${PROJECT_ID}.svc.id.goog Google Cloud: Create a Google Cloud Service Account Create a Google Cloud Service Account: gcloud iam service-accounts create GSA_ID=$(gcloud iam service-accounts describe --format json @ .iam.gserviceaccount.com  | jq -r ) # create GCP Service Account ${GSA_NAME} # get GCP SA UID to be used for AWS Role with Google OIDC Web Identity ${GSA_NAME} ${PROJECT_ID} '.uniqueId' Update Google Service Account with following roles: GSA_NAME - impersonate service accounts from GKE Workloads roles/iam.workloadIdentityUser - impersonate service accounts to create OAuth2 access tokens, sign blobs, or sign JWT tokens roles/iam.serviceAccountTokenCreator gcloud iam service-accounts add-iam-policy-binding \
  --role roles/iam.workloadIdentityUser \
  --role roles/iam.serviceAccountTokenCreator \
  --member \ @ .iam.gserviceaccount.com "serviceAccount: .svc.id.goog[ / ]" ${PROJECT_ID} ${K8S_NAMESPACE} ${KSA_NAME} ${GSA_NAME} ${PROJECT_ID} AWS: Create AWS IAM Role with Google OIDC Federation Prepare a role trust policy document for Google OIDC provider: cat > gcp-trust-policy.json << EOF
{ : , : [
    { : , : { : }, : , : { : { : }
      }
    }
  ]
}
EOF "Version" "2012-10-17" "Statement" "Effect" "Allow" "Principal" "Federated" "accounts.google.com" "Action" "sts:AssumeRoleWithWebIdentity" "Condition" "StringEquals" "accounts.google.com:sub" " " ${GSA_SA} Create AWS IAM Role with Google Web Identity: aws iam create-role --role-name --assume-role-policy-document file://gcp-trust-policy.json ${AWS_ROLE_NAME} Assign AWS Role desired policies: aws iam attach-role-policy --role-name --policy-arn arn:aws:iam::aws:policy/ ${AWS_ROLE_NAME} ${AWS_POLICY_NAME} Get AWS Role ARN to be used in K8s SA annotation: AWS_ROLE_ARN=$(aws iam get-role --role-name --query Role.Arn --output text) ${ROLE_NAME} GKE: Create a Kubernetes Service Account Create K8s namespace: kubectl create namespace ${K8S_NAMESPACE} Create K8s Service Account: kubectl create serviceaccount --namespace ${K8S_NAMESPACE} ${KSA_NAME} Annotate K8s Service Account with GKE Workload Identity (GCP Service Account email): kubectl annotate serviceaccount --namespace iam.gke.io/gcp-service-account= @ .iam.gserviceaccount.com ${K8S_NAMESPACE} ${KSA_NAME} ${GSA_NAME} ${PROJECT_ID} Annotate K8s Service Account with AWS Role ARN: kubectl annotate serviceaccount --namespace amazonaws.com/role-arn= ${K8S_NAMESPACE} ${KSA_NAME} ${AWS_ROLE_ARN} Run Demo Run a new K8s Pod with K8s Service Account: ${KSA_NAME} kubectl run -it --rm --generator=run-pod/v1 --image mikesir87/aws-cli --serviceaccount -pod aws sts get-caller-identity { : , : , : } # run a pod (with AWS CLI onboard) in interactive mod ${KSA_NAME} test # in Pod shell: check AWS assumed role # the output should look similar to below "UserId" "AROA9GB4GPRFFXVHNSLCK:gtoken-webhook-gyaashbbeeqhpvfw" "Account" "906385953612" "Arn" "arn:aws:sts::906385953612:assumed-role/bucket-full-gtoken/gtoken-webhook-gyaashbbeeqhpvfw" External References Securely access AWS services from GKE cluster with GitHub: doitintl/gtoken AWS Docs: Creating a Role for Web Identity or OpenID Connect Federation Kubernetes GKE Workload Identity Blog: link Introducing fine-grained IAM roles for service accounts AWS Blog: link AWS Auth using from Google Cloud GitHub project GitHub: Web Identity Federation shrikant0013/gcp-aws-webidentityfederation Using GCP Service Accounts to access AWS IAM Roles by Colin Panisset Blog: blog post Summary I hope, you find this post useful. I look forward to your comments and any questions you have. Want more stories? Check our blog on , or on Twitter. Medium follow Alexei

Flow

Amazon

Fetch

Google

Target

Twitter

Read My Stories

Too Long; Didn't Read

Make resilience your competitive advantage

Access AWS Services from Google Kubernetes Engine Securely [A How-To Guide]

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

101 Stories To Learn About Cloud Infrastructure

10 Things in Engineering We Don't Spend Enough Time On

10 Things I Did To Increase CloudTrail Logs Security

10 reasons to give cloud computing a go

10 Lessons from 10 Years of AWS (part 1)

101 Stories To Learn About Cloud Infrastructure

10 Things in Engineering We Don't Spend Enough Time On

10 Things I Did To Increase CloudTrail Logs Security

10 reasons to give cloud computing a go

10 Lessons from 10 Years of AWS (part 1)

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps