Using Workload Identity to Handle Keys in Google Kubernetes Engine

Workload identity is a modern way to provision keys for pods running on Google Kubernetes Engine. It allows individual pods to use a service account with a suitable set of permissions, without manually managing Kubernetes secrets. In this article, we will describe Workload identity, compare it to other approaches, and finally show a real world example on how to configure a Kubernetes cluster with Workload identity enabled. Here at , we love new security enhancing measures that helps companies and individuals to protect their data and reduce their cybersecurity risk. In this article, we describe Workload identity, and how you can use it to enhance the security of your cloud deployment. debricked Previous approaches Workload identity is a fairly recent addition to Google Cloud Platform, introduced in the middle of 2019. It tries to remedy the lack of a flexible but secure way to access other Google Cloud Services from within Kubernetes pods. It does so by associating Kubernetes service accounts with Google service accounts. Details can be found in the , as well as in the . Let's briefly discuss some previously used strategies, and their main drawbacks, providing a motivation for the introduction of Workload identity. introductory blog post from GCP official documentation for Workload identity Exported service account keys In this option a Google service account is manually created and granted the required roles for the desired cloud services. To actually use the service account inside the pod, the service account keys are exported as a JSON file. This JSON file must then be provided to the pod in some way, for example through the use of Kubernetes secrets. In this case, the secret is mounted inside the container by Kubernetes, and can be used by applications in the pod. Since the keys must be explicitly handled by the user, they are called user-managed keys. The main drawback of this solution is that the the exported keys have a very long lifetime, by default a lifetime of 10 years. Best practices include periodic rotation of service account keys, which is inconvenient to enforce when the keys are user-managed. Manual key management increases the likelihood of user mistakes, such as accidentally leaving keys on systems where they don’t belong. This is especially troublesome when the keys have a long lifetime. Compute engine service accounts Another option is to modify the service account the individual Compute Engine instances run as. Each Kubernetes node will run on an instance, and each pod can by default use the service account of the node it runs on. By modifying the roles of the instance’s service account, it can be granted permission to access the desired cloud services. Since these accounts are managed by GCP, it means that, e.g., key rotation is handled automatically. The main drawback is that Kubernetes nodes in a cluster can run several different pods, where each pod may require a different set of roles of the service account. Since the service account is connected to the instance, this requires the service account to be granted permission for the sum of all roles required by the different pods. This means that each individual pod may be granted higher privileges than required, which violates the principle of least privilege. Workload identity The idea of Workload identity is to provide construction to solve the drawbacks described above, by: Make the credentials handled by GCP, which provides automatic key rotation without having the users handle the keys manually, as well as preventing accidental exposure of the key by removing the key export step. Make it possible to assign a service account to individual pods, giving each pod only the set of permissions it requires. Each pod retrieves their key from the metadata server, which means that the instance’s service account does not need to be used. Thus, not only the pods run with a minimum required set of roles, the instance itself does as well. In the figure below, we see an overview of what is achieved by Workload identity, where each individual pod can use a dedicated service account with a suitable set of roles. Setting up a simple cluster In this section, we show a brief example of how to setup a small Kubernetes cluster with Google Kubernetes Engine. The service will run a simple Node.js application, which will read and write data to Google Storage, using Workload identity to securely provision the pods with keys. The underlying Compute engine instances will run with a minimal set of privileges. Preliminaries We assume you already have an account on Google Cloud Platform, the Google Cloud SDK installed on your local machine, and credentials for your own personal account available so that you can use the gcloud and gsutil commands. Furthermore, you need a storage bucket that can be used for the example below. If you have not used any of the tools before, we recommend that you familiarize yourself with the before you continue with this example. Google Kubernetes Engine Quickstart All source code for this example can be found in a GitHub repository at . https://github.com/debricked/example-gke-workload-identity Example application Our example application will be a small Node.js application which can list files in the bucket, as well as read and write to static filename. The source code can be found below, or in the GitHub repo for this example. express = ( ) { Storage } = ( ) app = express() storage = Storage() bucketName = process.env.BUCKET_NAME app.get( , (req, res) => { res.send( ) }) app.get( , (req, res) => { [files] = storage.bucket(bucketName).getFiles() res.send({ : files.map( file.name) }) }) app.get( , (req, res) => { storage.bucket(bucketName).file( ).createReadStream() .on( , err => { .error( ) .error(err) res.status( ).send( ) }) .pipe(res) }) app.post( , (req, res) => { file = storage.bucket(bucketName).file( ) req .pipe(file.createWriteStream()) .on( , err => { .error( ) .error(err) res.status( ).send( ) }) .on( , () => { res.sendStatus( ) const require 'express' const require '@google-cloud/storage' const const new const '/' 'This is an example application' '/list' async const await files => file '/test' 'test.txt' 'error' console 'Got error while reading file.' console 500 `Could not read file, got error: ` ${ .stringify(err)} JSON '/test' const 'test.txt' 'error' console 'Got error while writing file.' console 500 `Could not write file, got exception: ` ${ .stringify(err)} JSON 'finish' 204 Example Kubernetes cluster We now want to run this small application within a Kubernetes cluster. We define two objects in two separate files: one for the deployment, and one for the load balancer. Their contents can be seen below, or in the Github repository. An important part to notice is the namespace, which will be used later on when enabling workload identity. First, the deployment file. Note that you will have to modify at least the bucket name to match your own bucket name. If you want to modify the application from the previous section, you also need to modify the image name and push your own image to a container registry. # This deployment describes the pods. apiVersion: extensions/v1beta1 kind: Deployment metadata: name: example-gke-workload-identity namespace: storage-consumer-ns spec: replicas: selector: matchLabels: app: hello template: metadata: labels: app: hello spec: serviceAccountName: storage-consumer-ksa containers: - name: hello-app # TODO: Replace your own image, or use the debricked example. image: debricked/example-gke-workload-identity-app:latest ports: - containerPort: env: - name: PORT value: # TODO: Replace your own bucket name. - name: BUCKET_NAME value: gke-workload-identity-playground-bucket 2 with 8080 "8080" with Then the load balancer service. # The service provides a load-balancer to access the pods. apiVersion: v1 kind: Service metadata: name: hello namespace: storage-consumer-ns spec: type: LoadBalancer selector: app: hello ports: - port: targetPort: 80 8080 Commands to launch cluster with Workload identity enabled We now have all prerequisites required for our small example cluster, and now we actually need to enable workload identity. In the description below, I will describe the different commands, but you can also find them as a shell script in the GitHub repo. Don’t forget to modify the variables at the top of the file to match your environment! First, if you just wish to run the example code as-is, clone the GitHub repo, modify the variables at the top of the file, and: Run create to create the cluster, and ./cluster_workload_identity.sh Run destroy to destroy it when finished. ./cluster_workload_identity.sh However, keep reading to get an explanation of the most important steps of cluster_workload_identity.sh Creating the restricted instance runner account First, we create the service account the instances will run under. This account will have a very limited set of roles, namely: logging.logWriter, monitoring.metricWriter, and monitoring.viewer. The name of the service account is defined in the variable, while contains the fully-qualified identity string. RUNNER_GSA RUNNER_RSA_FULL # Create restricted service account to run cluster nodes under. gcloud iam service-accounts create --display-name= gcloud projects add-iam-policy-binding ${PROJECT} \ --member \ --role roles/logging.logWriter gcloud projects add-iam-policy-binding ${PROJECT} \ --member \ --role roles/monitoring.metricWriter gcloud projects add-iam-policy-binding ${PROJECT} \ --member \ --role roles/monitoring.viewer "${RUNNER_GSA}" "${RUNNER_GSA}" "serviceAccount:${RUNNER_GSA_FULL}" "serviceAccount:${RUNNER_GSA_FULL}" "serviceAccount:${RUNNER_GSA_FULL}" Creating the cluster with Workload identity support To enable Workload identity on the GKE cluster, it needs to be assigned to an This namespace contains the mapping between Google service accounts and Kubernetes service accounts. This assignment is done using the flag during cluster creation. We also ensure that our runner account above is used by the underlying instances of the cluster. identity namespace. --identity-namespace gcloud beta container clusters create \ --enable-ip-alias \ --enable-autoupgrade \ --zone= \ --network= \ --metadata disable-legacy-endpoints= \ --identity-namespace= .svc.id.goog \ --service-account= "${CLUSTER}" "$ZONE" "${NETWORK}" true "$PROJECT" "${RUNNER_GSA_FULL}" Creating the service account for cloud storage access Next, we create the service account that the pods will actually use to access Google Cloud Platform, and give it full access to the specified bucket. It will not have access to other services. # create service account that the pod should use gcloud iam service-accounts create --display-name= # give it admin permissions to storage bucket only gsutil iam ch "$GSA" "${GSA}" this "serviceAccount:${GSA_FULL}:roles/storage.objectAdmin" "gs://${BUCKET}" Create namespace and Kubernetes service accounts We can now move to the Kubernetes part and create a Kubernetes service account and a Kubernetes namespace. We will later connect these with the Google service account from the previous section. # get credentials to cluster gcloud container clusters get-credentials --zone= # create k8s namespace kubectl create namespace # create k8s service account namespace kubectl create serviceaccount --namespace "${CLUSTER}" "$ZONE" "$K8S_NAMESPACE" in "$K8S_NAMESPACE" "$KSA" Connecting the different accounts Finally, we can bind the two different types of service accounts together. This allows the Kubernetes service account to act as the Google service account, thus allowing the pod to access cloud services. # Allow the Kubernetes service account to use the Google service account by creating an Cloud IAM policy # binding between the two. This binding allows the Kubernetes Service account to act the Google service account. gcloud iam service-accounts add-iam-policy-binding \ --role roles/iam.workloadIdentityUser \ --member \ kubectl annotate serviceaccount \ --namespace \ \ as "serviceAccount:${PROJECT}.svc.id.goog[${K8S_NAMESPACE}/${KSA}]" "${GSA_FULL}" "${K8S_NAMESPACE}" "${KSA}" "iam.gke.io/gcp-service-account=${GSA_FULL}" Deployment Finally, we can actually deploy our application using kubectl as below. kubectl apply -f deployment.yaml kubectl apply -f loadbalancer.yaml After a while (up to a minute) we can fetch the external IP of our load balancer using the following command kubectl get services --namespace "${K8S_NAMESPACE}" Trying it out You can now connect to the application either by using the browser, or by using curl. To add some content to test.txt, use, e.g., curl to POST data to store in the file. curl -d 'Some test data' http: // /test Retrieve the data again curl http: // /test List all files in the bucket curl http: // /list Destroying the cluster When finished destroy the cluster and remove the created service accounts to avoid paying for an unused cluster. # to cluster when done gcloud container clusters storage-consumer --zone= # service account, and its assigned roles. gcloud iam service-accounts remove-iam-policy-binding --role roles/iam.workloadIdentityUser --member gsutil iam ch -d gcloud iam service-accounts # deleting runner, and its assigned roles. gcloud projects remove-iam-policy-binding ${PROJECT} --member --role roles/logging.logWriter gcloud projects remove-iam-policy-binding ${PROJECT} --member --role roles/monitoring.metricWriter gcloud projects remove-iam-policy-binding ${PROJECT} --member --role roles/monitoring.viewer gcloud iam service-accounts delete delete "$ZONE" delete "serviceAccount:${PROJECT}.svc.id.goog[${K8S_NAMESPACE}/${KSA}]" "${GSA_FULL}" "serviceAccount:${GSA_FULL}" "gs://${BUCKET}" delete "${GSA_FULL}" "serviceAccount:${RUNNER_GSA_FULL}" "serviceAccount:${RUNNER_GSA_FULL}" "serviceAccount:${RUNNER_GSA_FULL}" delete "${RUNNER_GSA_FULL}" Previously published at https://debricked.com/blog/2020/02/17/using-workload-identity-to-handle-keys-in-google-kubernetes-engine/