GitLab Runners in EKS with AWS IAM

When we started with Kubernetes, around June 2018, the most popular choice on AWS was kops. But later down the road we migrated to the managed solution provided by AWS: EKS. I think it happened around the time when AWS announced PCI and ISO compliance and it also helped that meanwhile the managed platform was becoming available in more and more regions (because in the summer of 2018 you could use it only in very few locations).

Of course since then more and more features were added, like managed nodes (initially it was only the master which was managed for you) or IAM roles for service accounts.

This part with IAM roles for service accounts I want to touch because it is useful in many scenarios. And some of those scenarios involved CI/CD so I am going to explain how to setup an IAM role for your own hosted Kubernetes GitLab runners. It comes in handy because more things are moving to git as a source of truth.

From infrastructure as code, pipeline as code, gitops it is not just code that resides in git nowadays, but also our configuration, infrastructure, all sorts of pipelines and a lot of other declarative manifests. And this means that your CI/CD pipelines for such repos need to perform a lot of things.

So in our pipelines besides building binaries and pushing them to production, we need to do stuff like read/write to S3 buckets, read some tokens/credentials from AWS Parameter Store, add entries to Route53 and so on. Using GitLab runners we can store some AWS credentials so we cna have access to these APIs in custom environment variables, but thats not a good practice because it is hard to rotate them and you might end up having them in many places. IAM roles that use temporary credentials is the way to go.

And luckily now it is possible to link a GitLab runner deployed in a Kubernetes cluster to an IAM role. GitLab provides a Helm chart for the runner, it can be found here and thanks to this merge request we can now link an IAM role to a gitlab runner installation.

Here is an example of a values.yaml file that can be used for a GitLab runner:

gitlab-runner:
  gitlabUrl: https://gitlab.com
  runnerRegistrationToken: "aaaaaa-bbbbbb" 
  
  unregisterRunners: false

  runners:
    outputLimit: 200000
    privileged: true
    imagePullSecrets: 
    - gitlab-registry-access-secret

    serviceAccountName: <release-name>-gitlab-runner

  rbac:
    create: true
    resources: ["pods", "pods/exec", "secrets"]
    verbs: ["get", "list", "watch", "create", "patch", "delete"]

    serviceAccountAnnotations:
      eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/gitlab-runner-role

One important thing to note is that when rbac is set to true, there is a service account created and its name will be <release-name>-<chart-name> (it is actually the function "gitlab-runner.fullname" from the _helpers.tpl file). How GitLab runner works is that there is a deployment and that will create a runner pod which calls the server for new jobs to execute.

Then these jobs are started in other pods. And these other pods don't use the service account created by the chart, instead it uses default or the value assigned to serviceAccountName from the runners section. So that's the parameter to use to set the service account for the job runners. When you use the same service account, you can also set the role directly in the chart values, you can see it on serviceAccountAnnotations.

Of course you could also use a separate service account, not created within the chart, with its own annotation, so all you need to use is the serviceAccountName parameter under the runners section.