The Art of the Helm Chart: Patterns from the Official Kubernetes Charts

Helm charts package up applications for installation on Kubernetes Clusters. Installing a helm chart is a bit like running an install wizard. So helm chart developers face some of the same challenges faced by developers producing installers: What assumptions can be made about the environment that the install is running into? Is the application meant to be able to interact with other applications? What configurations need to be made available to the user and how should they be offered to the user? But these questions come with helm-specific twists. To see why, let’s start with a picture of what happens when a user runs a`helm install`. Then we can move on to see how some of the deal with these questions. official Kubernetes charts A Picture of a Helm Install I want to install mysql in my cluster. But I don’t want the mysql version that the stable/mysql chart sets . So I create my own values.yaml file called mysql-values.yaml with just one line: in its values.yaml file in the official charts repo imageTag: “5.7.10” And then I run . Helm generates a unique release name for me (`ignorant-camel`) and mysql is deployed in my cluster. The output of tells me that my chosen has been applied. helm install stable/mysql --values=mysqlvalues.yaml kubectl describe pod ignorant-camel-mysql-5dc6b947b-lf6p8 imageTag Actually I didn’t need to install anything in my cluster to see that my selection for would be applied. I could have run and helm would have simply shown me the content of kubernetes deployment descriptor yaml it generated without installing anything. imageTag helm install stabe/mysql --values=mysqlvalues.yaml --dry-run --debug The process of getting to the generated kubernetes deployment descriptors can be better understood by thinking of the of a helm chart: structure ├── Chart.yaml├── README.md├── templates│ ├── NOTES.txt│ ├── _helpers.tpl│ ├── deployment.yaml│ ├── secrets.yaml│ └── ...more yaml...└── values.yaml When the user runs then the entries from the values.yaml in the chart and the helm release information (such as unique release name) get injected into the templated yaml resource descriptors as the template is evaluated and rendered into pure Kubernetes deployment descriptors. When the user runs with parameters or a values file, then the parameters or values file will be overlaid on the one in the chart. Effectively the in-chart values file sets defaults that can be overridden. helm install stable/mysql helm install stable/mysql So the values.yaml is the main interface between us as chart developers and our users. What we expose in the values.yaml determines what users can and can’t do with our charts. Our charts and our values.yaml files also need to satisfy another kind of scenario. Another chart developer might like our app and decide they want to include it inside a package they want to release. So they add our chart to the requirements.yaml of a new chart they create, along with a bunch of other charts. They override some of our defaults by setting values.yaml. And then they distribute it to users who do with that chart. their helm install So our chart goes into other charts and they feed onto users in a charts food chain. Seeing how to satisfy our consumers in this food chain is one of the key challenges of writing helm charts. The Challenges of Writing Helm Charts Applications often have lots of configuration options and Kubernetes Clusters can be configured in lots of different ways. So when writing a helm chart we naturally face questions like: What if I forget to expose a configuration parameter in the values.yaml? What if my application parses configuration dynamically so that I can’t say upfront what all the possible parameter names will be? How can I allow users to use resources defined not directly in my chart but in a chart in which my chart is used as a subchart? What if another chart developer is using my chart in their own and they need to add significant sections inside a resource that I defined (e.g. add an extra container to a pod or a custom initialization script)? How can I allow my users to expose the app externally in the various ways that they might want? We can learn a lot on how to deal with these kinds of problems from the charts in the . Let’s take a look at some of the patterns that those charts use, so that we can understand how to use the same patterns in our own charts. official helm charts repository Before diving in though it’s worth keeping in mind that the public charts are rather advanced. If you’re worrying about some of the above questions when developing your first helm charts, try to park the more difficult worries to begin with. Start by familiarising yourself with and try to build a chart that works for just the simplest cases first. You can then add later. the helm documentation advanced options So let’s try to get a snapshot of patterns from the official charts. This won’t be comprehensive and I expect new options to become available and new patterns to emerge when adds scripting with Lua. But it will give us look at the current state of the art. helm 3 1 Patterns for Exposing Configuration Parameters Let’s say we’ve defined a Deployment resource in our template and we’ve configured the section to allow some environment variables to be set from the values.yaml: env - name**:** ENV_VAR1value: {{ .Values.var1 }}- name**:** ENV_VAR2value: {{ .Values.var2 }} Now users of our chart can override these values in their values.yaml or with . But what will our users be able to do if we’ve overlooked one? Or even worse, what if our application can dynamically parse configuration options (e.g. it might read and create an internal variable called )? Then there isn’t even a finite set of configuration option names to expose. So how do we let users set the variable names as well as the values? —set var1=foo ENV_VAR1 var1 As the says, we could create a configmap with a range function. A nice example of this is in the stable/unbound chart. It contains a that defines its unbound.conf file. It this file into Pods created by its Deployment. Inside the configmap it has entries : helm charts developer guide configmap mounts such as {{- range .Values.localRecords }}local-data: "{{ .name }} A {{ .ip }}"local-data-ptr: "{{ .ip }} {{ .name }}"{{- end }} And its lets the entries in be set as a list e.g.: values.yaml localRecords localRecords:- name: "fake3.host.net"ip: "10.12.10.10"- name: "fake4.host.net"ip: "10.13.10.10" The applies a similar approach directly to environment variables, defining some explicitly and allowing further variables to be set using an collection: sonarqube chart extraEnv {{- range $key, $value := .Values.extraEnv }}— name: {{ $key }}value: {{ $value }}{{- end }} So that variables would be set in a values.yaml like: extraEnv:ENV_VAR1: var1ENV_VAR2: var2 of the official charts define an but in slightly different ways. The defines it differently, without using range. Instead it takes the values.yaml section and simply puts it in the template: Many extraEnv buildkite chart {{- if .Values.extraEnv }}{{ toYaml .Values.extraEnv | indent 12 }}{{- end }} So this means that instead of setting extraEnv entries in the values.yaml as simple pairs we would also need to name each of the keys ( ) and values ( ) in the pairs like: name value extraEnv:— name: ENV_VAR1value: "var1"— name: ENV_VAR2value: "var2" The does this differently again: keycloak chart {{- with .Values.keycloak.extraEnv }}{{ tpl . $ | indent 12 }}{{- end }} So it treats extraEnv as a string, sends it through the function so that it is evaluated as part of the template and ensures that the correct indenting is applied. This is interesting as it allows values to be set like: tpl extraEnv: |— name: KEYCLOAK_LOGLEVELvalue: DEBUG— name: HOSTNAMEvalue: {{ .Release.Name }}-keycloak Normally it wouldn’t be possible to use a template directives like inside the values.yaml but here we can because the content will go through . This could be a big advantage in cases where we’re including the chart inside a parent chart that we’re developing and we need to refer to a service that is part of the same parent chart. (More on this in the next section.) The disadvantages are that it’s a bit more abstract and that the content in the values.yaml is treated as a string there so formatting issues aren’t necessarily found by your editor. {{ .Release.Name }} tpl 2 Referencing Mutually-deployed Resources Typically resources deployed through helm are prefixed with a release name, so that multiple releases can be installed from the same chart (and in the same namespace) without naming conflicts between the installed resources. This means that template directives need to be used to prefix the resource names when one resource with a chart needs to refer to another, or to resources within the same parent chart. A common case where a mutually-deployed resource needs to be referred to is a database secret. The , for example, includes within it the option to deploy a postgres database. It sets up its indexer component’s Deployment with credentials by pointing to the postgres secret (which itself is part of the chart as a subchart and therefore also prefixed with the same release name): xray chart {{- if .Values.postgresql.enabled }}— name: POSTGRES_USERvalue: {{ .Values.postgresql.postgresUser }}— name: POSTGRESS_PASSWORDvalueFrom:secretKeyRef:name: {{ .Release.Name }}-postgresqlkey: postgres-password— name: POSTGRESS_DBvalue: {{ .Values.postgresql.postgresDatabase }}{{- else }}... Here the xray chart knows which database it needs to connect to since it is itself including the postgres chart. But what if we were developing a chart for an application where the user can choose to supply a database? Then it might be necessary to allow the user to include our chart inside a parent chart alongside a database of their choosing. How would we allow the user to set database configuration then? One option would be the approach we saw already with the keycloak chart. Then the user could then set our chart’s to point to postgres even if we’ve not defined it explicitly in our original chart. The values.yaml would then have an entry like: extraEnv extraEnv extraEnv: |- name: POSTGRES_USERvalue: {{ .Values.postgresql.postgresUser }}— name: POSTGRESS_PASSWORDvalueFrom:secretKeyRef:name: {{ .Release.Name }}-postgresqlkey: postgres-password— name: POSTGRESS_DBvalue: {{ .Values.postgresql.postgresDatabase }} The is there because the section is treated as a string to be passed explicitly to the function. | tpl A similar question emerges if a release name prefix needs to be used within an entry in a file that is loaded into a configmap. A typical way of loading a whole file into a configmap is to use . However, like the values.yaml, the contents of files loaded into this way are then not part of the template and so template directives can’t be used in them. What does allow template directives to be used inside a file is to load the . Then the data section of the configmap would have entries like: .Files.Get file content with .Files.Get and pass that into [tpl](https://stackoverflow.com/questions/47595295/how-do-i-load-multiple-templated-config-files-into-a-helm-chart/52009992#52009992) conf_file1: {{ tpl (.Files.Get "files/conf_file1") . | quote }} Or to load into a Secret we need to encode the content in base64: conf_file1: {{ tpl (.Files.Get "files/conf_file1") . | b64enc | quote }} Instead of just one file can with load a of files into a ConfigMap : set using .Files.Glob {{ (tpl (.Files.Glob "files/*").AsConfig . ) | indent 2 }} Though we can’t quite apply the same approach with as then the content would be encoded before going through . To load a set of files into a secret we could use : AsSecret tpl [Glob](https://github.com/helm/helm/blob/master/docs/chart_template_guide/accessing_files.md#glob-patterns) to find the files and [Get](https://github.com/helm/helm/blob/master/docs/chart_template_guide/accessing_files.md#glob-patterns) to load them {{ range $path, $bytes := .Files.Glob "files/*" }}{{ base $path }}: '{{ tpl ($root.Files.Get $path) . | b64enc }}'{{ end }} 3 Empowering Our Fellow Chart Developers The use of in the keycloak chart suggests a possible pattern for dealing with other kinds of definitions that might need to be injected into a chart. For example, the keycloak chart needs to allow a user to package the keycloak chart inside a parent chart that also supplies a user-defined a json file that the user needs to be able to mount into keycloak Pods. The chart supports this by an : extraEnv exposing extraVolumes {{- with .Values.keycloak.extraVolumes }}{{ tpl . $ | indent 8 }}{{- end }} : And extraVoumeMounts volumeMounts: - name: scripts mountPath: /scripts {{- with .Values.keycloak.extraVolumeMounts }}{{ tpl . $ | indent 12 }}{{- end }} The user point to their secret (which contains the json file) and mount it through the values.yaml: can then extraVolumes: |— name: custom-secretsecret:secretName: custom-secretextraVolumeMounts: | name: custom-secretmountPath: "/realm/"readOnly: true Effectively the configuration of volumes and volumeMounts is externalized to the values.yaml. The chart also applies this pattern to allow users to inject further resources into the template such as initContainers and even adding entire additional containers (or ‘sidecars’). This pattern gives chart users a lot of power and flexibility, especially for other chart developers using our chart. The pattern can be especially powerful when combined with the keycloak chart’s exposing of other parameters, such as a variable that is used within the chart’s initialization script: preStartScript {{- with .Values.keycloak.preStartScript }} echo 'Running custom pre-start script...'{{ . | indent 4 }}{{- end }} The user can inject whatever shell script content they like into in their values.yaml. So this approach makes it possible for users to not just set their own configuration paramters but also load their own files and run their own custom scripts using those files. .Values.keycloak.preStartScript 4 Exposing Externally in Different Ways The starter helm chart generated by includes a Service specification but not an Ingress. Many of the public charts do define an Ingress resource. This needs to be configurable as users might not want to use ingress. So the , like many others, wraps its whole Ingress resource definition with: helm create rabbitmq chart {{- if .Values.ingress.enabled }}...{{-end} This is in line with the for the official charts repo, which recommends that ingress should be disabled by default. review guidelines The rabbitmq chart, for example, needs to provide the option whether to expose it by setting a ingress rule: host-based rules:{{- if .Values.ingress.hostName }} host: {{ .Values.ingress.hostName }}http:{{- else }} http:{{- end }} (Actually the user might want multiple hosts to route to the service so the suggest using a range function for hosts. The rabbitmq chart just offers a single host.) review guidelines The rabbitmq chart also allows for the host to be not set (the else condition above). In that case it’s likely that the user will instead override the path (so that rabbitmq is exposed on a unique route, distinct from other exposed services): - path: {{ default "/" .path }}backend:serviceName: {{ template "rabbitmq.fullname" . }}servicePort: {{ .Values.rabbitmq.managerPort }} It’s also especially important that the user has flexibility to set the annotations on an ingress resource, since these are used to control routing configuration options. The suggest supporting this with : review guidelines toYaml {{- with .Values.ingress.annotations }}annotations:{{ toYaml . | indent 4 }}{{- end }} This allows the user of the chart to set annotations in the values.yaml such as: annotations:kubernetes.io/ingress.class: nginxnginx.ingress.kubernetes.io/rewrite-target: / So the user can set whatever annotations they need to on the ingress through the values.yaml without the chart needing to be aware of what those annotations might be. Just exposing these annotations can amount to giving the power to apply script-like configuration on the ingress routing. For example, with an nginx ingress a user can apply rules to set headers in custom ways through configuration snippets: annotations:kubernetes.io/ingress.class: nginxnginx.ingress.kubernetes.io/rewrite-target: /nginx.ingress.kubernetes.io/configuration-snippet: |more_set_headers 'Access-Control-Allow-Origin: $http_origin'; What We’ve Learned About the Art of the Helm Chart This is just chillin because his charts are so good guy A good helm chart needs to anticipate what level of flexibility is needed by its users. More flexibility tends to mean either more complexity or greater abstraction or both. This can hamper chart readability and put more burden on chart users. The challenge is to choose the tools that best fit with what users need for that particular chart. There are other concerns to balance too that we’ve not touched on, such as testing and security. This has just been a look at a particular slice of the official charts. I’ve tried to focus on patterns that seem particularly helpful to me in making sure that users can do what they need to do with your charts. The official Kubernetes charts have been extremely helpful to me in my experience of working on for the project. Hopefully the explanation in this post can help encourage others to dive into the official repo and take inspiration from its charts. the helm charts Activiti