Using the Kubernetes Controller for Envoy

In this article, I open a series that will consider the approach to the Envoy configuration with Kubernetes CRD. I will tell about the implementation of the controller for Kubernetes, which watch custom resources in the Kubernetes cluster and performs the Envoy configuration based on this data. This approach is used for example in Istio. After reading the article you will also be able to better understand how it works.

The xDS protocol allows Envoy to dynamically configure proxy rules without rebooting the proxy itself and changing its settings. This gives many advantages in modern multi-component and distributed systems. Services can be created and deleted, become temporarily unavailable, or authentication rules can be changed and much more. All this must be done instantly in the production environment. It is important not to change the application code because the infrastructure can change even faster than the application itself.

Custom Resource Definition

So the first thing we’re going to talk about is the Custom Resource Definition. It’s kind of like a table in a database. You first describe the schematic of this table, and then you manipulate the elements: create, update, delete. Within the Kubernetes cluster, we can also track events with these elements and react to those events.

In my example, I will manage an object that should contain behavior information on the circuit breaker pattern

• Connection timeout
• Number of tries before the error is returned

That is, if you receive a request, the envoy will make three attempts to get a response from the backend before returning the error code. This can be useful when we know in advance that the backend does not respond to an average of every tenth request.

Custom Resource Definition

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: sidecars.proxy.company.com
spec:
  group: proxy.company.com
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                appName:
                  type: string
                cb:
                  type: object
                  properties:
                    timeout:
                      type: integer 
                    tries:
                      type: integer
  scope: Cluster
  names:
    plural: sidecars 
    singular: sidecar 
    kind: Sidecar
    shortNames:
      - sc 

• group: proxy.company.com. Group for the resource type. Must contain the domain.
• name: sidecars.proxy.company.com. Name of a new resource type. Formed by {resource_name}. {domain}
• names. Here I specify all possible names of the resource type. In singular, plural, etc. These names will be used when working with kubectl.

Note that the schematic description takes place inside the versions block. This is important as the schematic is likely to change. So let’s apply the new configuration.

kubectl apply -f crd.yaml
customresourcedefinition.apiextensions.k8s.io/sidecars.proxy.company.com created

And now I can manage objects (Custom resource or CR) with a new type of proxy resource that I defined.

apiVersion: proxy.company.com/v1
kind: Sidecar
metadata:
  name: service1
spec:
  appName: service1
  cb:
    timeout: 5000
    tries: 3

---
apiVersion: proxy.company.com/v1
kind: Sidecar
metadata:
  name: service2
spec:
  appName: service2
  cb:
    timeout: 500
    tries: 1

---
apiVersion: proxy.company.com/v1
kind: Sidecar
metadata:
  name: service3
spec:
  appName: service3
  cb:
    timeout: 3000
    tries: 3

In this example, I want to have three CR records of Sidecar type with different values. The first thing I did was specify which version of the API I wanted to use. ApiVersion: proxy.company.com/v1. The CRD Declaration is an extension to the Kubernetes API, so you must specify the name of the API and course the version. You can go back and look at the line group: proxy.company.com in CRD. In the spec block, I set the field values I want.

kubectl apply -f services.cr.yaml
sidecar.proxy.company.com/service1 created
sidecar.proxy.company.com/service2 created
sidecar.proxy.company.com/service3 created

Three objects were created. And now you can check that they are what we expect.

kubectl get sidecars -o yaml                                                                    1ms
apiVersion: v1
items:
- apiVersion: proxy.company.com/v1
  kind: Sidecar
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"proxy.company.com/v1","kind":"Sidecar","metadata":{"annotations":{},"name":"service1"},"spec":{"appName":"service1","cb":{"timeout":5000,"tries":3}}}
    creationTimestamp: "2022-09-11T12:26:36Z"
    generation: 1
    name: service1
    resourceVersion: "189731"
    uid: 67f03ae5-30f6-42a5-af28-75f041c0c41d
  spec:
    appName: service1
    cb:
      timeout: 5000
      tries: 3
- apiVersion: proxy.company.com/v1
  kind: Sidecar
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"proxy.company.com/v1","kind":"Sidecar","metadata":{"annotations":{},"name":"service2"},"spec":{"appName":"service2","cb":{"timeout":500,"tries":1}}}
    creationTimestamp: "2022-09-11T12:26:36Z"
    generation: 1
    name: service2
    resourceVersion: "189732"
    uid: b874770a-257e-4044-bfb8-7ea291cdfbc8
  spec:
    appName: service2
    cb:
      timeout: 500
      tries: 1
- apiVersion: proxy.company.com/v1
  kind: Sidecar
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"proxy.company.com/v1","kind":"Sidecar","metadata":{"annotations":{},"name":"service3"},"spec":{"appName":"service3","cb":{"timeout":3000,"tries":3}}}
    creationTimestamp: "2022-09-11T12:26:36Z"
    generation: 1
    name: service3
    resourceVersion: "189733"
    uid: 934b06fb-6851-4df6-beb5-b7279d0b59b4
  spec:
    appName: service3
    cb:
      timeout: 3000
      tries: 3
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

I can now manage the CR with the usual commands. For example, I can first delete

kubectl delete sidecar service1                                                                
sidecar.proxy.company.com "service1" deleted
kubectl get sidecars                                                                           
NAME       AGE
service2   5m4s
service3   5m4s

Then apply the declaration again

kubectl apply -f services.cr.yaml                                                               
sidecar.proxy.company.com/service1 created
sidecar.proxy.company.com/service2 unchanged
sidecar.proxy.company.com/service3 unchanged
kubectl get sidecars                                                                           
NAME       AGE
service1   46s
service2   6m49s
service3   6m49s

How Can I Apply it

In the diagram, I showed how the architecture works, which consists of three components.

Sidecar

This is a container in the pod that accepts all requests and proxies to localhost, where the request should already be received directly by the service that runs in production. Sidecar is needed to implement the request processing policy. In this example, it is necessary to implement the pattern of Circuit Breaker, about which I have written above. As a sidecar, I will use Envoy, which can be easily configured via the xDS protocol.

Custom Resources

Kubernetes API. The database in which the settings of sidecars are specified. Yes, you can store this data in any other storage, such as Redis. But why do it when Kubernetes already has distributed storage, based on etcd. In addition, it provides a convenient API for work, as well as the ability to subscribe to resource creation, modification, or removal events.

Controller

This is an application that interacts with Kubernetes API and Sidecar. It may be an application in different programming languages, but in this series, there will be examples on Go, as it is for Go already have well-developed libraries.

Conclusion

In this part, we have looked at the general architecture of the solution, as well as at the Custom Resource Definition. In the next part, I will describe the implementation of the controller. More precisely, the part that can subscribe to events on change custom resources and respond to them.

If you like my articles please encourage me in the Noonies.