Using the Kubernetes Controller for Envoy (Part 2) by@antgubarev

Using the Kubernetes Controller for Envoy (Part 2)

In the last article, I explained what CRD is and how it can be useful to solve the problem. In this article, I will show you how you can write a controller that will monitor changes in custom resources. And in the next article, the controller will already begin to respond to these changes and configure the Envoy.
image
Anton Gubarev HackerNoon profile picture

Anton Gubarev

Software engineer with 11 years of expirience. Focused on fault tolerant, distributed systems, PaaS, golang, HA.

github social iconlinkedin social icon

Credibility

In the last article, I explained what CRD is and how it can be useful to solve the problem. In this article, I will show you how you can write a controller that will monitor changes in custom resources. And in the next article, the controller will already begin to respond to these changes and will configure the Envoy.


So, what is a controller? The controller is a program that communicates with Kubernetes through its API. When in the last article, I announced a new type of Custom Resources and announced a new API for it: proxy.company.com/v1.


To interact with the Kubernetes API already, have a lot of ready-made clients in different programming languages, but I will create the examples of golang. Especially since, for golang, there is even an "official" code generator. That’s what I plan to use.

Entities

The first thing I need to do is create entities to which the Sidecar resource data from the API will be mapped. For this, we need two packages:



Let me remind you that in the last article, I announced a new type of Sidecar resource. Now, I will describe the structure for it in golang using the packages listed above in the file types.go.


package v1

import metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

type SidecarSpec struct {
	AppName string `json:"appName"`
	Cb      struct {
		Timeout int `json:"timeout"`
		Tries   int `json:"tries"`
	}
}

// +genclient
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object

type Sidecar struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`

	Spec SidecarSpec `json:"spec"`
}

// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object

type SidecarList struct {
	metav1.TypeMeta `json:",inline"`
	metav1.ListMeta `json:"metadata"`

	Items []Sidecar `json:"items"`
}


SidecarSpec matches the description from Custom Resource Definition, recall what it looks like.


spec:
  type: object
    properties:
      appName:
        type: string
      cb:
        type: object
        properties:
          timeout:
            type: integer 
          tries:
            type: integer


Sidecar contains additional fields in which various meta-information must be collected. SidecarList is a structure for a set of Sidecar structures. For cases when a list of sidecars is requested from the API, his fields for meta-information are slightly different.


I will also need the register.go file to register the entity.


package v1

import (
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/runtime"
	"k8s.io/apimachinery/pkg/runtime/schema"
)

// SchemeGroupVersion is group version used to register these objects
var SchemeGroupVersion = schema.GroupVersion{
	Group:   "proxy.company.com",
	Version: "v1",
}

// Kind takes an unqualified kind and returns back a Group qualified GroupKind
func Kind(kind string) schema.GroupKind {
	return SchemeGroupVersion.WithKind(kind).GroupKind()
}

// Resource takes an unqualified resource and returns a Group qualified GroupResource
func Resource(resource string) schema.GroupResource {
	return SchemeGroupVersion.WithResource(resource).GroupResource()
}

var (
	// SchemeBuilder initializes a scheme builder
	SchemeBuilder = runtime.NewSchemeBuilder(addKnownTypes)
	// AddToScheme is a global function that registers this API group & version to a scheme
	AddToScheme = SchemeBuilder.AddToScheme
)

// Adds the list of known types to Scheme.
func addKnownTypes(scheme *runtime.Scheme) error {
	scheme.AddKnownTypes(SchemeGroupVersion,
		&Sidecar{},
		&SidecarList{},
	)
	metav1.AddToGroupVersion(scheme, SchemeGroupVersion)
	return nil
}


I specified the API name and version.

Group:   "proxy.company.com",
Version: "v1",


And added two Sidecar and SidecarList entities to the known data schemes. It is important to place the file according to the name and version of internal/apis/proxy.company.com/v1.

Generator

The main task of the generator - on the basis of the described entities to generate a code, which in essence can make queries to the Kubernetes API and a mapped response to the described data structures. Generator downloaded as standard.


go get k8s.io/code-generator


But since the generator will not import into any of the packages, it will not actually be downloaded to the vendor directory. (I prefer to use this directory to work for more convenient debugging). So, I used a little hack to create a file with a generator import.


package hack

import _ "k8s.io/code-generator"


And now, when doing go mod download, the generator is in vendor and I have the ability to use its bash script generate-groups.sh, which has already solved most of the possible needs of developers. I’ll give you the launch code right away.


SCRIPT_ROOT=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )/.." &> /dev/null && pwd )

bash vendor/k8s.io/code-generator/generate-groups.sh \
	"deepcopy,client,informer" \
	github.com/antgubarev/xds-controller/internal/generated \
	github.com/antgubarev/xds-controller/internal/apis \
	proxy.company.com:v1 \
	--output-base=$SCRIPT_ROOT/../../.. \
	--go-header-file=$SCRIPT_ROOT/hack/boilerplate.go.txt


Argument assignment


  • The entities I need to generate. You can also simply specify all, and then all possible will be generated.
  • The path by which the generated files will be placed.
  • The path where the data scheme from which the generation takes place.
  • API name and version. It is important to coincide with the way where the structures are placed, as I wrote above.


There are also two options in the generator.


  • --output-base The root directory into which the generated files will be added. That is to say, the --output-base + github.com/antgubarev/xds-controller/internal/generated In my case, this value will be GOPATH/src
  • --go-header-file The header file if you want to place any additional information in each generated file.


So, let’s run it and look at the result:

./hack/upd.sh                                                               15ms (master*)
Generating deepcopy funcs
Generating clientset for proxy.company.com:v1 at github.com/antgubarev/xds-controller/internal/generated/clientset
Generating listers for proxy.company.com:v1 at github.com/antgubarev/xds-controller/internal/generated/listers
Generating informers for proxy.company.com:v1 at github.com/antgubarev/xds-controller/internal/generated/informers


3 subdirectories appeared in the internal/generated directory


  • clientset Client for API queries
  • informers A set of components for tracking changes

listers Informer is needed and in this article, it is of no additional interest


This set will be sufficient for the current task so far.

Controller

Now, I can start tracking changes in Sidecar resources. You can see the full controller code here. I’ll describe it in more detail.


import versionedclientset "github.com/antgubarev/xds-controller/internal/generated/clientset/versioned"

//...

cfg, err := clientcmd.BuildConfigFromFlags("", filepath.Join(homedir.HomeDir(), ".kube", "config"))
if err != nil {
	logger.Fatalf("BuildConfigFromFlags: %v", err)
}

_, err = kubernetes.NewForConfig(cfg)
if err != nil {
	logger.Fatalf("Error building kubernetes clientset: %s", err.Error())
}

proxyVersionedClient, err := versionedclientset.NewForConfig(cfg)
if err != nil {
	logger.Fatalf("Error building example clientset: %s", err.Error())
}


First, I need to create a client. I used the config for kubectl, which is in the home directory, usually, it’s ~/. kube/config. Next, I used the generated code from internal/generated/clientset. I can already create, delete, and edit Sidecar custom resources using this client.


The entire generated code works with the data structures from the internal/apis generator; it has done a lot of work for us.


proxyInformerFactory := proxyinformers.NewSharedInformerFactory(proxyVersionedClient, time.Second*60)

idecarsInformer := proxyInformerFactory.Proxy().V1().Sidecars().Informer()
sidecarsInformer.AddEventHandler(cache.ResourceEventHandlerFuncs{
	AddFunc: func(obj interface{}) {
		sidecar, ok := obj.(*v1.Sidecar)
		if !ok {
			logger.Errorf("Can't parse added object: %v", obj)
		}
		logger.Infof("Sidecar %s is added", sidecar.Name)
	},
	UpdateFunc: func(oldObj interface{}, newObj interface{}) {
		switch newObj.(type) {
		case *v1.Sidecar:
			newSidecar, ok := newObj.(*v1.Sidecar)
			if !ok {
				logger.Errorf("Can't parse updated object: %v", newSidecar)
			}
			if oldObj != newObj {
				logger.Infof("Sidecar %s is updated", newSidecar.Name)
			}
		default:
			logger.Errorf("update event: unknown object %v", newObj)
		}
	},
	DeleteFunc: func(obj interface{}) {
		sidecar, ok := obj.(*v1.Sidecar)
		if !ok {
			logger.Errorf("Can't parse deleted object: %v", obj)
		}
		logger.Infof("Sidecar %s is deleted", sidecar.Name)
	},
})


Informer is also a client, but it is designed to inform the client about new updates. In the code above, I announced the handler for 3 events


  • AddFunc created a new Sidecar resource
  • UpdateFunc updated Sidecar range resource
  • DeleteFunc removed Sidecar resource


So far, these features do not do anything useful, only log events in the console and later we will definitely see how it works.


stop := make(chan struct{})
defer close(stop)

proxyInformerFactory.Start(stop)
if !cache.WaitForCacheSync(stop, sidecarsInformer.HasSynced) {
	logger.Info("Failed to sync cache")
}

<-stop


I called the Start function which creates a go routine to track events. The client stores a cache with the required objects so as not to query them every time in real-time. He also gave me a channel to get the program done properly.


To be sure of the integrity and reliability of this cache in the previous example, the interval of the full update was set to time.Second*60. Note that this procedure generates refresh events for those data that are already in the cache (because they will be updated in the cache).


In order to avoid unnecessary events, I perform a check on the functions UpdateFunc


if oldObj != newObj {
	logger.Infof("Sidecar %s is updated", newSidecar.Name)
}


Now, we can test how it works. First, I run the program and wait for readiness (all connections I executed with locally running minikube). I created a manifest with 3 new resources.


apiVersion: proxy.company.com/v1
kind: Sidecar
metadata:
  name: service1
spec:
  appName: service1
  cb:
    timeout: 5000
    tries: 3

---
apiVersion: proxy.company.com/v1
kind: Sidecar
metadata:
  name: service2
spec:
  appName: service2
  cb:
    timeout: 500
    tries: 1

---
apiVersion: proxy.company.com/v1
kind: Sidecar
metadata:
  name: service3
spec:
  appName: service3
  cb:
    timeout: 3000
    tries: 3


Then I applied the manifest (controller is running).


kubectl apply -f services.cr.yaml                                        
sidecar.proxy.company.com/service1 created
sidecar.proxy.company.com/service2 created
sidecar.proxy.company.com/service3 created


Output

INFO[0015] Sidecar service1 is added
INFO[0015] Sidecar service2 is added
INFO[0015] Sidecar service3 is added


That’s right. Three resources were created and triggered the add event for each resource created. Now, I have changed the parameters of the variables in the first two resources to any other values and applied the manifest again.


kubectl apply -f services.cr.yaml                                           
sidecar.proxy.company.com/service1 configured
sidecar.proxy.company.com/service2 configured
sidecar.proxy.company.com/service3 unchanged


Output

INFO[0011] Sidecar service1 is updated
INFO[0011] Sidecar service2 is updated


Everything is working properly again. And now, I will try to remove one of the resources.


kubectl delete sidecar service1                                             
sidecar.proxy.company.com "service1" deleted


Output

INFO[0014] Sidecar service1 is deleted

Conclusion

I showed a way to track the changes that occur with custom resources in Kubernetes. In the next article, I will use these events to configure a sidecar based on Envoy via the xDS protocol. You can see the full example in the repository.


If you like my articles, please encourage me in the Noonies.


Photo by C Dustin on Unsplash

react to story with heart
react to story with light
react to story with boat
react to story with money

Related Stories

L O A D I N G
. . . comments & more!