In , I explained what CRD is and how it can be useful to solve the problem. In this article, I will show you how you can write a controller that will monitor changes in custom resources. And in the next article, the controller will already begin to respond to these changes and will configure the Envoy. the last article So, what is a controller? The controller is a program that communicates with Kubernetes through its API. When in the last article, I announced a new type of Custom Resources and announced a new API for it: . proxy.company.com/v1 To interact with the Kubernetes API already, have a lot of ready-made clients in different programming languages, but I will create the examples of golang. Especially since, for golang, there is even an "official" . That’s what I plan to use. code generator Entities The first thing I need to do is create entities to which the Sidecar resource data from the API will be mapped. For this, we need two packages: Provides all necessary abstracts and interfaces k8s.io/apimachinery API client for golang k8s.io/client-go Let me remind you that in the last article, I announced a new type of Sidecar resource. Now, I will describe the structure for it in golang using the packages listed above in the file . types.go package v1

import metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

type SidecarSpec struct {
	AppName string `json:"appName"`
	Cb      struct {
		Timeout int `json:"timeout"`
		Tries   int `json:"tries"`
	}
}

// +genclient
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object

type Sidecar struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`

	Spec SidecarSpec `json:"spec"`
}

// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object

type SidecarList struct {
	metav1.TypeMeta `json:",inline"`
	metav1.ListMeta `json:"metadata"`

	Items []Sidecar `json:"items"`
} SidecarSpec matches the description from , recall what it looks like. Custom Resource Definition spec:
  type: object
    properties:
      appName:
        type: string
      cb:
        type: object
        properties:
          timeout:
            type: integer 
          tries:
            type: integer contains additional fields in which various meta-information must be collected. is a structure for a set of structures. For cases when a list of sidecars is requested from the API, his fields for meta-information are slightly different. Sidecar SidecarList Sidecar I will also need the file to register the entity. register.go package v1

import (
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/runtime"
	"k8s.io/apimachinery/pkg/runtime/schema"
)

// SchemeGroupVersion is group version used to register these objects
var SchemeGroupVersion = schema.GroupVersion{
	Group:   "proxy.company.com",
	Version: "v1",
}

// Kind takes an unqualified kind and returns back a Group qualified GroupKind
func Kind(kind string) schema.GroupKind {
	return SchemeGroupVersion.WithKind(kind).GroupKind()
}

// Resource takes an unqualified resource and returns a Group qualified GroupResource
func Resource(resource string) schema.GroupResource {
	return SchemeGroupVersion.WithResource(resource).GroupResource()
}

var (
	// SchemeBuilder initializes a scheme builder
	SchemeBuilder = runtime.NewSchemeBuilder(addKnownTypes)
	// AddToScheme is a global function that registers this API group & version to a scheme
	AddToScheme = SchemeBuilder.AddToScheme
)

// Adds the list of known types to Scheme.
func addKnownTypes(scheme *runtime.Scheme) error {
	scheme.AddKnownTypes(SchemeGroupVersion,
		&Sidecar{},
		&SidecarList{},
	)
	metav1.AddToGroupVersion(scheme, SchemeGroupVersion)
	return nil
} I specified the API name and version. Group:   "proxy.company.com",
Version: "v1", And added two and entities to the known data schemes. It is important to place the file according to the name and version of . Sidecar SidecarList internal/apis/proxy.company.com/v1 Generator The main task of the generator - on the basis of the described entities to generate a code, which in essence can make queries to the Kubernetes API and a mapped response to the described data structures. Generator downloaded as standard. go get k8s.io/code-generator But since the generator will not import into any of the packages, it will not actually be downloaded to the vendor directory. (I prefer to use this directory to work for more convenient debugging). So, I used a to create a file with a generator import. little hack package hack

import _ "k8s.io/code-generator" And now, when doing , the generator is in vendor and I have the ability to use its bash script generate-groups.sh, which has already solved most of the possible needs of developers. I’ll give you the right away. go mod download launch code SCRIPT_ROOT=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )/.." &> /dev/null && pwd )

bash vendor/k8s.io/code-generator/generate-groups.sh \
	"deepcopy,client,informer" \
	github.com/antgubarev/xds-controller/internal/generated \
	github.com/antgubarev/xds-controller/internal/apis \
	proxy.company.com:v1 \
	--output-base=$SCRIPT_ROOT/../../.. \
	--go-header-file=$SCRIPT_ROOT/hack/boilerplate.go.txt Argument assignment The entities I need to generate. You can also simply specify all, and then all possible will be generated. The path by which the generated files will be placed. The path where the data scheme from which the generation takes place. API name and version. It is important to coincide with the way where the structures are placed, as I wrote above. There are also two options in the generator. The root directory into which the generated files will be added. That is to say, the + In my case, this value will be --output-base --output-base github.com/antgubarev/xds-controller/internal/generated GOPATH/src The header file if you want to place any additional information in each generated file. --go-header-file So, let’s run it and look at the result: ./hack/upd.sh                                                               15ms (master*)
Generating deepcopy funcs
Generating clientset for proxy.company.com:v1 at github.com/antgubarev/xds-controller/internal/generated/clientset
Generating listers for proxy.company.com:v1 at github.com/antgubarev/xds-controller/internal/generated/listers
Generating informers for proxy.company.com:v1 at github.com/antgubarev/xds-controller/internal/generated/informers 3 subdirectories appeared in the directory internal/generated Client for API queries clientset A set of components for tracking changes informers Informer is needed and in this article, it is of no additional interest listers This set will be sufficient for the current task so far. Controller Now, I can start tracking changes in Sidecar resources. You can see the full controller code . I’ll describe it in more detail. here import versionedclientset "github.com/antgubarev/xds-controller/internal/generated/clientset/versioned"

//...

cfg, err := clientcmd.BuildConfigFromFlags("", filepath.Join(homedir.HomeDir(), ".kube", "config"))
if err != nil {
	logger.Fatalf("BuildConfigFromFlags: %v", err)
}

_, err = kubernetes.NewForConfig(cfg)
if err != nil {
	logger.Fatalf("Error building kubernetes clientset: %s", err.Error())
}

proxyVersionedClient, err := versionedclientset.NewForConfig(cfg)
if err != nil {
	logger.Fatalf("Error building example clientset: %s", err.Error())
} First, I need to create a client. I used the config for , which is in the home directory, usually, it’s . Next, I used the generated code from . I can already create, delete, and edit Sidecar custom resources using this client. kubectl ~/. kube/config internal/generated/clientset The entire generated code works with the data structures from the ; it has done a lot of work for us. internal/apis generator proxyInformerFactory := proxyinformers.NewSharedInformerFactory(proxyVersionedClient, time.Second*60)

idecarsInformer := proxyInformerFactory.Proxy().V1().Sidecars().Informer()
sidecarsInformer.AddEventHandler(cache.ResourceEventHandlerFuncs{
	AddFunc: func(obj interface{}) {
		sidecar, ok := obj.(*v1.Sidecar)
		if !ok {
			logger.Errorf("Can't parse added object: %v", obj)
		}
		logger.Infof("Sidecar %s is added", sidecar.Name)
	},
	UpdateFunc: func(oldObj interface{}, newObj interface{}) {
		switch newObj.(type) {
		case *v1.Sidecar:
			newSidecar, ok := newObj.(*v1.Sidecar)
			if !ok {
				logger.Errorf("Can't parse updated object: %v", newSidecar)
			}
			if oldObj != newObj {
				logger.Infof("Sidecar %s is updated", newSidecar.Name)
			}
		default:
			logger.Errorf("update event: unknown object %v", newObj)
		}
	},
	DeleteFunc: func(obj interface{}) {
		sidecar, ok := obj.(*v1.Sidecar)
		if !ok {
			logger.Errorf("Can't parse deleted object: %v", obj)
		}
		logger.Infof("Sidecar %s is deleted", sidecar.Name)
	},
}) Informer is also a client, but it is designed to inform the client about new updates. In the code above, I announced the handler for 3 events created a new Sidecar resource AddFunc updated Sidecar range resource UpdateFunc removed Sidecar resource DeleteFunc So far, these features do not do anything useful, only log events in the console and later we will definitely see how it works. stop := make(chan struct{})
defer close(stop)

proxyInformerFactory.Start(stop)
if !cache.WaitForCacheSync(stop, sidecarsInformer.HasSynced) {
	logger.Info("Failed to sync cache")
}

<-stop I called the function which creates a go routine to track events. The client stores a cache with the required objects so as not to query them every time in real-time. He also gave me a channel to get the program done properly. Start To be sure of the integrity and reliability of this cache in the previous example, the interval of the full update was set to . Note that this procedure generates refresh events for those data that are already in the cache (because they will be updated in the cache). time.Second*60 In order to avoid unnecessary events, I perform a check on the functions UpdateFunc if oldObj != newObj {
	logger.Infof("Sidecar %s is updated", newSidecar.Name)
} Now, we can test how it works. First, I run the program and wait for readiness (all connections I executed with locally running minikube). I created a with 3 new resources. manifest apiVersion: proxy.company.com/v1
kind: Sidecar
metadata:
  name: service1
spec:
  appName: service1
  cb:
    timeout: 5000
    tries: 3

---
apiVersion: proxy.company.com/v1
kind: Sidecar
metadata:
  name: service2
spec:
  appName: service2
  cb:
    timeout: 500
    tries: 1

---
apiVersion: proxy.company.com/v1
kind: Sidecar
metadata:
  name: service3
spec:
  appName: service3
  cb:
    timeout: 3000
    tries: 3 Then I applied the manifest (controller is running). kubectl apply -f services.cr.yaml                                        
sidecar.proxy.company.com/service1 created
sidecar.proxy.company.com/service2 created
sidecar.proxy.company.com/service3 created Output INFO[0015] Sidecar service1 is added
INFO[0015] Sidecar service2 is added
INFO[0015] Sidecar service3 is added That’s right. Three resources were created and triggered the add event for each resource created. Now, I have changed the parameters of the variables in the first two resources to any other values and applied the manifest again. kubectl apply -f services.cr.yaml                                           
sidecar.proxy.company.com/service1 configured
sidecar.proxy.company.com/service2 configured
sidecar.proxy.company.com/service3 unchanged Output INFO[0011] Sidecar service1 is updated
INFO[0011] Sidecar service2 is updated Everything is working properly again. And now, I will try to remove one of the resources. kubectl delete sidecar service1                                             
sidecar.proxy.company.com "service1" deleted Output INFO[0014] Sidecar service1 is deleted Conclusion I showed a way to track the changes that occur with custom resources in Kubernetes. In the next article, I will use these events to configure a sidecar based on Envoy via the xDS protocol. You can see the full example in the . repository If you like my articles, please encourage me in the . Noonies Photo by on C Dustin Unsplash

This story contains new, firsthand information uncovered by the writer.

Routine

2022 - HackerNoon Contributor of the Year - Github

Read My Stories

Nominated for 2022 - HackerNoon Contributor of the Year - Github

Too Long; Didn't Read

In your car, at home, or at work — Bosch technology shapes many areas of life.

Using the Kubernetes Controller for Envoy (Part 2)

Too Long; Didn't Read

Companies Mentioned

Anton Gubarev

STORY’S CREDIBILITY

Original Reporting

About Author

TOPICS

THIS ARTICLE WAS FEATURED IN...

Using the Kubernetes Controller for Envoy (Part 2)

Too Long; Didn't Read

Companies Mentioned

Anton Gubarev

STORY’S CREDIBILITY

Original Reporting

About Author

TOPICS

THIS ARTICLE WAS FEATURED IN...

RELATED STORIES