Let’s say you want to do more with Kubernetes than run off-the-shelf apps. Perhaps you want to stitch apps together into a bespoke platform. Imagine that when your user clicks a button you want to provision a new database or open up a new public-facing endpoint. If you’re only running off-the-shelf tools and apps, then probably k8s yaml manifests and helm charts have you covered. Interacting with Kubernetes itself is the realm of custom resources and operators. Let’s understand this before getting into a particular custom resource platform-type use case. Custom Resources and Operators Let’s say that what you provide is more of a tool than an app. Maybe you’ve got one department running one instance of the tool and you want more departments to automatically provision their own instances. Databases, monitoring tools, and storage tools might all be cases like this. A common pattern is to extend the Kubernetes API by creating your own Custom Resource Definition. MongoDB of a custom resource to create a MongoDB instance: offers the below example apiVersion: mongodb.com/v1 kind: MongoDB metadata: name: my-standalone spec: version: 4.4 .0 -ent service: my-service opsManager: configMapRef: name: my-project credentials: my-credentials type: Standalone persistent: true This can be created with . The MongoDB operator manages the custom resource and ensures the right kind of MongoDB instance will be created in response. There are many other such operators listed on . kubectl apply operatorhub If you’re looking to create an operator then the best support for doing so is with Golang. The best options are or operator-sdk (which is in the process of refactoring to be ). I’m not going to talk about creating an operator as there’s good material out there - . kubebuilder based on kubebuilder kubebuilder has an official ebook on it What is harder to find info on is projects that interact with multiple custom resources. Imagine you want a platform to provision instances of monitoring tools and also expose new external endpoints for them. Or watch for when new databases are created and automatically add monitoring around them. Then you’re going to have to go beyond kubebuilder’s scaffolding tools or even what the kubebuilder book tells you. Interacting with Multiple CRDs For true operators you want to keep the responsibility isolated to managing a single CRD. But you might need to interact with other CRDs that the operator doesn’t manage. For example, maybe you’re providing a way to provision instances of a content management system and the content management system itself uses mongodb as a dependency. Then you might want your content management operator to submit mongobdb instances to the kubernetes API. Working with multiple custom resources in Golang code can get interesting. One way to work with custom resources in Go is to use a ClientSet. Many projects offer a ClientSet as an easy way to import code to work with the custom resources. For example, the Seldon project provides custom resources to deploy machine learning models to Kubernetes and it currently . I can then import the package, instantiate a ClientSet and use it to list or create SeldonDeployments: exposes a ClientSet ( machinelearningv1 seldonclientset metav1 _ ) clientset *seldonclientset.Clientset {
	clientset, _ = GetSeldonClientSet()
} {
	config, err := clientcmd.BuildConfigFromFlags( , ) err != { , err
	}
	kubeClientset, err := seldonclientset.NewForConfig(config) err != { , err
	} kubeClientset, } { clientset.MachinelearningV1().SeldonDeployments(namespace).List(context.TODO(), metav1.ListOptions{})
} { clientset.MachinelearningV1().SeldonDeployments(namespace).Create(context.TODO(), deployment, metav1.CreateOptions{})
} import "context" "github.com/seldonio/seldon-core/operator/apis/machinelearning.seldon.io/v1" "github.com/seldonio/seldon-core/operator/client/machinelearning.seldon.io/v1/clientset/versioned" "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/client-go/plugin/pkg/client/auth" "k8s.io/client-go/tools/clientcmd" var func init () func GetSeldonClientSet () (*seldonclientset.Clientset, error) "" "" if nil return nil if nil return nil return nil func ListSeldonDeployments (namespace ) string (result *machinelearningv1.SeldonDeploymentList, err error) return func CreateSeldonDeployment (deployment *machinelearningv1.SeldonDeployment, namespace ) string (sdep *machinelearningv1.SeldonDeployment, err error) return Read and update are very similar. But what if you want to track changes to resources of the type? Perhaps you want to know immediately when the resource is ready. That calls for a watch and its implementation is a bit different. Watching a SeldonDeployment is possible by importing: github.com/seldonio/seldon-core/operator/client/machinelearning.seldon.io/v1/informers/externalversions and using an . Simplifying a little, that basically looks like: Informer (
	SeldonVersion SeldonInformers )

seldonFactory := SeldonInformers.NewSharedInformerFactoryWithOptions(s.Settings.SeldonAPI.Clientset, , SeldonInformers.WithNamespace(namespace))
	runSeldonInformer(seldonFactory) {
	seldoninformer := seldonFactory.Machinelearning().V1().SeldonDeployments().Informer()
	stopper := ( {}) (stopper)
	seldoninformer.AddEventHandler(cache.ResourceEventHandlerFuncs{
		AddFunc: {
			d := obj.(*SeldonVersion.SeldonDeployment) },
		DeleteFunc: {
			d := obj.(*SeldonVersion.SeldonDeployment) },
		UpdateFunc: {
			d := newObj.(*SeldonVersion.SeldonDeployment) },
	})

	seldoninformer.Run(stopper)
} import "github.com/seldonio/seldon-core/operator/apis/machinelearning.seldon.io/v1" "github.com/seldonio/seldon-core/operator/client/machinelearning.seldon.io/v1/informers/externalversions" 0 func runSeldonInformer (seldonFactory SeldonInformers.SharedInformerFactory) make chan struct defer close func (obj {}) interface //... func (obj {}) interface //... func (oldObj, newObj {}) interface //... Note here that we have an import called SeldonVersion as well as the SeldonInformers. The SeldonVersion here gives us Go code to represent what a SeldonDeployment custom resource is. It’s the Go type, whereas the ClientSet is the way to interact with instances of the type. There’s also a helpful fake clientset for testing: ( ). github.com/seldonio/seldon-core/operator/client/machinelearning.seldon.io/v1/clientset/versioned/fake All of this effectively comes from kubebuilder as the clientset code being imported was generated by kubebuilder. However, . kubebuilder no longer generates clientsets Interacting with CRDs - The New Way Now the is to use the Kubernetes controller-runtime package for interacting with CRDs. The provides ways to interact with Kubernetes that are a bit more flexible but also more technical than . (FWIW controller-runtime client .) suggestion from kubebuilder controller-runtime package kubernetes client-go uses client-go It is still suggested to import the Go types for representing custom resources. Typically the types are in a package named after the API version for that type. What's different now is using controller-runtime for the interactions instead of a ClientSet. Here is an example for v1 of the SeldonDeployment custom resource with functions to list and create SeldonDeployments. ( SeldonVersion ctrl ) kclient client.Client {
  kclient = GetClient()
} {
  scheme := runtime.NewScheme()
  SeldonVersion.AddToScheme(scheme)
  kubeconfig := ctrl.GetConfigOrDie()
  controllerClient, err := client.New(kubeconfig, client.Options{Scheme: scheme}) err != {
     log.Fatal(err) } controllerClient
} {
  list := &SeldonVersion.SeldonDeploymentList{}
  err = kclient.List(context.TODO(), list, &client.ListOptions{Namespace: namespace}) list, err
} {
  err = kclient.Create(context.TODO(), deployment) deployment, err
} import "context" "log" "github.com/seldonio/seldon-core/operator/apis/machinelearning.seldon.io/v1" "k8s.io/apimachinery/pkg/runtime" "sigs.k8s.io/controller-runtime" "sigs.k8s.io/controller-runtime/pkg/client" var func init () . func GetClient () client Client if nil return nil return func ListSeldonDeployments (namespace ) string (result *SeldonVersion.SeldonDeploymentList, err error) return func CreateSeldonDeployment (deployment *SeldonVersion.SeldonDeployment) (sdep *SeldonVersion.SeldonDeployment, err error) return The controller-runtime client also has a available for test purposes - . There’s a NewFakeClientWithScheme function that can be used to create a fake client that’s the equivalent for unit tests of the real client from GetClient above. fake implementation sigs.k8s.io/controller-runtime/pkg/client/fake A watch without a clientset is quite different from a watch with a clientset. Most notably, the informer and the objects it tells us about are generic that have to be converted to the intended type. needs to be told which type to watch unstructured types ( SeldonVersion corev1 ctrl ) {
  cfg := ctrl.GetConfigOrDie() dc, err := dynamic.NewForConfig(cfg) err != { , err
  } factory := dynamicinformer.NewFilteredDynamicSharedInformerFactory(dc, , corev1.NamespaceAll, ) gvr, _ := schema.ParseResourceArg(resourceType) informer := factory.ForResource(*gvr) informer, } { seldoninformer, _ := GetDynamicInformer( )
  stopper := ( {}) (stopper)
  runSeldonCRDInformer(stopper, seldoninformer.Informer(), namespace)
} {
  handlers := cache.ResourceEventHandlerFuncs{
     AddFunc: {

        d := &SeldonVersion.SeldonDeployment{} err := runtime.DefaultUnstructuredConverter.
           FromUnstructured(obj.(*unstructured.Unstructured).UnstructuredContent(), d) err != {
           fmt.Println( )
           fmt.Print(err) } },
     DeleteFunc: { },
     UpdateFunc: { },
  }
  s.AddEventHandler(handlers)
  s.Run(stopCh)
} import "fmt" "github.com/seldonio/seldon-core/operator/apis/machinelearning.seldon.io/v1" "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured" "k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/client-go/dynamic" "k8s.io/client-go/dynamic/dynamicinformer" "k8s.io/client-go/informers" "k8s.io/client-go/tools/cache" "sigs.k8s.io/controller-runtime" func GetDynamicInformer (resourceType ) string (informers.GenericInformer, error) // Grab a dynamic interface that we can create informers from if nil return nil // Create a factory object that can generate informers for resource types 0 nil // "GroupVersionResource" to say what to watch e.g. "deployments.v1.apps" or "seldondeployments.v1.machinelearning.seldon.io" // Finally, create our informer for deployments! return nil func (s *SeldonDeployServer) seldonCRDWatcher (namespace ) string //dynamic informer needs to be told which type to watch "seldondeployments.v1.machinelearning.seldon.io" make chan struct defer close func runSeldonCRDInformer (stopCh <- {}, s cache.SharedIndexInformer, namespace ) chan struct string func (obj {}) interface // try following https://erwinvaneyk.nl/kubernetes-unstructured-to-typed/ if nil "could not convert obj to SeldonDeployment" return // do what we want with the SeldonDeployment/event func (obj {}) interface // convert the obj as above do what we want with the SeldonDeployment/event func (oldObj, newObj {}) interface // convert the obj as above do what we want with the SeldonDeployment/event Handling Go Dependencies One upside of the controller-runtime approach to handling CRDs is it gives you more flexibility when importing Go libraries. If you don't use a ClientSet then you can import the types for a CRD with minimal coupling to a k8s version. ClientSets are generated to use a particular version of client-go and that can be subject to breaking changes across versions, such as when the . Importing multiple ClientSets can lead to dependency clashes. This can be avoided with the controller-runtime approach. context parameter was introduced on many key functions Summary Using Go gives us a lot of flexibility for interacting with Kubernetes. We can manage custom resources and interact with custom resources we don’t manage. We could even interact with custom resources . that we know some properties of but don’t know their full definitions Here we’ve only covered cases where we know the full type as that’s more common but there are for cases where you don’t know the types either. The range of options can be a little confusing. Hopefully, this article helps with navigating the choices. articles out there (Title image Image from Binoculars Water View by kisistvan77 on Pixabay .)

2021 - HackerNoon Contributor of the Year - DATA

2022 - HackerNoon Contributor of the Year - Agile

2022 - HackerNoon Contributor of the Year - Agile Software Development

2022 - HackerNoon Contributor of the Year - Digital Transformation

2022 - HackerNoon Contributor of the Year - Innovation

2022 - HackerNoon Contributor of the Year - Kubernetes

2022 - HackerNoon Contributor of the Year - Machine Learning

2022 - HackerNoon Contributor of the Year - Monitoring

2022 - HackerNoon Contributor of the Year - Product Management

2021 - HackerNoon Contributor of the Year - SCRUM

Nominated for 2022 - HackerNoon Contributor of the Year - Innovation

Nominated for 2022 - HackerNoon Contributor of the Year - Scrum

Nominated for 2022 - HackerNoon Contributor of the Year - Agile

Nominated for 2022 - HackerNoon Contributor of the Year - Digital Transformation

Nominated for 2022 - HackerNoon Contributor of the Year - Kubernetes

Nominated for 2022 - HackerNoon Contributor of the Year - Machine Learning

Nominated for 2022 - HackerNoon Contributor of the Year - Agile Software Development

Nominated for 2022 - HackerNoon Contributor of the Year - Product Management

Too Long; Didn't Read

Platforms on k8s with Golang - Watch any CRD

Platforms on k8s with Golang - Watch any CRD

Too Long; Didn't Read

Company Mentioned

@ryandawsonuk

RELATED STORIES