You are certainly familiar with Kubernetes events, especially when you investigate a dysfunction in your cluster using the infamous or the event API resource. It’s a goldmine of information. kubectl describe command 15m Warning FailedCreate replicaset/ml-pipeline-visualizationserver-865c7865bc Error creating: pods "ml-pipeline-visualizationserver-865c7865bc-" is forbidden: error looking up service account default/default-editor: serviceaccount "default-editor" not found $ kubectl get events However useful this information is, it is only temporary. The . You may want to keep this precious information for auditing purposes or ulterior diagnosis in more durable and efficient storage like . You can then react to certain events by having a tool (e.g. ) or your own applications subscribing to a Kafka topic. retention period is generally between 5 minutes and 30 days Kafka Argo Events In this article, I will show you how to build such a pipeline for processing and storing Kubernetes cluster events. What We Want to Build We will build a whole Kubernetes events processing chain. The main components are: from HeptioLab, a Kubernetes event handler that consolidates all the cluster events into various sinks among which a Kafka topic. Eventrouter operator as an easy way to manage Kafka brokers in Kubernetes. Strimzi A custom Go binary to distribute the events in their target Kafka topics. Why do we want to distribute the events into different topics? Let’s say for example that in each namespace of your cluster you have Kubernetes assets related to a specific customer. You clearly want to segregate the customer events from each other before using them. All the configurations, source code, and detailed setup instructions may be found in this GitHub repository Create your Kafka Brokers and Topic I chose to deploy Kafka in Kubernetes with . To keep it short, it’s a Kafka operator for creating and updating Kafka brokers or topics. You can find detailed instructions for installing the operator in the . Strimzi official documentation We first create a new Kafka cluster: apiVersion: kafka.strimzi.io/v1beta1 kind: Kafka metadata: name: kube-events spec: entityOperator: topicOperator: {} userOperator: {} kafka: config: default.replication.factor: 3 log.message.format.version: "2.6" offsets.topic.replication.factor: 3 transaction.state.log.min.isr: 2 transaction.state.log.replication.factor: 3 listeners: - name: plain port: 9092 tls: false type: internal - name: tls port: 9093 tls: true type: internal replicas: 3 storage: type: jbod volumes: - deleteClaim: false id: 0 size: 10 Gi type: persistent-claim version: 2.6 .0 zookeeper: replicas: 3 storage: deleteClaim: false size: 10 Gi type: persistent-claim Then we create the Kafka topic to ingest our events: apiVersion: kafka.strimzi.io/v1beta1 kind: KafkaTopic metadata: name: cluster-events spec: config: retention.ms: 7200000 segment.bytes: 1073741824 partitions: 1 replicas: 1 Setup EventRouter Follow through the official instructions from but a simple kubectl apply should be enough. We need to edit the router configuration to indicate our Kafka endpoint and the topic to use: EventRouter apiVersion: v1 data: config.json: |- { "sink": "kafka" , "kafkaBrokers": "kube-events-kafka-bootstrap.kube-events.svc.cluster.local:9092" , "kafkaTopic": "cluster-events" } kind: ConfigMap metadata: name: eventrouter-cm Verify the Setup is Working Our cluster-events Kafka's topic should now receive all the events. The simplest way to check it is the case is to run a consumer on the topic. To do so, we use for convenience one of our Kafka broker pods which already has all the utilities required. You should see events flowing: kubectl -n kube-events exec kube-events-kafka-0 -- bin/kafka-console-consumer.sh \ --bootstrap-server kube-events-kafka-bootstrap:9092 \ --topic kube-events \ --from-beginning {"verb":"ADDED","event":{...}} {"verb":"ADDED","event":{...}} ... Write the Golang Consumer We now want to distribute our Kubernetes events to multiple topics depending on the namespace they originate from. We will write a Golang consumer and producer to implement this logic: The consumer part of the program listens for incoming cluster event on the cluster-events topic The producer section write into a Kafka topic matching the namespace of the event There’s for Kafka (they are by default) as Kafka will create topics for you implicitly. It’s a really cool feature from the Kafka client API. no explicit need to create the new topics if the proper options are configured Have a look at the full code here: https://github.com/esys/kube-events-kafka/blob/master/events-fanout/cmd/main.go There are of course more performant options, depending on the projected volume of events and complexity of the fanout logic. For a more robust implementation, a consumer making use of would be a great choice. Spark Structured Streaming Deploying the Consumer After building and pushing our binary into a Docker image, we wrap it into a proper Kubernetes deployment: apiVersion: apps/v1 kind: Deployment metadata: labels: app: events-fanout name: events-fanout spec: replicas: 1 selector: matchLabels: app: events-fanout template: metadata: labels: app: events-fanout spec: containers: - image: emmsys/events-fanout:latest name: events-fanout command: [ "./events-fanout" ] args: - -logLevel=info env: - name: ENDPOINT value: kube-events-kafka-bootstrap:9092 - name: TOPIC value: cluster-events Check the Destination Topics are Created By now, new topics should have been created: kubectl -n kube-events get kafkatopics.kafka.strimzi.io -o name kafkatopic.kafka.strimzi.io/cluster-events kafkatopic.kafka.strimzi.io/kube-system kafkatopic.kafka.strimzi.io/default kafkatopic.kafka.strimzi.io/kafka kafkatopic.kafka.strimzi.io/kube-events You will find your events neatly stored into these topics according to their namespace. Takeaway The ability to access historical Kubernetes event logs provides you with a better understanding of the state of your Kubernetes system than kubectl alone would allow you. More importantly, it’s a gateway to automating your cluster or application operations by reacting to events and as such building reliable, reactive, and smart software. Alsopublished behind a paywall at https://codeburst.io/watch-your-kubernetes-cluster-events-with-eventrouter-and-kafka-f1221bea3d5e
