How to Build a ChatOps-Bot with Slack and Kubernetes

ChatOps allows you to manage your DevOps tasks using chat based interfaces. Learn how to build a simple bot to control a Kubernetes cluster using Slack. In the end you are able to view Kubernetes logs and info using Slack chat messages. No prior knowledge of Kubernetes, or the Slack API is necessary. So let’s get started building by exploring what ChatOps is. What is ChatOps ChatOps is a way to execute DevOps tasks, such as deployments, monitoring and system management using chat messages. For example sending a logs message to a chatbot retrieves the latest log messages. Or a deployment could be triggered from a chat message. This offers a few important advantages: A very human way to manage your infrastructure, by chatting with a bot. is much easier to understand than . @chatbot logs kubectl logs hello-minikube-64b64df8c9-fkb6w -ndefault It can be part of a shared chat, so that people can collaborate and share information. This also offers a record of executed commands and actions. It can help safely overcome network and firewall restrictions to make working from home or on the go possible. A unified interface over DevOps tools, manage Kubernetes and OpenShift with the same interfaceIt can simplify and secure infrastructure tasks, so they can be done by the developers themselves. Setup This article describes building a minimal chatbot using: Minikube as a Kubernetes environment. is labeled as a “Production-Grade Container Orchestration”. Kubernetes allows us to deploy, manage and scale docker images. Kubernetes Minikube is a turnkey solution for running Kubernetes on a dev machine. Slack as a chat server Python to implement the actual chat bot server. Minikube To quickly run Kubernetes on a develop machine, implements a Kubernetes cluster in a single virtual machine image. Detailed installation instructions can be found . Minikube here To install it on my macOs System I used VirtualBox as virtualization driver. VirtualBox can be found . here After installing VirtualBox, Minikube can be installed on macOs using the commands below. It’ll also deploy a sample application. This assumes you have installed. homebrew brew install minikube minikube start - driver=virtualbox kubectl create deployment hello-minikube-image=k8s.gcr.io/echoserver:1.10 # install via Homebrew # start and use Virtualbox # install sample app To verify the installation use: and the result should show the example pod, something like kubectl get all pod/hello-minikube-64b64df8c9-fkb6w. A (incredibly) short introduction to Kubernetes Kubernetes is a software that allows the management of docker images in a cluster. This includes deployment, scaling, managing and monitoring. The base deployment unit is a Pod. Pods can contain multiple docker images or containers. The chatbot we will develop in this article only supports pods with a single image. Kubernetes can be controlled through the kubectl command and other means. Our server will use the following Kubernetes commands : to retrieve the pods for an application in a namespace. kubectl get pods --selector=app={app} --namespace={namespace} : to get the logs for a container in a pod (if only a single image is in a container) kubectl logs {pod} --namespace={namespace} : to describe details about a pod. kubectl describe pod {pod} --namespace={namespace} Slack If you don’t have a Slack account you can get your own workspace at https://slack.com. For this article we’ll create a so called classic app, so we are able to use the API. Real Time Messaging (RTM) A classic app can be created Make sure not to just create a New Slack App, as it does not support Realtime Messaging. here. The app will need the following scopes, bot and chat:write:bot. If you don’t find those scope, you probably created a non-classic app in the last step. We’ll add a description and icon for the app. For my bot I’m using an image from . Wikimedia The last step is to install the app to a workspace and note down the bot token, clicking on “Install App to Your Team”. We’ll allow the App to access our workspace and note down the “Bot User OAuth Access Token”. ChatOps Server The code for the server can be found at . It requires python 3 which can for example on macOs be installed with . https://gitlab.com/alexk/chatops-kubernetes brew install python3 Afterwards download and install the requirements with git git@gitlab.com:alexk/chatops-kubernetes.git chatops-kubernetes/ pip3 install -r requirements.txt clone cd Afterwards set the slack token to be used with SLACK_API_TOKEN= export and start the chat bot server with python3 chatbot.py Implementing the Server As discussed in the the setup section we’ll make use of the Reatltime Messaging functionality in Slack. To use this functionality we have to create a Classic App. Classic Slack App Realtime Messaging Models using WebSockets In the current Slack app model, Slack sends a HTTP Post message to the chatbot server for each chat message or command. Modern Slack App Model using HTTP POSTS’s However in the context of ChatOps the classic apps allow us to connect to Slack using WebSockets. Our chatbot server will use a HTTP GET call to a Slack endpoint. The Slack server will keep the connection open and stream updates to our chatbot. This means we do not need to open an incoming endpoint on our DevOps infrastructure. Instead we’ll use an outgoing connection. Since a ChatOps server will often run with elevated rights, it can be hard to open a port to the outside world. By using the classic apps and the websockets connection, we close another angle of attack for cybercriminals. The Server will support 4 commands to set the application for a user. So we don’t need to provide an application name every time we use another command. An easy way to secure access would be to allow only admin users to execute this command. set-app to get the application for a user get-app and to retrieve the logs and info on the pod for the selected application. logs describe To store the selected application we’ll use an embedded database in the db.py module. sqllite3 The main event loop looks like this: data = payload[ ] web_client = payload[ ] print(payload) is_service = data data[ ] is_service data: channel_id = data[ ] thread_ts = data[ ] user = data[ ] text = data[ ] tokens = text.split() me = tokens[ ] conv = Conversation(web_client, channel_id, user) len(tokens) > : print(tokens) command = tokens[ ] print( + command) command commands: command_func = commands[command] : args = tokens[slice( , len(tokens))] result = command_func(conv, args) result : conv.msg(result) Exception e: conv.msg(str(e)) : web_client.chat_postMessage( conv.msg(welcome.format(user=user, me=me)) ) : conv.msg(welcome.format(user=user, me=me)) @RTMClient.run_on(event="message") # subscribe to 'message' events : def process_command (**payload) 'data' 'web_client' # ignore service messages, like joining a channel 'subtype' in and 'subtype' is not None if not and 'text' in 'channel' 'ts' 'user' 'text' # get data from the event # split it up by space characters 0 # user id of the cht bot # object to track the conversation state if 1 # first token is my userid, second will be the command e.g. logs 1 'received command ' if in # get the actual command executor try 2 # execute the command if is not None # and return the value from the # command back to the user except as else # show welcome message else # show welcome message It is annotated with which will be called by the Python Slack client every time a message is sent in the current chat. @RTMClient.run_on(event=”message”) To make sure that we are not receiving our own messages and no service messages (“… has joined the Conversation”) we use this line of code is_service = ‘subtype’ data data[‘subtype’] in and is not None After getting the message we turn it into tokens and get the actual handler for each command and the individual command will then parse the incoming parameters. E.g. the set-app command will store the application in the db for the user. To achieve this we make use of os.popen(cmd) cmd = print( ) stream = os.popen(cmd) stream.read() : def logs (pod, namespace) f'kubectl logs --namespace= ' {pod} {namespace} f'Executing ' {cmd} return Conclusion Using the Realtime Messaging API supported by the Slack client we were able to build a simple chatbot that executes kubernetes commands. ChatBots can also be built using the modern POST events in Slack. Let me know if you are interested in that API and I’ll add a follow-up article. Next steps to make it functional as a ChatOps bot, is to improve security, by building a authorization model. When orchestrating multiple DevOps tools, it may make sense to implement a DevOps API, that handles the actual orchestration and provides a common interface. The interface could then be used to build a multi-channel DevOps toolset, that could be used for example by a Dash Board and ChatOps.