How to Build a ChatOps-Bot with Slack and Kubernetes

ChatOps allows you to manage your DevOps tasks using chat based interfaces. Learn how to build a simple bot to control a Kubernetes cluster using Slack.

In the end you are able to view Kubernetes logs and info using Slack chat messages.

No prior knowledge of Kubernetes, or the Slack API is necessary.

So let’s get started building by exploring what ChatOps is.

What is ChatOps

ChatOps is a way to execute DevOps tasks, such as deployments, monitoring and system management using chat messages. For example sending a logs message to a chatbot retrieves the latest log messages. Or a deployment could be triggered from a chat message.

This offers a few important advantages:

A very human way to manage your infrastructure, by chatting with a bot.
```
@chatbot logs
```
is much easier to understand than
```
kubectl logs hello-minikube-64b64df8c9-fkb6w -ndefault
```
.
It can be part of a shared chat, so that people can collaborate and share information. This also offers a record of executed commands and actions.
It can help safely overcome network and firewall restrictions to make working from home or on the go possible.
A unified interface over DevOps tools, manage Kubernetes and OpenShift with the same interfaceIt can simplify and secure infrastructure tasks, so they can be done by the developers themselves.

Setup

This article describes building a minimal chatbot using:

Minikube as a Kubernetes environment.
Kubernetes is labeled as a “Production-Grade Container Orchestration”. Kubernetes allows us to deploy, manage and scale docker images.
Minikube is a turnkey solution for running Kubernetes on a dev machine.
Slack as a chat server
Python to implement the actual chat bot server.

Minikube

To quickly run Kubernetes on a develop machine, Minikube implements a Kubernetes cluster in a single virtual machine image. Detailed installation instructions can be found here.

To install it on my macOs System I used VirtualBox as virtualization driver. VirtualBox can be found here.

After installing VirtualBox, Minikube can be installed on macOs using the commands below. It’ll also deploy a sample application. This assumes you have homebrew installed.

brew install minikube # install via Homebrew
minikube start - driver=virtualbox # start and use Virtualbox
kubectl create deployment hello-minikube-image=k8s.gcr.io/echoserver:1.10 # install sample app

To verify the installation use:

kubectl get all

and the result should show the example pod, something like

pod/hello-minikube-64b64df8c9-fkb6w.

A (incredibly) short introduction to Kubernetes

Kubernetes is a software that allows the management of docker images in a cluster. This includes deployment, scaling, managing and monitoring. The base deployment unit is a Pod. Pods can contain multiple docker images or containers. The chatbot we will develop in this article only supports pods with a single image. Kubernetes can be controlled through the kubectl command and other means.

Our server will use the following Kubernetes commands

kubectl get pods --selector=app={app} --namespace={namespace}

: to retrieve the pods for an application in a namespace.

kubectl logs {pod} --namespace={namespace}

: to get the logs for a container in a pod (if only a single image is in a container)

kubectl describe pod {pod} --namespace={namespace}

: to describe details about a pod.

Slack

If you don’t have a Slack account you can get your own workspace at https://slack.com.

For this article we’ll create a so called classic app, so we are able to use the Real Time Messaging (RTM) API.

A classic app can be created here. Make sure not to just create a New Slack App, as it does not support Realtime Messaging.

The app will need the following scopes, bot and chat:write:bot. If you don’t find those scope, you probably created a non-classic app in the last step.

We’ll add a description and icon for the app. For my bot I’m using an image from Wikimedia.

The last step is to install the app to a workspace and note down the bot token, clicking on “Install App to Your Team”. We’ll allow the App to access our workspace and note down the “Bot User OAuth Access Token”.

ChatOps Server

The code for the server can be found at https://gitlab.com/alexk/chatops-kubernetes. It requires python 3 which can for example on macOs be installed with brew install python3.

Afterwards download and install the requirements with

git clone [email protected]:alexk/chatops-kubernetes.git
cd chatops-kubernetes/
pip3 install -r requirements.txt

Afterwards set the slack token to be used with

export SLACK_API_TOKEN=<Your Slack token starts with xoxb-…>

and start the chat bot server with

python3 chatbot.py

Implementing the Server

As discussed in the the setup section we’ll make use of the Reatltime Messaging functionality in Slack. To use this functionality we have to create a Classic App.

Classic Slack App Realtime Messaging Models using WebSockets

In the current Slack app model, Slack sends a HTTP Post message to the chatbot server for each chat message or command.

Modern Slack App Model using HTTP POSTS’s

However in the context of ChatOps the classic apps allow us to connect to Slack using WebSockets. Our chatbot server will use a HTTP GET call to a Slack endpoint. The Slack server will keep the connection open and stream updates to our chatbot.

This means we do not need to open an incoming endpoint on our DevOps infrastructure. Instead we’ll use an outgoing connection.

Since a ChatOps server will often run with elevated rights, it can be hard to open a port to the outside world. By using the classic apps and the websockets connection, we close another angle of attack for cybercriminals.

The Server will support 4 commands

set-app

to set the application for a user. So we don’t need to provide an application name every time we use another command. An easy way to secure access would be to allow only admin users to execute this command.

get-app

to get the application for a user

logs

and

describe

to retrieve the logs and info on the pod for the selected application.

To store the selected application we’ll use an embedded sqllite3 database in the db.py module.

The main event loop looks like this:

@RTMClient.run_on(event="message")  # subscribe to 'message' events
def process_command(**payload):
    data = payload['data']
    web_client = payload['web_client']
    print(payload)
    # ignore service messages, like joining a channel
    is_service = 'subtype' in data and data['subtype'] is not None

    if not is_service and 'text' in data:
        channel_id = data['channel']
        thread_ts = data['ts']
        user = data['user']
        text = data['text']  # get data from the event
        tokens = text.split()  # split it up by space characters
        me = tokens[0]  # user id of the cht bot
        # object to track the conversation state
        conv = Conversation(web_client, channel_id, user)
        if len(tokens) > 1:
            print(tokens)
            # first token is my userid, second will be the command e.g. logs
            command = tokens[1]
            print('received command ' + command)
            if command in commands:
                # get the actual command executor
                command_func = commands[command]
                try:
                    args = tokens[slice(2, len(tokens))]
                    # execute the command
                    result = command_func(conv, args)
                    if result is not None:
                        # and return the value from the 
                        # command back to the user
                        conv.msg(result)
                except Exception as e:
                    conv.msg(str(e))

            else:
                # show welcome message
                web_client.chat_postMessage(
                    conv.msg(welcome.format(user=user, me=me))
                )
        else:
            # show welcome message
            conv.msg(welcome.format(user=user, me=me))

It is annotated with

@RTMClient.run_on(event=”message”)

which will be called by the Python Slack client every time a message is sent in the current chat.

To make sure that we are not receiving our own messages and no service messages (“… has joined the Conversation”) we use this line of code

is_service = ‘subtype’ in data and data[‘subtype’] is not None

After getting the message we turn it into tokens and get the actual handler for each command and the individual command will then parse the incoming parameters. E.g. the set-app command will store the application in the db for the user. To achieve this we make use of os.popen(cmd)

def logs(pod, namespace):
    cmd = f'kubectl logs  {pod} --namespace={namespace}'
    print(f'Executing {cmd}')
    stream = os.popen(cmd)
    return stream.read()

Conclusion

Using the Realtime Messaging API supported by the Slack client we were able to build a simple chatbot that executes kubernetes commands. ChatBots can also be built using the modern POST events in Slack. Let me know if you are interested in that API and I’ll add a follow-up article.

Next steps to make it functional as a ChatOps bot, is to improve security, by building a authorization model. When orchestrating multiple DevOps tools, it may make sense to implement a DevOps API, that handles the actual orchestration and provides a common interface. The interface could then be used to build a multi-channel DevOps toolset, that could be used for example by a Dash Board and ChatOps.