Open-Source Testing Mechanism for Kubernetes Applications

Nowadays, applications are adapting microservices like never before with a focus on making them distributed and cloud-agnostic. This has increased the need for more organized testing.

All staged and non-production features must be rigorously tested to deliver efficient and reliable service. Unfortunately, testing currently happens only at a certain point in the development life cycle. To improve the end user's experience of the services in distributed cloud clusters, one needs a framework to implement unit testing during all phases of the application lifecycle.

The above statement is especially true for developing containerized applications. When distribution and load sharing comes into the picture, DevOps teams have to design and develop strategies and implement the same through configurations.

Without a doubt, Kubernetes is one of the top players in the container orchestration platforms game. Fortune 500 and major IT leaders are leveraging the benefits of Kubernetes in their business use cases.

Many things can go wrong when implementing a complex system (mainly configurations). Two out of ten implementations end up with misconfiguration, and teams have to perform code evaluation and reviews to correct the misconfigurations.

This issue contributes heavily to delayed code delivery with missed SLAs and an impact on deadlines. Let's first understand what methods are available in Kubernetes for testing.

Testing Methods for K8s

Kubernetes is an advanced platform with a diverse range of tools and components. As the platform is evolving, these components are being developed and supported outside Kubernetes.

Usually, multiple services are involved in Kubernetes applications, each running in their container. Therefore, developing and debugging these services on a remote K8s cluster can pose challenges at times and require you to use ssh to debug.

Using ssh in the remote cluster is easy, but running scripts and test cases for testing can be challenging due to many factors, like unformatted output, error handling, documenting the cause and fix, and many more. Kubernetes can be easily tested and debugged in the following ways:**
**

• Locally with Telepresence: Telepresence offers easy custom tools to locally debug services while services are proxying to a remote cluster.
• End-to-End with Kubetest2 & Go: Kubetest2 is the Kubernetes framework for launching and running end-to-end tests. The Kubernetes end-to-end framework supports tests and assertions written in Go Language that rely on Ginkgo for managing tests and Gomega for assertions.
• Third-Party Drivers: Kubernetes supports many third-party drivers that support testing functionalities. One of these third-party drivers is Kubescape.

Open Source Option: Kubescape

__Kubescape __is an open-source tool for K8, providing risk analysis, security compliance, an RBAC visualizer, and image vulnerabilities scanning under one umbrella.

As testing is a crucial part of any development activity, dynamic execution of the test cases can save a lot of time and resources. Kubespace is one such tool that offers exceptional services to get the job done with minimal effort.

Kubescape retrieves the K8 objects of the ARMO API server and scans them by running a set of predefined scripts that meet the best practices recommendations. Then, it compares how well your Kubernetes configuration meets these best practices, recommends solutions based on the misconfigurations, and calculates a risk score instantly by showing risk trends over time.

The tool is flexible enough to print to the console in a friendly manner by default and extracts results in JSON format. Kubescape is cloud-agnostic, and for now, it is available in both AWS and GCP.

Due to its simplicity and ease of use, I will be using GCP to test Kubescape.

Prerequisites

1. Setup the Google Kubernetes Engine (GKE) and cloud shell.

2. Install Kubescape in Google Shell.

3. Setup Google storage and create a bucket.

   
   

Step 1 is straightforward to set up if you follow the GCP documentation.

Let's install Kubescape in our Google Cloud Shell.

<script src="https://gist.github.com/jayachandra27/a7174abd35ae6891791acc39ffa44b29.js"></script>

https://gist.github.com/jayachandra27/a7174abd35ae6891791acc39ffa44b29.js

curl -s https://raw.githubusercontent.com/armosec/kubescape/master/install.sh | /bin/bash

Now, it's time to test our configurations with the tool.

Demo

This demo uses a sample YAML file to scan and perform tests and provide a high-level overview of what is happening under the hood. Without further ado, let's use gsutils and bring our data into the cloud shell for execution.**
**

gsutils cp path_to_your_bucket foldername_in_cloud_storage

our sample YAML file looks like this:

version: "3"

services:

  prometheus:
    image: prom/prometheus:v2.8.1
    volumes:
      - ./prometheus/entrypoint.sh:/bin/prometheus-entrypoint.sh
      - ../prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
    user: root
    entrypoint:
      - /bin/sh
      - /bin/prometheus-entrypoint.sh
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.path=/prometheus
      - --web.console.libraries=/usr/share/prometheus/console_libraries
      - --web.console.templates=/usr/share/prometheus/consoles
    ports: ["9090:9090"]
    networks: ["sandbox"]

  grafana:
    image: grafana/grafana:6.0.2
    volumes: ["../grafana/plugins/cinnamon-prometheus-app:/var/lib/grafana/plugins/cinnamon-prometheus-app"]
    ports: ["3000:3000"]
    networks: ["sandbox"]
    depends_on: ["prometheus"]

networks:

  sandbox:
    driver: bridge

Once the configuration is in place, we can run the Kubescape scan framework against the file to scan and detect misconfigurations for us.

The tool provides a detailed description and overview of what checks have been performed, what was the status of the check, and what needs to be done to prevent or improve the overall configurations.

In the end, we get a detailed, user-friendly output in a tabular format with the status and stats of our checks.

Conclusion

Kubernetes is an advanced and complex orchestration platform with a lot of abstractions. Most testers are not aware of all these abstractions, yet they test the end product/service.

K8s docs provide a comprehensive set of best practices that you can refer to and implement based on your use case. But why reinvent the wheel when you can use the existing one with modern and easy-to-use controls. Kubescape and other third-party connectors aim to bridge this gap for testers by making use of best practices to test against the configurations and provide recommendations to improve the overall quality.

I hope this post was informational and you learned something new. Thanks!

Cover Photo by Susanna Marsiglia on Unsplash