Photo by on Lukas Blazek Unsplash Decentralized systems observability has always been challenging. Dealing with latency, distributed transactions, failures etc. became increasingly complex. The more abstraction a decentralized system has, the more difficult it is to reason about it, debug and troubleshoot. Debugging on K8s is hard The main reason that makes Kubernetes observability so difficult is the volatile and dynamic nature of workloads and resources. Instead of dealing with one server, we now deal with an unknown number of servers (due to autos calling). Rather than having one monolithic application, we now have multiple distributed services. Same with databases, which often reside outside of the cluster. Let’s imagine making an HTTP(s) call to an API running on a Kubernetes cluster hosted on a cloud provider. Here is a simplified sequence diagram showing critical points where At any point of this communication chain, things can go wrong, performance can degrade, security issues might occur, etc. Knowledge of what is happening on the cluster and detailed insights into every step of the communication chain is essential for operational performance. Where to observe Now we know what to observe, but the question is how and where do we put our observability points, gateways of insight. There are a couple of options: Built the observability into service and code. This brings a high degree of control but is cumbersome to maintain and not scalable. Use a sidecar pattern to inject observability logic into every pod. Better, but this can cause performance issues and is difficult to develop, as some different workloads need different metrics which can change over time. Use low-level system calls to monitor the usage of common protocols, stdout and stderr. This means installing something on the cluster itself, which is better, but often means giving this something extended privileges and Inge renting in Linux Kernel. Use low-level system calls with eBPF probes. This brings high extensibility and low overhead. Combine this with exporting more important metrics into Prometheus and you are good to go. Underlying technology Such a granular level of observability is possible thanks to eBPF (Extended Barkley Packet Filter). A protocol that makes the kernel programmable in a safe and performant way. eBPF is a revolutionary technology with origins in the Linux kernel that can run sandboxed programs in an operating system kernel. It is used to safely and efficiently extend the capabilities of the kernel without requiring to change kernel source code or load kernel modules. To learn more about eBPF, visit . Source: Introduction to eBPF https://ebpf.io/what-is-ebpf/ Here is a with Liz Rice explaining eBPF in detail. great video Below diagram shows how eBPF works on a high level Use Cases & Demo Earlier we’ve seen a diagram with an example of traffic flow in Kubernetes. Every step in this traffic should produce valuable insights into our workloads. Here is a list of typical insights both Dev and Ops will be interested in. how do the pods perform what is the latency between different calls what is the HTTP payload how does the cluster behave under load In the demo part, we will be looking into HTTP traffic on a sample app. Pixie CLI comes with predefined demo apps that we can install directly from the command line. However, those demos take a long to load, instead, we will use a different application. Prerequisites To follow along with the demo, you will need to install the following components: docker-engine minikube cluster kubectl helm Installing Pixie Pixie is an open-source observability tool for Kubernetes applications. Pixie uses eBPF to automatically capture telemetry data without the need for manual instrumentation. We will choose the docker option for Pixie CLI to minimize system clutter. alias px="docker run -i --rm -v ${HOME}/.pixie:/root/.pixie pixielabs/px" Installing Minikube Pixie currently only supports minikube, following install instructions are for Debian Linux Other installation instructions available at Minikube Page curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube_latest_amd64.debsudo dpkg -i minikube_latest_amd64.deb Start minikube This will start minikube with a KVM driver. minikube start --driver=kvm2 --cni=flannel --cpus=4 --memory=8000 -p=pixie-cluster If you are running Windows/MacOs use the option --driver=hyperkit Create an account with Pixie Cloud It is possible to self-host Pixie, but for demo purposes, we will create a free account to access metrics UI. px auth login Obtain deploy-key px deploy-key create
export PIXIE_DEPLOY_KEY=<copy key from the command result> Install Pixie on the cluster helm install pixie pixie-operator/pixie-operator-chart --set deployKey=$PIXIE_DEPLOY_KEY --set clusterName=pixie-cluster --namespace pl --create-namespace The installation might take a couple of minutes. Install Kuard is a demo K8s application from the book “Kubernetes Up and Running” Kuard kubectl run --restart=Never --image=gcr.io/kuar-demo/kuard-amd64:blue kuard Once the pod is ready, forward port and access web UI kubectl port-forward kuard 8080:8080
open http://localhost:8080/ Explore Data It is possible to run a query directly from a command line, but we are going to go right into a live UI. Navigate to https://work.withpixie.ai/live/ From the menu select your cluster cluster Click on the script drop-down and select . http/data In the destination filter type to filter only the traffic to our pod. kuard Refresh the kuard page a few times and rerun the script using button on the right-hand side of Pixie UI. RUN Feel free to further explore Pixie UI and find metrics you are interested in. Architectural concerns If you are considering Pixie for your workloads there might be architectural considerations that you want to address. Here are a few facts about how Pixie works that might help address some of them. Data stays on the cluster, you can decide if you want to export anything for example to Prometheus metrics. Data is kept for 24h and the footprint is small enough to work well on edge devices. Queries are extensible and can be written in PiXL language (a derivative of Python). Somewhere on the Roadmap is integration with to enable seamless data export between well-known tools like Prometheus or Jaeger. Open Telemetry Learn more Pixie is a sandbox CNCF project. To learn more about Pixie, check out their with more examples and in depth explanations as well as their . web page Github Repo Also published here.

Flow

Chain

YouTube

How to Use In-Cluster Communication to Secure Kubernetes 

How To Get Better at Kubernetes

Nominated for 2022 - HackerNoon Contributor of the Year - Open Source

Nominated for 2022 - HackerNoon Contributor of the Year - Github

Nominated for 2022 - HackerNoon Contributor of the Year - Docker

Nominated for 2022 - HackerNoon Contributor of the Year - Devops

Nominated for 2022 - HackerNoon Contributor of the Year - Cloud Computing

Nominated for 2022 - HackerNoon Contributor of the Year - Kubernetes

Too Long; Didn't Read

An Essential Guide to Kubernetes Observability Challenges with Pixie

An Essential Guide to Kubernetes Observability Challenges with Pixie

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

GitOps Tutorial: How to Provision an EC2 Instance with Crossplane and Flux

105 Stories To Learn About K8s

10 Ways to Future-Proof Your Business With Cloud

10 Ways to Reduce Data Loss and Potential Downtime Of Your Database

10 Upcoming DevOps Conferences for 2018

101 Stories To Learn About Cloud Infrastructure

GitOps Tutorial: How to Provision an EC2 Instance with Crossplane and Flux

105 Stories To Learn About K8s

10 Ways to Future-Proof Your Business With Cloud

10 Ways to Reduce Data Loss and Potential Downtime Of Your Database

10 Upcoming DevOps Conferences for 2018

101 Stories To Learn About Cloud Infrastructure

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps