Developers typically need to look at performance bottlenecks in production applications to determine the cause of the problem. To do this, you usually need information that can be collected through logs and code tools. Unfortunately, this approach is usually time-consuming and does not provide enough details about the underlying problem. A modern and more advanced approach is to apply and use profiling techniques and tools that highlight the slowest application code, that is, the area consuming most of your resources. In this blog post, we will be talking about continuous profiling, and then instrument a couple of microservices running on Kubernetes using an open-source tool called Pyroscope. What is profiling? Code must be analyzed, debugged, and reviewed to determine the most effective way to make it run faster. Using a profiling tool to examine an application’s code helps us to locate and fix performance bottlenecks. This can quickly diagnose how an application performs and enable programmers to get down to core details of poor performance. The result is a streamlined codebase that decreases CPU/memory consumption and makes the user experience better! Profiling is a program analysis that measures the memory, time complexity of a program or the frequency and duration of function calls. Profiling information serves to aid program optimization and performance. Profiler programs can track each individual line of code. Continuous profiling Continuous Profilers are used to make troubleshooting even faster and easier. Continuous Profilers are production code profilers that allow you to analyze code-level performance across your environment over time. As profiles are collected continuously, they can reveal the most resource-intensive features (or lines of code) quickly after new code is introduced. Optimization can reduce end-user delays and cloud provider accounts. What continuous profilers are out there? So, here’s a list of some of the profilers you may have come across: Pyroscope is an open-source platform, consisting of server and agent. It allows the user to collect, store, and query the profiling data in a CPU and disk-efficient way. Pyroscope Parca collects, stores and makes profiles available to be queried over time. It is open source and can be deployed on production environments as Parca focuses on sampling profiling two main types of profiles: tracing and sampling. Parca Datadog analyzes and compares code performance all the time and in any environment, including production. It pinpoints the hard-to-replicate production issues caused by inefficient code. Also has automated code profiling insights. Datadog Continuous Profiler Google - Cloud Profiler is a statistical, low-overhead profiler that continuously gathers CPU usage and memory-allocation information from your production applications. It has Actionable application profiling, Low-impact production Profilin, and Broad Platform support. Cloud Profiler Why use Pyroscope Before we start exploring Pyroscope, let’s see how it is different from a few of the other continuous profiling tools available in the market. DataDog and Google Cloud Profiler are widely used in the industry. As pointed out by , below are some of the reasons why Pyroscope is better compared to the other two. one of the Reddit user Source: Comparison of Datadog, Google Cloud, and Pyroscope continuous profiling capabilities Pyroscope, focuses on building a that is built specifically for profiling data to both store and query that data as efficiently as possible. It uses an agent-server model to send the profiles from applications to the Pyroscope server: storage engine Source: How does Pyroscope Work? Pyroscope allows for profilers from any language to send data to it and have that data efficiently stored by the storage engine. For example, Pyroscope has language-specific agents for Go, Python, Ruby, eBPF, Java, .NET, PHP, and Rust. Parca on the other hand takes a slightly different approach and relies on eBPF for . At the time of writing this article, support for other languages is work in progress. it can read from any as well. compiled languages like C, C++, Go, etc Similar to Pyroscope pprof formatted profiles from HTTP endpoints Theoretically, since all these languages eventually, compile down and run on the kernel, eBPF should work for any of these languages. However, in practice, if you actually run eBPF for interpreted languages like Python, function names are unreadable for humans in many cases. This is because symbols are not stored in those languages. For this reason, Pyroscope supports both language-specific profilers as well as eBPF profiling. This comes at the expense of slightly more work to integrate language-specific agents as compared to eBPF, which can just run at the kernel level. But it also comes with the benefit of having much more actionable and human-readable profiles. How to install Pyroscope? You can start the server followed by the agent no matter what you use, Docker, Linux, or are looking for Ruby or Go docs, Pyroscope covers you. Even if you aim for ten seconds or ten months of software profiling data, their custom-designed storage engine makes fast queries.— Pyroscope website We will use minikube for running a Kubernetes cluster. Create a cluster using minikube: minikube start Add the Helm chart repo: helm repo add pyroscope-io https://pyroscope-io.github.io/helm-chart Install Helm chart: helm install pyroscope pyroscope-io/pyroscope --set service.type=NodePort To check Pyroscope Helm chart installed successfully: helm list Check if Pyroscope is running: kubectl get all Now we have Pyroscope running in our Kubernetes cluster, we will proceed with the steps of using the application with it. We will be using , for this demo. Google microservices Integrating Google microservices demo with Pyroscope We will modify our container images to use pyroscope binary. This binary will start our application and inject itself to monitor. You can refer to more . in this Pyroscope document We will be working on Python, Go, and .NET microservice from Google microservices for the demo. All the modifications are pushed to on GitHub, let's take a look at those changes per serivce. the fork of Google microservices : To try out Pyroscope with Google microservices demo, you don't need to build the Docker images yourself. You can just apply the Kubernetes manifest as shown by the section. Note Obtaining the profiling data from microservice Python We will use the Email Service application which is written in Python. Following changes in the are required, for using a Python application with Pyroscope. Dockerfile COPY --from=pyroscope/pyroscope:latest /usr/bin/pyroscope /usr/bin/pyroscope CMD [ "pyroscope", "exec", "python", "email_server.py" ] After editing the Dockerfile, under the same folder, we proceeded with building and pushing the image. docker build . -t beellzrocks/emailservice:latest docker push beellzrocks/emailservice:latest .NET We will use the application Cart Service for .NET. For using the .NET application with Pyroscope, the following changes in Dockerfile are required. COPY --from=pyroscope/pyroscope:latest /usr/bin/pyroscope /usr/bin/pyroscope ENTRYPOINT ["pyroscope", "exec", "-spy-name", "dotnetspy", "/app/cartservice"] After editing the Dockerfile, we proceeded with building and pushing the image. Go We will take the Product Catalog Service application which is written in Go. For using the Go application with Pyroscope following changes in are required. server.go import ( pyroscope "github.com/pyroscope-io/pyroscope/pkg/agent/profiler" ) func main() { pyroscope.Start(pyroscope.Config{ ApplicationName: os.Getenv("APPLICATION_NAME"), ServerAddress: os.Getenv("SERVER_ADDRESS"), }) // code here ) After editing the server.go, we proceeded with building and pushing the image. Obtaining the profiling data from microservice We modified the Kubernetes manifests to use our images with Pyroscope. The file contains resources for all the applications. We edited it to use the images which we built in above steps i.e. emailservice, cartservice, productcatalogservice. kubernetes-manifests.yaml containers: - name: server image: beellzrocks/emailservice When running Pyroscope in Kubernetes, we need to do following changes: Add capability. SYS_PTRACE Tell the agent about the location of Pyroscope server, and the application name using environment variables. containers: - name: server env: - name: PYROSCOPE_SERVER_ADDRESS # To change Pyroscope Server Port change the value value: "http://pyroscope:4040" - name: PYROSCOPE_APPLICATION_NAME # Application name shown in the UI value: "email.service" securityContext: capabilities: add: - SYS_PTRACE Now, to deploy all the services, you can apply the Kubernetes manifest to your cluster. kubectl apply -f https://raw.githubusercontent.com/infracloudio/microservices-demo-dev/master/release/kubernetes-manifests.yaml Get the service url for Pyroscope: minikube service pyroscope |-----------|-----------|-------------|---------------------------| | NAMESPACE | NAME | TARGET PORT | URL | |-----------|-----------|-------------|---------------------------| | default | pyroscope | http/4040 | http://192.168.49.2:30639 | |-----------|-----------|-------------|---------------------------| 🎉 Opening service default/pyroscope in default browser To access the Pyroscope UI, you can go to the URL: (your's will be different). http://192.168.49.2:30639 Pyroscope UI with Pyroscope server CPU As you can see in the above screenshot, Pyroscope itself takes low CPU usage while storing the data locally. It uses the Badger database to store data locally. Pyroscope resource utilization Monitoring Kubernetes pods is also important in the context of resource usage, utilization, and cost control. Pyroscope uses low resources with low overhead. Pyroscope CPU utilization Monitoring using Pyroscope Pyroscope profiles the code using different agents depending upon the programming language. Here are some examples of the profiled application's flame graph using Pyroscope. Pyroscope with Go Product Catalogue Service application Pyroscope with .Net Cart application Pyroscope with Python Email application Conclusion Continuous profiling performance is a crucial factor in fulfilling the expectation of end-users. And if performance issues occur, you must be ready to diagnose the issue before impacting the end-user experience. Hence, keep optimizing your applications and fix the issues immediately to continue delivering super-fast application performance to the users using tools like Pyroscope. Pyroscope showcases a layer of visibility to help you understand how to improve the performance of your code in Production and reduce cloud infrastructure costs. That’s a wrap folks :) Hope the article was informative and you enjoyed reading it. I’d love to hear your thoughts and experience - let’s connect and start a conversation on . LinkedIn Also published on: https://www.infracloud.io/blogs/continuous-profiling-kubernetes-pyroscope/
