Debugging Node Services in Kubernetes With Linkerd 2.0

Node is one of the most popular languages for microservices. With the rise of , increasingly, Node developers are being asked to deploy their services to a Kubernetes cluster. But what’s required to safely deploy and run Node services on Kubernetes? In this post, we focus on one specific, but vital, component: how do I understand what’s happening with my Node service on Kubernetes, and how do I debug it when things go wrong? Kubernetes At this point, you may be thinking, can’t I just use the Kubernetes dashboard? Unfortunately, while Kubernetes can automatically keep your service up in the event of node failure or a software crash, it doesn’t actually know whether your service is behaving In fact, a service may be returning 500’s or taking too long to respond and the Kubernetes dashboard will show all green lights. correctly. This is why Kubernetes is often used in conjunction with something like Linkerd. Linkerd is a that augments any Kubernetes service, providing zero-config “golden metrics” dashboards and UNIX-style CLI for runtime debugging, diagnostics, and reliability. (Linkerd can also act as a by running on multiple services, providing a uniform layer of telemetry, security, and control across them.) service sidecar tools service mesh Linkerd works by inserting an ultralight proxy (<10mb) into each pod of a service, and automatically sending traffic to and from that pod through its proxy. These proxies provide telemetry data to, and receive signals from, a control plane. Because the proxy runs alongside your application and is transparent, there are zero code changes required. And, of course, Linkerd is completely open source, Apache v2 licensed, and is hosted by the Cloud Native Computing Foundation–just like Kubernetes itself. In this tutorial, we’ll walk you through deploying Linkerd 2.0 to a Kubernetes cluster and using it to debug failures in an example application. (If you don’t have a Kubernetes cluster, don’t worry! Take a look at , or just check out the for this blog post that you can run straight from your browser.) minikube interactive tutorial The demo application we’ll deploy is composed of multiple services that allow voting for favorite gifs. These services communicate via gRPC and have all been written in Javascript. You can check out the source on . GitHub The application is composed of the following 4 services: : Web frontend and REST API nodevoto-web : gRPC API for finding and listing gif nodevoto-gif : gRPC API for voting and leaderboard nodevoto-voting : a traffic generator vote-bot This application has a major bug in it, which we’ll debug by using Linkerd’s web dashboard and UNIX-like CLI tools. Step 1: Install the demo app 🚀 Before we install Linkerd, let’s install the Nodevoto demo app. In your laptop’s terminal window, run: This command downloads the Kubernetes manifest for Nodevoto, and uses to apply it to your Kubernetes cluster. Nodevoto is comprised of several services that run in the namespace. You can see the services by running: kubectl nodevoto You can also see the app live by running: (If you see something like , you don’t have Kubernetes set up properly.) kubectl: command not found Click around the app a bit. You should see that some parts of the application are broken! (hint — click the upper left vote button a few times and see what happens). If you look at the Kubernetes dashboard, of course, you’ll see that everything appears green — all pods are running. This is a very common type of failure scenario: Kubernetes thinks everything’s fine, but you know the application is returning errors, because you can see them. In the next few steps, we’ll walk you through how to use Linkerd to diagnose the problem. Step 2: Install Linkerd’s CLI We’ll start by installing Linkerd’s command-line interface (CLI) onto your local machine. Visit the , or simply run: Linkerd releases page Once installed, add the command to your path with: linkerd Let’s validate that your cluster is prepared for Linkerd by running: As there are many different types of Kubernetes clusters, and an infinite set of ways to configure them. This handy command will report any problems that would interfere with your ability to install Linkerd. Hopefully everything looks OK and you’re ready to move on to the next step. Note: if your Kubernetes cluster is on GKE with RBAC enabled, you’ll need an extra step: you must grant a of to your Google Cloud account first, in order to install the control plane. To do that, run: ClusterRole cluster-admin kubectl create clusterrolebinding cluster-admin-binding-$USER \--clusterrole cluster-admin \--user=$(gcloud config get-value account) Step 3: Install Linkerd’s control plane onto the cluster In this step, we’ll install Linkerd’s lightweight control plane into its own namespace ( ) on your cluster. To do this, run: linkerd This command generates a Kubernetes manifest and uses to apply it to your Kubernetes cluster. (Feel free to read the full manifest before you apply it.) kubectl Depending on the speed of your connection, it might take a minute for your Kubernetes cluster to pull down the Linkerd container images and get them started. While that’s happening, you can validate that everything’s installing correctly by running: This command will wait until Linkerd has been installed and is running. Now, let’s check out Linkerd’s dashboard! Just run: If you see something similar to the screenshot below, Linkerd is now running on your cluster. 🎉 Step 4: Add Linkerd to the Nodevoto web service At this point we have the control plane installed in the namespace, and we have our nodevoto demo app installed in the namespace. But we haven’t actually added Linkerd to our service yet. So let’s do that. linkerd nodevoto In this example, let’s pretend we are the owners of the service. Let’s just say that other services, like and , are owned by other teams — we don’t want to touch them. While these are local services in this example, it is entirely possible that they could be remote APIs owned by a completely separate entity. web gif voting There are a couple ways to add Linkerd to our service. For demo purposes, the easiest is to do something like this: This command retrieves the manifest of the service from Kubernetes, runs this manifest through , and then reapplies it to the Kubernetes cluster. The command adds to the manifest to include the data plane’s proxies. As with , is a pure text operation. This means that you can inspect the input and output before you use it. Since is a Deployment, Kubernetes is kind enough to slowly roll the service one pod at a time–meaning that can be serving traffic live while we add Linkerd to it! web linkerd inject linkerd inject install inject web web We now have a service sidecar running on the service! web Step 5: Debugging for Fun and for Profit Congratulations! You now have a full gRPC application running on your Kubernetes cluster with Linkerd installed on the service. Unfortunately, that application is failing when you use it–so now let’s use Linkerd to track down those errors. web If you glance at the Linkerd dashboard (by running ), you should see all services in the namespace show up. Since has the Linkerd service sidecar installed on it, you’ll also see success rate, requests per second, and latency percentiles show up. linkerd dashboard nodevoto web That’s cool, but notice is that success rate is well below 100%! Click on and let’s debug. web You should now be looking at the Deployment page for the web service. The first thing you’ll see here is that web is taking traffic from (a service included in the Nodevoto manifest to continually generate a low level of live traffic), and has two outgoing dependencies, and . vote-bot gif voting The gif service is operating at 100%, but the voting service is failing! A failure in a dependent service may be exactly what’s causing the errors that web is returning. Let’s scroll a little further down the page, we’ll see a live list of all traffic endpoints that is receiving. This is interesting: web There are two calls that are not at 100%: the first is vote-bot’s request to the endpoint. The second is a request to from to . Very interesting! Since is an incoming request, and is an outgoing request, this is a good clue that that response from is what’s causing the problem! /api/vote VoteCartoonHalloweenGhost web voting /api/vote VoteCartoonHalloweenGhost voting Finally, if we click on the 🔬 icon for that row in the far right column, we’ll be taken to live list of requests that match this endpoint. This allows us to confirm that the requests are failing (they all have , indicating an error). gRPC status code 2 At this point we have the ammunition we need to talk to the owners of the service. We’ve identified an endpoint on their service that consistently returns an error, and have found no other obvious sources of failures in the system. voting Want more? In this tutorial, we’ve shown you how to install Linkerd on a cluster, add it as a service sidecar to just one service–while the service is receiving live traffic! — and use it to debug a runtime issue. But this is just the tip of the iceberg. For example, everything we did above using the web UI can also be accomplished via pure CLI commands, e.g. , , and . linkerd top linkerd stat linkerd tap Also, did you notice the little Grafana icon on the very first page we looked at? Linkerd ships with automatic Grafana dashboards for all those metrics, allowing you to view everything you’re seeing in the Linkerd dashboard in a time series format. Check it out! Linkerd has a thriving community of adopters and contributors, and we’d love for YOU to be a part of it. For more, check out the and repo, join the and mailing lists ( , , ), and, of course, follow on Twitter! We can’t wait to have you aboard! docs GitHub Linkerd Slack users developers announce @linkerd Originally published at blog.linkerd.io on November 13, 2018.