groundcover

Get to know Caretta - a lightweight, standalone tool that instantly creates a visual network map of the services running in your cluster. Caretta leverages eBPF to efficiently map service network interactions in a K8s cluster, and Grafana to query & visualize the collected data. Scientists have long known that the turtles, like many animals, navigate at sea by sensing the invisible lines of the magnetic field, similar to how sailors use latitude and longitude. But the common sea turtle does much more than that. Turtles also rely on the Earth's magnetic field to find their way home, using the unique magnetic signature of their birth coastline as an internal compass. Turtles have effectively tamed the wilderness of the open ocean. What can be seen to some animals as an infinite unknown is mapped to the finest of details inside the head of the sea turtle. From the sea to the cloud, it’s all too easy to get lost in a typical Kubernetes cluster. Gaining a decent understanding of the inter-dependencies between the different workloads running in the cluster is a complicated task, leaving teams to work hard for impactful, actionable insights such as identifying central points of failure or pinpointing security anomalies. One approach to tackle this issue is visualization: In many ways, a K8s cluster can be seen as a geographic area, with paths and trails forged by communications between different workloads. And just as a map helps familiarize you with your neighborhood and how to navigate around it, it can help you “get around” your K8s cluster. This is part of the mission of cloud-native observability tools - no APM product is complete without network tracing capabilities of some kind, and data from these traces can help one answer those aforementioned questions. But continuing the approach from our previous post on , what if I only want a minimalistic, efficient solution? Murre So, what is the easiest way we can map our cluster? Introducing Caretta - a standalone OSS tool that does just that. Let’s dive into how it works, how it leverages eBPF technology to be lightweight and frictionless, and the obstacles and challenges we encountered on our journey to building it. The end result could be digested directly as raw Prometheus metrics, or you can integrate it into your Grafana with some of our pre-made panels: Exploring the land First thing we’ll need to figure out is how we get the network data. The naive approach would be using a sniffing tool like tcpdump to gain network observability. But that can be overkill - we don’t need to actually capture the network traffic, we just want to know it exists. As suggested above, we can use eBPF to probe just the data we need. eBPF who? If you haven’t already heard of eBPF I can tell you that people like Brendan Gregg call it an "invaluable technology" and compare it to JavaScript: was introduced in early 2014, expanding on BPF's original architecture by providing tools that allow complex programs to run directly in the Linux kernel space. OK, now you’re probably asking yourself what running the kernel space even means. Basically, it's code running in higher privileges, as opposed to running in "user space" like standard applications.  It allows the code to run very efficiently, access low-level kernel resources that would otherwise be complicated and costly (in terms of resource overhead) to access from within user space, but most importantly: it lets you observe any and all programs running in user space – which is hard to do when relying on observability tools that operate in user space themselves. This is a big thing. It’s basically a new way of equipping the Linux kernel with a programmable, highly efficient virtual machine, allowing programmers access into what was before the sole realm of kernel-developers. Observability is where eBPF shines. It allows teams to monitor their running applications with a completely out-of-band approach that requires zero code changes or R&D efforts. eBPF enables powerful advantages for observability applications by providing a faster, less resource intensive and holistic approach to gather high precision data. eBPF offers many ways to capture data from a running system. The kernel allows developers who use eBPF to attach their programs to various types of probes - places in the code of the kernel or applications that when reached, will run the programs attached to them before or after executing their original code. This diagram shows some of the probes available for eBPF: So instead of a static HTML website, JavaScript lets you define mini programs that run on events like mouse clicks, which are run in a safe virtual machine in the browser. And with eBPF, instead of a fixed kernel, you can now write mini programs that run on events like disk I/O, which are run in a safe virtual machine in the kernel. eBPF Image from ebpf.io/what-is-ebpf Starting with the basics Working on this project, I was inspired by - a nifty tool to calculate statistics and information of TCP lifespans, using only a single eBPF probe. The thing is, the kernel already does half of the job for us as it maintains information, such as throughput, for each socket. By probing the tcp_set_state kernel function, tcplife is aware of the active network connection and is accessible to data the kernel maintains for them. Beside having minimal footprint, another advantage of probing specific kernel functions is covering all possible data flows - compared to probing a bunch of possible syscalls the application might use, such as read() or recv(). When application-level context is unnecessary, “sitting” close to the root of the tree lets you cover its branches more easily. tcplife Sounds great - but we’d still be missing something. tcp_set_state is called when, obviously, the state of the TCP connection is being set. For example, when a server starts listening, or when a server-client connection is established, or when a connection is closed. But TCP connections can go long without changing their state, and relying on tcp_set_state will keep us blind to them. We set out to look for an additional probe that can help us complete the picture. Even then, we find eBPF useful to explore the linux TCP stack. Tools like or can be used to understand the flow a network packet is going through when it’s processed and compare different functions to see how “noisy” each function is. Searching for data probing locations consists of a constant trade-off between being too nosy and being blind, and we’re looking for the sweet spot in the middle. stacksnoop stackcount In our case, we found the function suitable for our needs: tcp_data_queue This function is called as one of the final steps of , the function used when receiving packets in the ESTABLISHED state: tcp_rcv_established So it covers the blindspot of , and is called only if all previous steps were successful. tcp_set_state Another advantage to using this function is that like , and most of the kernel TCP functions, its first argument is a struct sock object. This is a big, important kernel struct and we’ll soon dive into the advantages and disadvantages of relying on it to retrieve data, but for now we’ll say it’s convenient that both our probes share a consistent approach to the data. tcp_set_state To recap our collection mechanism - we’ll probe to observe the network sockets updating its stats in their established state, and tcp_set_state to track their lifecycle. tcp_data_queue Understanding the data As mentioned, our “supply” of data comes from internal kernel network stack objects of a struct called sock. Socks actually wrap other data structures for TCP and INET, so having them means having the data for lower levels of the connection as well. Information like the connection’s 4-tuple (source IP, source port, destination IP and destination port) can be easily retrieved from the sock, especially using the CO-RE approach. Some issues arise from using kernel internal structures. For example, instead of allocating a new sock each time, the kernel may reuse closed structs. Apparently, socks reusing might occur frequently, and if we do not keep track of socks ending and starting it can lead to collisions and disinformation. Another difficult issue for extracting the data out of the was understanding the role of our host. The sock contains source IP and destination IP, but has no information about who’s the client or the server in this connection. For new socks, we can see (in ) if it goes through TCP_SYN_RECV or TCP_SYN_SENT to understand the side of the equation it’s on; But if the connection was already established, we can’t see the previous states it’s gone through. The solution we found was examining a field in the sock struct called . It holds the maximal length of the accept queue, and is assigned as part of the listen() function of the connection. Which means, that if it is assigned a positive number, the sock went through a listen state, and therefore is a server - and vice versa. That information gives us the direction of the link and the relevant port to store. sock tcp_set_state sk_max_ack_backlog Aggregation So we figured out how to deal with those sock structures. But we’re not really interested in them - we’re interested in it at a higher level. If a client queries a server hundreds of times one after another, we’d like to address it as a single edge in the graph. So the next thing we’re going to do is aggregating what we see in the kernel probes to start counting (tuples of client, server and server-port) and their . The eBPF program will maintain a map of the sockets it observes, and a userland program accessible to this map will poll over it and generalize each connection to a link and aggregate the throughputs. We’ll run this program on each node in our cluster, and every instance of it will publish the results as to be scraped by a centralized pod. Let’s talk about the structure of our data. We basically want to have a counter for each link, which is populated by aggregating smaller counters - the ones maintained by the kernel. Sockets can come and go, so we need to keep track of the total throughput of past, closed sockets as well. Eventually, our metrics will expose the total throughput of each link observed since launching the program. That information, scraped and consolidated by a prometheus agent, can be easily analyzed with standard queries such as sorting, calculating rate, filtering namespaces and time ranges, and of course - visualizing as a network map. Below is an example of a timeseries published by a Caretta’s agent: Links Throughputs metrics As you can see, the identifiers are resolved from an IP address to the controllers owning the workloads communicating. For example, in this link we can see sending 2537 bytes to a service named Note that some of the labels are generated solely to comply with the format Grafana expects for displaying a Node Graph. checkoutservice productcatalogservice. Speaking in Kubernetes terms One thing left before publishing the metrics is resolving the source and destination of each link to the language we want to speak - our cluster’s language, with our k8s-defined names. That might sound trivial - the information should be easily retrievable using the K8s API. But again, some obstacles arise. The first and simple one was API requests overhead. Querying the API for each object we encounter, especially in the beginning of the run of the program, leads to a big amount of REST messages which simply take too much time. The solution was using a bunch of big queries in the beginning of each polling iteration and storing a snapshot of the cluster’s state. With the snapshot, we can build a mapping of the IPs used by objects in the cluster, and for each object get its ancestors in the hierarchy to consolidate the nodes of the graph to meaningful items like DaemonSets and Deployments. On the other hand, some cases are very hard to resolve. For example, Caretta doesn’t support resolution of Pods set to “host” network, as they use the node’s IP address directly themselves. Keeping track of bound ports and their processes can be a solution for that in the future, but it will make the program more complicated. Same problem occurs with traffic originated in NAT’d services arriving in other pods in the cluster, which makes it impossible to associate it with a workload given the source IP alone. We can also add reverse-DNS lookup to try and resolve external IP addresses. Putting it all together With resolved and aggregated metrics, what’s left for us is running an agent pod as a daemonset in our cluster along with a single metrics scraper. To visualize a map, we will use Grafana’s Node Graph feature - a fairly new panel in beta version. Of course, there are countless libraries to create an image of a map out of a description of relations, but we wanted to use a tool which is very much likely already installed on your cluster. Caretta is equipped with a Grafana dashboard consisting of a Node Graph panel and some PromQL witchcraft queries to adjust the data to Node Graph’s specifications. After setting Caretta as a Prometheus data source in Grafana, the dashboard will display the map of your cluster. But actually, instead of trying to explain the architecture of Caretta with words, let’s use… Caretta itself! Here’s what an instance of Caretta shows when filtered on its own namespace: As you can see, we have a Grafana instance querying a VictoriaMetrics ( ) agent (and displaying this map on its web UI); The Victoria agent scrapes metrics from caretta daemonset; And both the Victoria agent and Caretta make use of the kubernetes API exposed by the Kubernetes service. But I’m sure you've already figured that out :) Beside that, we have the Kubernetes Node itself probing the Grafana’s and Victoria’s health probes and the Grafana pulling external resources from the web. caretta-vm Gaining landscape insights Along the road, analyzing (and debugging :)) the data collected by Caretta’s agent revealed some surprising behaviors in our dev cluster. Although the main purpose of creating this tool was understanding the communications between different services, the “background” traffic caught our attention and enlightened us. Here are some examples of that usually-transparent traffic: 1. Non-container networking: A lot of the traffic observed by Caretta wasn’t coming from a pod at all, but from “normal” processes running on the nodes themselves. In our case, it was the kubelet process itself next to aws-specific agents. This communication shouldn’t come off as a surprise, as it is essential to controlling and running the cluster, but seeing its intensity clears some mist surrounding it. 2. Control-Plane networking: like how the kubelet is a “transparent” component in the node, the control plane is a cluster-wide “transparent” component. We all know the control plane and its role, but have you ever wondered where it actually resides? The answer to this question is vendor-specific, but for AWS and GCP the control plane is actually hosted in servers outside of your cluster managed by the vendor itself, bridged to the cluster with dedicated interfaces. 3. Endpoint with no targets: continuing with the control plane, the API server is exposed via the kubernetes service in the default namespace. But because, as mentioned, the API server doesn’t actually reside in any workload inside the cluster, and this endpoint merely forwards the traffic out to the control plane. Final Words Mapping the inter-dependencies between the different services running in the cluster is a complicated but critical task. Unlocking this rare type of information can help developers, devops and SRE teams get a much clearer view into their production K8s clusters. eBPF is shaking the foundations of how we understand and experience observability. What was once almost impossible to get can now be obtained in just a few minutes on huge complex clusters running hundreds of different microservices, and Caretta implements exactly this modern lightweight approach. The next tools that will rule the K8s eco-system will be efficient, ad-hoc and fast. Teams just have too much on their plate to work hard for critical data. Caretta is open-source, and we’d love to get your feedback and contributions. Head over to our GitHub , create an issue with a bug or a feature request, or drop your thoughts on our community slack Previously Posted on groundcover's blog by Udi Rot, Founding Engineer

Navigate Your Way to Production Bliss with Caretta

THE GIANT RATS

Meet Murre. Murre is an on-demand, scaleable source of container resource metrics for Kubernetes. Murre fetchs CPU & memory resource metrics directly from the kubelet on each K8s Node and enriches the resources with the relevant K8s requests and limits from each PodSpec. Minimalism. Yeah, big word I know, but bear with me here. Basically, it is all about owning only what adds value and meaning to your life, and removing the rest. It's about removing the clutter and using your time and energy for the things that remain, since we only have a certain amount of energy, time, and space in our lives. The best software developers are minimalists in heart. Minimalism does not mean writing less code, but writing code that counts with elegant and tight structure, and does one thing well. Minimalism, in this sense, means designing systems that use the least hardware and software resources possible - because you can. Because that’s how you believe things should be. Like many developers we implement this approach to every thing we design and build. Our journey to building Murre starts when we wanted to expose K8s node resources and usage metrics for our customers. As a company building a comprehensive Kuberenetes application monitoring solution the need was clear. We wanted to monitor and export the resources that are part of the cluster’s infrastructure layer - CPU, memory, and disk usage on each node aggregated by each container. Sounds simple right? Thing is, as a monitoring platform embedded in the heart of the Kubernetes cluster, we always bear the token of minimalism. We aim to be as lightweight as possible wherever we can - which also means we prefer to not install any 3rd party tools on the cluster, if we don’t absolutely have to. So we’ve set course to get our hands on the K8s node metrics, BUT without having to install the famous . Metrics Server The traditional way to K8s metrics monitoring Very often we see customers struggling with monitoring the physical layer that has been abstracted by K8s (you know, the same old CPU, Memory, Disk, etc). This becomes especially important when you’ve got a noisy neighbors effect which means your pod might be using the expected resource allocation but still . suffer from missing resources The Kubernetes ecosystem includes a few complementary add-ons for aggregating and exposing monitoring data from your Kubernetes cluster. The Metrics Server is one of these useful add-ons. The Kubernetes Metrics Server is a cluster-wide aggregator of resource usage data. It collects metrics like CPU or memory consumption for containers or nodes, from the Summary API, exposed by Kubelet on each node. However, it is an add-on which means it’s not part of the out-of-the-box K8s and not deployed by default in standard managed Kubernetes platforms. The Metrics Server is intended to be scalable and resource-efficient, requesting about 100m core CPU and 200 MiB of memory for a typical cluster of 100 nodes. It allows storing only near-real-time metrics in memory, supporting ad-hoc checks of CPU or memory usage, or for periodic querying by a monitoring service that retains data over longer timespans. Though designed to scale well with your K8s cluster, it doesn’t come free of issues. Installing a 3rd party service running in your cluster means you have to maintain and troubleshoot it from time to time. It can , take up more resources than you intended, or fail to do its job. The Metrics Server is just a glimpse into what it means to take responsibility for another ever-running service in your already busy cluster. crash known issues page And there’s one more point that  concerned us. At we’ve built a distributed observability solution that can scale with the cluster and can take decisions at the edge. For example - we want to sample the HTTP span that caused a CPU spike in a relevant pod. groundcover Going the Metrics Server route would (beyond the need to install and run it) also mean each of our edge units (running on each node) would have to query the central Metrics Server regularly to create such a trigger. That would force our distributed architecture to rely on a centralized point and would make things hard to scale. Taking it to ultra-light groundcover is all about offering a lightweight and frictionless approach to Kubernetes application monitoring, so we started looking for other ways to get the job done. First, we started by measuring resources ourselves by operating Linux OS tools, like , , etc directly on each node in the cluster. That solved one piece of the puzzle of enabling resource monitoring that didn’t require any prior installations or maintenance. However, it did introduce an efficiency issue. The outputs of tools like and required parsing. They were also Linux and not K8s tools, so it required a second layer that made sense of process resources and turned into the containers resources our customers know and understand. top ps top ps But, like any good answer to a problem, ours has been staring at us all along. The Metrics Server is built in a centralized fashion, one per cluster. So how does it get all the metrics it needs from all the different nodes? A quick look at the code reveals the clear answer - by simply querying the Kubelet running on each node. Suddenly it all made sense, since K8s itself must also measure node resources as well for the Kubelet workflow to operate! Kubelet who? Kubelet is a process that runs on each node of a Kubernetes cluster and creates, destroys, or updates pods and their containers for the given node when instructed to do so. Basically, the Kubelet is the primary "node agent" that runs on each node and it works using PodSpecs (a YAML or JSON object that describes a pod). It is responsible for taking PodSpecs that are provided through various mechanisms (primarily through the apiserver) and ensures that the containers described in those PodSpecs are running and healthy. In Kubernetes, scheduling, preemption and eviction are an important part of the cluster’s life. Scheduling refers to making sure that pods are matched to nodes so that the Kubelet can run them. Preemption is the process of terminating pods with lower priority so that pods with higher priority can schedule on nodes and eviction is the process of terminating one or more pods on nodes. The Kubelet plays a critical role in these major scenarios. For example, a scenario called node-pressure eviction. Kubernetes constantly checks node resources, like disk pressure, CPU or Out of Memory (OOM). In case a resource (like CPU or memory) consumption in the node reaches a certain threshold, Kubelet will start evicting Pods in order to free up the resource. This is exactly why the Kubelet must constantly use Kubernetes resource metrics to do its job, and expose these metrics to other services that might need them as well. Great. So if the Kubelet is already exposing this API to the Metrics Server, it means we can use this API ourselves. That’s two birds with one stone! We can use this API and query it directly without deploying a Metrics Server on the cluster, but we can also query it inside each node - without having to leave the node.Oh, but it’s not documented anywhere…So we had to dig deeper. ‍ A deep dive into the Kubelet source-code We started to study the Kubelet sources to figure out how it attains the K8s resource metrics and how it exposes this data as APIs down the stream. What we found is that not only is Kubelet measuring the resource usage on each cluster node, but it also exposes the data using Prometheus formatted metrics - which we love! The Kubelet API is not documented, but from its we found the endpoints. There are more endpoints that are not used for metrics / stats, but these are out of scope for this research. There’s definitely more gold there to be uncovered, but we’re going to focus on K8s metrics for now. sources You can find some of the metrics definitions here : pkg/kubelet/server/server.go In order to reach the Kubelet API so we can see the data: Some of Kubelet’s endpoints return Jsons and some export actual metrics. Here’s a few worth noticing: - The endpoint is metrics originating from cadvisor and it provides all the metrics on container resource consumption like: /metrics/cadvisor CPU Memory File System Network - This endpoint also lists container resources (CPU, Memory) like the cadvisor endpoint, but also gives us pod level and node level resources. /metrics/resource ‍ - This endpoint provides an aggregated resource consumption data in JSON format. By default it will describe all resources (CPU, Memory, File System, and Network like the cadvisor endpoint), but you can pass a flag of as a request param and get only CPU and Memory data if that’s what you’re after. /stats/summary only_cpu_and_memory=true Here’s how the response structure of the endpoint looks: - This endpoint provides information about pods running on the node, with full pod specs and status. This data can be also fetched using the K8s client podLister interface. While useful, this specific data is irrelevant to our cause of resource metrics monitoring. /pods ‍ - This endpoint exposes metrics related to Kubelet’s own internal statistics. It’s good to know it’s out there but like before it will not be used for our needs. /metrics And then there was Murre The Common Murre is an interesting bird. It breeds in high density colonies, and make no nest; Their single egg is incubated on a bare rock ledge on a cliff. Minimalist? They take their nesting skills to the extreme. No wonder we liked that name :) After we figured out how to get those metrics without storing them, we thought - we can help solving , , and many more with the same pattern; introducing . this this Murre Murre is an OSS tool that helps basically you get your container’s CPU & memory metrics without the need to install anything on the cluster. It works the same way as metrics-server just without storing the metrics. With Murre you can filter a specific namespace, pod or even container name to focus on the exact metrics you want. Murre utilize 3 different APIs to get the data needed to craft these metrics: - in order to discover and maintain the list of all available nodes. NodeList - to get the actual usage metrics. This API is used for each and every Node as we described earlier. /metrics/cadvisor - to enrich the data with K8s resource requests and limits for each and every container. PodList Here’s how Murre's processing flow looks: And voilà! murre demo - Watch Video Murre is a useful CLI tool that any team using Kubernetes can leverage. Feel free to think big, open issues or add capabilities yourself. The sky is (literally) the limit for this special bird. Questions about Murre? Join our community Slack and get in touch with the creators!

Murre - the lightweight K8s metrics monitoring tool

Whoever came up with the famous saying that you can choose from two of three options – fast, cheap or good – was probably not an observability engineer. But, he or she may as well have been, because when it comes to observability, deciding whether you want fast, affordable or in-depth insights has traditionally been one of the first tasks of engineering teams. Historically, the monitoring architectures and tools that we depended on didn't allow us to have it all, at least not in the context of distributed, cloud-native apps. Fortunately, that has changed. It turns out that, by rethinking the fundamentals of your approach to monitoring, you can have it all – when it comes to observability at least. You can get deep insights quickly, without paying through the nose for your monitoring workflow. Sounds too good to be true? Well, keep reading for a look at how to square the circle surrounding cost and depth (and speed, too) in regard to cloud-native observability. When engineers had it all: Traditional approaches to observability The engineers of generations past didn't have to choose between cost, depth or speed when it came to observability. They got to have it all without even trying. This is because they were working with centralized, monolithic apps. In that context, it didn't cost much time or money to collect monitoring data in quantities sufficient to enable fully informed management decisions. In most cases, you simply integrated a lightweight monitoring SDK into your monolith and let it collect basic metrics and log data. The design was simple, the process was simple and the costs were low. This approach worked perfectly well in a world dominated by monoliths and single-node application deployments. No one questioned it, because there was no reason to question it. How things fell apart: The dawn of cloud-native observability The problem that many engineering teams have run into over the past decade or so is that, when you take a conventional, monolith-friendly monitoring architecture and try to graft it onto a distributed, cloud-native app, you can no longer have it all. You can't necessarily monitor quickly because integrating an SDK into all your microservices takes time. Plus, you have loads more data to collect, since you're not just dealing with some basic metrics from a single app. Instead, you’ve got a whole host of logs and metrics (and don't forget your traces!) from a bunch of microservices. It's also very hard to collect all of this cost-effectively. You can run into steep egress bills just to move the data into a place where you can analyze it, and then you have to pay storage fees on top of that. Glacier-tier storage may be cheap, but it adds up when you have reams and reams of monitoring data that you need to retain for years. Now, one way to speed up cloud-native observability, and reduce costs, would be to collect only some data that your tools select at random, instead of trying to collect and analyze every single log, metric and trace available to you. But then you'd be sampling – and , because data sampling means you may miss important information due to incomplete data collection and analysis. no one wants to be a data sampler So, if you depend on a traditional observability strategy for distributed apps, you end up facing, what I like to call, the cost-depth trade-off. You can observe quickly and cheaply. Or you can observe in-depth, but at a high cost in terms of time and effort. You can't have it all. Escaping the cost-depth trade off Fortunately, things don't have to be this way. If you step back and rethink your approach to observability, you realize that you can have both cheap and in-depth observations. Here's the trick: Instead of trying to collect and analyze every bit of data available or sample it randomly, you sample it intelligently by identifying the most interesting data right at the source, then select only that data to send to your observability platform. You also translate the data into granular, actionable metrics, so that it's primed to be analyzed as soon as it hits your observability portal. Still too good to be true, right? The thing is that an instrumentation-based approach for monitoring will essentially mean that if we’re trying to measure key things about your code, you will need to “wrap” parts of it with external code segments that will manage the logic of the monitoring tool. Make the logic too complex, and . Generally speaking, the more you analyze the data coming from your application before just sending it out, the larger the overall impact of the generated overhead on your application. you pay with an overhead That’s why conventional monitoring tools turn to simple logic like sample it all, or sample it randomly. Without diving too deep into details, that limitation can be lifted by new emerging technologies like . eBPF The result is much less data for you to transfer, analyze and store. But at the same time, because you're focusing on relevant data, you don't have to lie awake at night wondering if the data you've selected will reliably yield the in-depth insights you need. Keep the needles outside the haystack If you like analogies, here's one that sums up the observability strategy we're talking about here: It's akin to sorting through each bundle of dried grass as you build a haystack, checking to see if there's a needle inside. That way, you catch the needles early on, without having to wait until you have a whole haystack in place to sort through it and try to pull out the needle. The needles are easy to find because they never get buried inside the haystack in the first place. In fact, you have no haystack to deal with at all, because you can discard the hay you don't care about before it turns into a costly mess of a haystack. The future of cloud-native observability I think that this approach to observability – which is the one that inspired the design of – is key for any team that wants to maintain reliable visibility into its systems without paying a fortune in monitoring and data storage costs. The volume and complexity of cloud-native log, metric, and tracing data will only increase, making conventional monitoring and observability strategies less and less viable for cloud-native environments. groundcover If you want to have it all – and you can. Adopt an observability architecture that lets you hone in on relevant data right at the source, and you’re golden.

Goodbye, Cost-depth Trade Off: Cloud-Native Observability, Redefined

Introduction Launching a dapp may not be as hard as launching a rocket… but it’s no moonwalk, either. A dapp, portmanteau of “decentralized app,” is any app that runs on decentralized infrastructure, like on Ethereum. For your coding pleasure, our team is happy to present Muttniks, a friendly Ethereum space doggo “kernel” (or kennel) that you can build from the comfort of your spacecraft. Muttniks is an open source sample dapp, built to guide you through our exciting journey with Astro Ledger. Like the real stars and planets featured on , you can securely adopt, name, and trade Ethereum space doggos with Muttniks (and then donate your testnet ETH to build more monuments). The following is a step-by-step technical exploration of the inner workings of Astro Ledger, which you can recreate on your local machine with Muttniks. astroledger.org Laika Muttniks was the “pet” project of Astro Ledger CWM (Chief Wow Muttniks) . Thanks Tony! Tony This star’s Henry Draper catalog number is HD137052. ’. It has been named ‘Tlon Keeping Track of Stars & Muttniks In order to assign names to actual stars, we needed a way to uniquely identify them. For this we used the Henry Draper Catalog of the brightest stars in the sky, with meta-information provided by the from The Astronomy Nexus. HYG database Our specially selected Muttniks have no such prior catalog so we pseudo-randomly assigned each Muttnik a number, 1–6. The meta-information about each Muttnik (name and image) is loaded into the database in this . Our Astro Ledger creative content (spearheaded by Chief Chocolate Dehydrator ) and image links are similarly loaded into the database, with SQL code created by offline Python scripts that load data from HYG and other sources. Flyway migrations file Aleeza When a Muttnik is added to a database, we use a to add the same Muttnik to the contract, putting it up for adoption. For Astro Ledger stars, those same offline Python scripts create corresponding Truffle migration scripts to create stars and put them up for space-grant-supporting-auctions at the price of about a dollar. Truffle migration script Contract & Events: Creating and Adopting The Muttniks are stored in a where each pet has an address variable for the adopter, initialized to address(0), and a boolean variable exists to distinguish this pet from one that hasn’t been created yet. “pets” mapping The function createAdoptee is used by the owners to create each Muttnik in the contract. This is typically done in batch with a , but could be called by any function. Adoption happens via the adopt function, which checks to see that the Muttnik hasn’t already been adopted before assigning the Muttnik’s address to the adopter. Adoption events are emitted so the browser can inform the astronaut/cosmonaut of the adoption when the blockchain confirms it. Muttnik Kernel Truffle migration script Contract & Events: Naming Muttniks The obvious solution to handling Muttnik (and star) names was to store the name as a variable in the contract. To find the name, folks could call a view function (for zero gas) that would return the name. However, storing data in an Ethereum contract is never a cheap operation. Because Astro Ledger’s goal is to allow anyone to be able to assign meaningful names to stars, we implemented the assignName function using instead. Names are forever stored on the blockchain as 32 bytes of data on a log of the event, but because they aren’t accessible to the Ethereum Virtual Machine (EVM), the gas cost to assign a name is almost negligible. This philosophically-motivated decision allowed us the exhilarating discovery that much of the existing software for Ethereum does not fully support event-logs, but we went to the edges of the universe to achieve a solution for pulling up the names quickly, as described later. events The owner of is well aware that changing a star’s name is not to be taken lightly… HD162004 Web Stack While Chief Rubber Taster worked on the contract and the star database, Tony began building the Astro Ledger web stack using Play! for a web framework ( ), and Postgres for the database (ol’ ). At first, Tony chose to generate HTML pages of available stars. He quickly learned that the majority of the anticipated Astro Ledger user base had settled on a browser plugin called MetaMask to talk to Ethereum blockchains, and so began sneaking in some JavaScript (including a popular library called ) into the webpages to support this. The code to generate web pages soon became a funny mix of Scala, HTML, and ES5, where all of the blockchain interactions were executed using web3js. This worked, and was fast enough in our test environment. Jonathan ol’ reliable good enough for everything Twirl web3js Though it’s probably still fair to call the main Ethereum network itself a “test environment,” deploying our app to it in our first production alpha revealed one that was far slower than even Tony’s 2014 Macbook Air. This was particularly true for the for-loops we used in our contract to search for a user’s star cards, which were executed individually in the user’s web browser whenever they went to our page (note that, to guard against this performance hit, the uses no for-loops). To prevent expensive, redundant contract calls, we decided to move some of these queries to the Scala side of things, where they’d execute once on our Heroku instance every few minutes, and then be accessible to a user’s web browser without it having to call any contract code itself. Muttniks Solidity code isn’t for sale, but it’s hot enough to melt your heart. Jellybean This was great for querying a user’s star cards, but things became interesting when we needed to query the names that they assigned to stars. As mentioned earlier, rather than put these names in contract storage as we did with auctions and card ownership, we chose to keep a record of star names using Ethereum events: this was intended as a . So why not cache on our server? As we soon discovered, (a library similar to web3js that we used for querying the blockchain in Scala) on Infura, the free client we used for accessing the main Ethereum network. Web3js support querying events in the user’s browser, but this had even worse performance effects on the browser than querying ownership, because it would poll the blockchain every second or so to get event information (web3js 1.0 ). less expensive form of storage these web3j did not support filters did has since addressed this To get a better handle on things, we decided to go closer to the source. In earlier investigations, Tony had coded up some Clojure code to talk directly to Infura’s RPC endpoints. With it running on our server, we could essentially do the same thing as web3js would in someone’s browser, but in one place (for $7/month) and at a longer poll interval. So, we decided to turn it into a full blown service. Coupled with Redis, this gave us our own centralized names cache, which, as with auctions and card ownership, could be accessed in someone’s browser much more quickly than if it were to query the blockchain directly itself. Single-Page In the meantime, Chief Rubber Taster Jonathan was tasting the rubber over in Tel Aviv when he met the team developing , a browser based smart-contract enabled wallet. Portis would allow folks to use Ethereum on a vanilla web browser (i.e. without MetaMask installed), including iOS or Android. However, , our initial approach of rendering web pages server-side (where clicking a link would open a brand new page) would oblige users to log in over and over again whenever they navigated anywhere on our site. The solution was to migrate to a that could be downloaded all at once in a user’s web browser, to provide full functionality without a user ever having to refresh the page or log in again. Portis since the Portis library only stored user credentials in browser memory single page application Easily accessing astroledger.org with Portis, so we can add to our star collection. Libre To accomplish this, we settled on , since it afforded us the ability to easily use another popular library, to interoperate with web3js in a singly-paged way. However, The Drizzle library itself relied on web3js 1.0, which at the time . We eventually abandoned Drizzle, but continued on with React. React Drizzle-React was not fully compatible with MetaMask Muttniks! Muttniks is a pared down version of the Astro Ledger app, available here: https://github.com/tlazenka/muttniks To get started with Muttniks, head over to the ! The overall structure can be gleaned from the , and by drilling down into each service’s individual list of dependencies (note that some dependencies aren’t relevant to Muttniks, but were used by Astro Ledger). is a convenient tool that lets you package up all these fun little services so you can easily launch them together. cheat sheet docker compose file Docker compose Docker Compose Services app The main Scala app that sits atop Postgres. When the app is started, it runs the database migrations if needed with , and then from the resulting schema. A periodic task is scheduled with to query the blockchain with web3j and update the Postgres database. Flyway Slick generates the case classes Akka db/redis The Postgres/Redis databases. jobs The Clojure service that sits atop Redis. When started, it schedules a task with to periodically query the local Ethereum blockchain, and caches the results in Redis using . Quartzite Carmine sbt/lein Centos images that map sbt/ivy2/m2 directories to your local machine to speed up dependency downloads. truffle A Truffle service for compiling the Ethereum contract and running the tests. ganache A Ganache service used here for a local development blockchain. react The React version of the website. web3j A web3j image that includes the for generating code from the Truffle JSON artifacts. command line tools firefox A service with a Firefox browser plus MetaMask that you can access using VNC, you don’t want to install MetaMask in your own browser. Wrap-Up In case you are struggling for a reason why Muttniks needs to be on any blockchain… you are not alone in the universe. However, let’s suspend disbelief and say that the gray area of “ownership over the naming privileges of non-existing Soviet dogs long gone to the happy hunting ground in the sky” needs to be codified in some way… This vegetating llama could be vegetating on blockchain… Photo by Cheryl Strahl You can put literally anything countable on blockchain! is just one of needing a name… HD16620 many awesome stars We are really happy with how the framework runs and would love to hear about how Muttniks has helped you to put vegetating llamas on blockchain, or maybe adopt every single Amur Leopard on blockchain, or perhaps slam slowly slaloming snails… yes, on blockchain!!! If you are having any difficulty with your implementations, just radio for help. The would be happy to work with you — shoot an email to . Astro Ledger team groundcontrol@astroledger.org And if you just can’t get enough Muttniks, don’t forget to beam over to and try naming some blockchain stars! astroledger.org

Stories, News & Mentions about GROUNDCOVER

Groundcover’s Stories on HackerNoon