A 5-min Intro to Programming the Kernel with eBPF

Welcome to this issue of Activation Function. Every other week, I introduce you to a new and interesting open-source backend technology (that you’ve probably only kind of heard about… ) and explain it to you in 5 minutes or less so you can make better technical decisions moving forward.

In this issue, we’ll explore eBPF (Extended Berkeley Packet Filter), an exciting new technology that makes programming the kernel flexible, safe, and accessible to developers.

TLDR

• eBPF is a mechanism that makes the kernel dynamically programmable without modifying the source code.
• eBPF is safe, fast, incredibly flexible, and extensible.
• eBPF has been running in production for over half a decade at an internet scale on millions of servers.
• eBPF use cases range from observability, networking, security, tracing, and profiling.

[IMPORTANT NOTE] eBPF is now a standalone term that doesn’t stand for anything. You'll see the term BPF in Linux source code, and you'll see BPF and eBPF used interchangeably in tooling and documentation. The original BPF is sometimes called cBPF (classic BPF) to distinguish it from eBPF.

Why program the Kernel?

The kernel can oversee and control the entire system, which, on the one hand, makes it the ideal place to implement networking, security, and observability capabilities and, on the other hand, makes it very risky to fiddle with.

As a result, innovation at the kernel level has been super slow! After all, as the saying goes,

With great power comes great responsibility.

Up until recently, if you wanted to add functionality to the kernel, you had two options:

1. Try to change the kernel's source code and convince the community that the change is required, which, as you can imagine, took ages.

• Load kernel modules (LKMs) can be risky on many levels… (security, performance, stability, compatibility, and the list goes on.) and costly to maintain as every kernel version upgrade can break them.

Enters eBPF

‍

eBPF is a mechanism that makes the kernel dynamically programmable, kind of like how JavaScript lets you dynamically change the behavior of a webpage. Here is a great high-level explanation by Brendan Gregg:

eBPF does to Linux what JavaScript does to HTML. (Sort of.) So, instead of a static HTML website, JavaScript lets you define mini-programs that run on events like mouse clicks, which are run in a safe virtual machine in the browser. And with eBPF, instead of a fixed kernel, you can now write mini-programs that run on events like disk I/O, which are run in a safe virtual machine in the kernel. In reality, eBPF is more like the v8 virtual machine that runs JavaScript rather than JavaScript itself. eBPF is part of the Linux kernel.

Although eBPF is far from completely replacing LKMs, it sets itself apart by bringing great flexibility while mitigating risk by putting solid safety and controls in place.

How we got here?

1992 – Van Jacobson wanted to troubleshoot network issues, but existing network filters were too slow. He and his team developed BPF (Berkeley Packet Filter) to be fast, efficient, and easily verifiable to run in the kernel safely.

BPF was a great technology, but it had a few limitations that became apparent over the years as networking technology evolved. Among other things:

• It wasn't adapted to modern processors and multi-processor systems.
• It was stateless, which made it a bad fit for complex packet operations.
• It took a lot of work to extend for developers.

2014 – Alexei Starovoitov introduced the extended BPF (eBPF) design that took things to a whole new level by:

• Overhauling the BPF instruction set to take advantage of modern hardware.
• Introducing Helper functions that eBPF programs can call to interact with the system.
• Introducing the bpf() system call so that user space programs can interact with eBPF programs in the kernel.
• Introducing the eBPF verifier which ensures that an eBPF program is loaded only if it’s safe to run.
• Moving beyond packet filtering and opening the door for many use cases around networking, observability, security, tracing, and profiling.

Today, eBPF is a general-purpose compute engine within the Linux kernel that allows you to hook into, observe, and act upon anything happening in the kernel.

Check out by Alexei Starovoitov for an in-depth history of eBPF.

How does eBPF work?

Before we delve into this, it’s essential to understand the difference between the kernel and user space in Linux.

Here is a quick rundown:

• The Linux kernel is the software layer that sits between your applications and the hardware they run on in a layer called the user space.
• The user space is unprivileged; therefore, it can’t access the hardware directly.
• When an application requires something from the hardware, it will need to request the kernel, which is privileged, to do it on its behalf using the system call (syscall) interface.
• The kernel then relays the request to the hardware, coordinating concurrent requests and ensuring everything runs smoothly.

Alright, back to eBPF. Without going the rabbit hole, here is how it works on a high level:

Step #1 | Program Development

You can write your own eBPF program using a tool like bpftrace that provides an easy-to-learn high-level language or the BPF Compiler Collection (BCC) Python framework. The program is then compiled into bytecode.

[Note] As a beginner, you don’t need to write eBPF code from scratch, as BCC comes with over 70 tools you can use out of the box. Here is a glimpse of what you have at your disposal:

Source — https://www.brendangregg.com/

​

Step #2 | Program Verification

The bytecode runs through the eBPF verifier inside a VM to ensure it will not harm the system before being loaded into the kernel.

[Note] The verification process is quite complex. You can read more about it here. Although much work has gone into improving and simplifying it, you can still run into strange errors when developing your program. If you need help, check out the eBPF Slack community channel.

Step #3 | Program Attachment

The verified program is loaded into the kernel and attached to predefined hook points before being further JIT compiled at runtime into native machine instructions to ensure maximum performance.

Step #4 | Program Execution

The program is triggered on predefined events, and helper functions are called.

Maps are then used to pass data between the kernel and user space or other eBPF functions and to maintain the state.

[Note] eBPF program becomes active when loaded into the kernel. You don’t need to reboot the machine, restart existing processes, or change anything about other applications.

eBPF in production

Since its inception in 2014, eBPF capabilities have continued to grow, supported by 300+ kernel developers and major tech players, including Netflix, Meta, Google, Cloudflare, DoorDash, and many others, running eBPF-based tools in production for over half a decade 24/7 at internet scale on millions of servers. Let’s look at some examples:

• LinkedIn uses eBPF for Infrastructure Observability.
• Netflix has developed a network observability sidecar called Flow Exporter, which uses eBPF tracepoints to capture TCP flows in near real-time.
• Cloudflare uses eBPF for network security, performance monitoring, and network observability.
• Apple uses eBPF for kernel security monitoring.
• Meta uses eBPF to process and load balance every packet coming into their data centers.
• DoorDash uses eBPF for application monitoring.
• and use eBPF for GPU performance monitoring.

And the list goes on.

eBPf limitations

No technology is perfect, and eBPF isn’t an exception. Let’s discuss a few current limitations you should be aware of:

• eBPF was initially released in a limited capacity in 2014 with Linux 3.18. You need at least Linux 4.4 or above to use eBPF fully.
• Despite much work, eBPF portability between kernel versions and distributions is still not 100% there.
• eBPF is still a pretty complex technology that isn’t easy to grasp for the average developer. Anyone working with eBPF will need a solid knowledge of networking and kernel inner workings.
• eBPF is still in the early stages of expanding to other OS ecosystems, with Windows leading the charge.
• Despite significant efforts by the community and large companies like Google to secure eBPF, it’s still vulnerable to cyber-attacks.

eBPF-based projects

Despite a lot of effort put in by the community to make eBPF more accessible, the reality is that it’s still a pretty complex technology to work with for the majority of developers.

The good news is that if you want to leverage the power of eBPF, there are a growing number of projects that can help you do that without writing eBPF programs:

• ​​Falco is a behavioral activity monitor designed to detect anomalous activity in applications.
• Tetragon provides eBPF-based transparent security observability combined with real-time runtime enforcement.
• Parca helps you track memory, CPU, and I/O bottlenecks broken down by method name, class name, and line number over time.
• Cilium is an open-source project that provides eBPF-powered networking, security, and observability.
• Calico Open Source is designed to simplify, scale, and secure container and Kubernetes networks.
• Hubble is a fully distributed networking and security observability platform for cloud-native workloads.
• pwru is an eBPF-based tool for tracing network packets in the Linux kernel with advanced filtering capabilities.
• Pixie is an open-source observability tool for Kubernetes applications.

And the list goes on.

Where to next?

If eBPF sounds like your cup of tea, and you’re interested in exploring further, you’re in luck, as many great free resources are available. Here are a few:

Documentation

• BPF Documentation
• BPF & XDP Reference Guide
• BPF Design Q&A

Articles & Blogs

• A thorough introduction to eBPF
• Brendan Gregg's blog
• eBPF’s official community website
• eBPF - From a Programmer’s Perspective
• The Beginner's Guide to eBPF

Tutorials

• bcc Python Developer Tutorial. This tutorial is about developing bcc tools and programs using the Python interface
• Learn eBPF Tracing: Tutorial and Examples.
• The bpftrace One-Liner Tutorial - Learn bpftrace for Linux in 12 easy lessons, where each lesson is a one-liner you can try running

Books

• Learning eBPF
• What is eBPF?
• Linux Observability with BPF
• Security Observability with eBPF

Videos & talks

Communities

• eBPF Slack community channel
• eBPF Stack Overflow
• eBPF reddit community

People to follow

• Brendan Gregg
• Liz Rice
• Alexei Starovoitov
• David S. Miller

That’s it for today. I hope this gave you a good idea of what eBPF is and how you can use it for your next project. There is still SO MUCH to unpack when it comes to eBPF, so I encourage you to go out and explore!

Until next time!

Also published here.