Faster, Stronger, and 100% More Custom: A Grand Tour of Linux Kernels

The beating heart of any Linux installation is the kernel. Most distributions that you'll come across add unique features and capabilities to this foundation, but the kernel is the common unifying piece of software across all distributions that calls themselves Linux. Unlike a commercial operating system, the kernel is both very configurable and can be completely customized by any user.

Why would you want to tinker with your kernel configuration? While the stock kernel provided by your distribution is a reasonable option, there are unique sets of patches that can fundamentally change how the kernel behaves and, in some cases, may tune your system to be even more effective for your particular use case.

Let's take a look at the landscape of specialized Linux kernels!

Kernel Basics: The Upstream

The kernel.org web site is the ultimate source of truth for the mainline Linux kernel that the kernel team releases on a regular basis. If you take a look at the landing page, you may see a variety of different kernel versions listed:

If you're a Linux user, you may have never actually seen this page for the entirety of your time actually using Linux. What gives?

A distribution like Ubuntu usually selects a kernel version that's known to be reliable and stable, applies any necessary patch sets in order to customize it as they see fit, and then provides the compiled kernel to all their users so that that actual compilation process (which can take a while!) occurs on their end instead of your desktop's. This means that two up-to-date users of Ubuntu 22.04, for example, are likely using the same kernel build, which simplifies supporting a wide variety of users across a range of different installations. It also permits the engineers developing Ubuntu to focus on providing a single kernel build that is well-tested and meets most of their user's needs.

If you were to compare the running kernel on two different up-to-date Linux distributions, though, their versions would almost certainly be different. While the kernel has an excellent reputation for stability, some distributions choose to be cautious about when to make the jump to a newer kernel. This is typically a good idea - the numbers between two kernel versions like 5.18.16 and 5.10.135 seem large, but kernel developers are conservative about adding and removing features, so there typically aren't massive practical differences between them. If one Linux distribution sticks a few versions behind the latest kernel, doing so in order to achieve more predictable releases is often worth the tradeoff.

But while the default kernel doesn't make massive leaps between releases, different downstream developers maintain collections of changes that can fundamentally alter how the kernel behaves. Some of these include:

• CPU scheduler changes. This is one of the core functions of the kernel and dictates how different processes on your machine are chosen to get executed on your processor.
• I/O scheduler changes. The kernel supports different scheduler methods like deadline or cfq which can make a significant impact upon the performance of disk operations, particularly for spinning disks where the physical realities of spindles and read heads impact how quickly data can be accessed.
• Security. Some patch sets make significant changes to attempt to harden the Linux kernel against potential exploit vectors.

The beauty of running Linux is that all of these different options are completely open to you! Let's take a look at a few of them.

Delicious, Default Vanilla

The default Linux kernel doesn't have any flashy customizations, but there are still quite a few settings that are still under a user's control without any need to recompile their kernel or make changes to the source code.

For example, the aforementioned I/O scheduler dictates how the kernel prioritizes pending operations to either read from or write to persistent storage. This is an important setting for a variety of reasons. Does the kernel push through a document that you're trying to save to the disk first to improve desktop latency, or should it flush the system journal to disk to ensure that its available in the event of a crash? These sort of I/O scheduler decisions factor in a large number of variables like time spent waiting on the kernel, the urgency of the request, and more.

Like nearly all low-level Linux settings, you can directly inspect which I/O scheduler your system is using by using a filesystem path. This information is controlled on a per-device basis using the sys virtual filesystem. On my system, I can find this by inserting the device that I'm interested in, sda, to a path in /sys:

$ cat /sys/block/sda/queue/scheduler
[mq-deadline] kyber none

This indicates that my sda drive is using the mq-deadline scheduler with the other methods, kyber and none, available for use but not actively chosen at this time. The mq-deadline scheduler is described as "[a] good all-rounder with fairly low CPU overhead", which sounds good to me!

You can also change many of these settings using a similarly simple interface. The Linux Kernel documentation explains this well. In short, simply sending the desired scheduler name to the same path as before is sufficient. The following command would change the scheduler for this particular device to use the kyber scheduler. Note that root access is necessary to change these types of settings.

$ echo kyber > /sys/block/sda/queue/scheduler 

Per-device schedulers are just one of the runtime-tunable settings available on a running kernel. Check out these other settings for additional options:

• swappiness
• net.core

Long Term Support Kernels: Oldies, but Goodies

What if new, fancy features seem great, but you're primarily concerned about support and stability?

The long-term support kernel, or LTS, is a version of the Linux kernel specially optimized for support over a much longer period of time than a typical kernel. For systems that favor predictable, stable upgrades, using a kernel package suffixed with -lts is often a very good choice. While no special patches have been applied, using a kernel like 5.15 means that support is guaranteed until October 2023 instead of any earlier. If you were to have adopted kernel 4.9 when it was released in December 2016, it will have been supported until January 2023 – that's a long time to not have to worry about breaking changes!

The long-term support kernel is probably the most widely available kernel variant outside the mainline version. Some distributions like Ubuntu LTS ship with a long-term support kernel by default, but most other distributions provide the option to use an LTS kernel by installing and enabling a package with names similar to linux-lts. Your chosen distribution likely has guides explaining how to install and use different kernels, so to try out the rock-solid LTS kernel, look for kernel packages ending with -lts and follow your chosen distribution's instructions for using an alternate kernel package.

Hardened Kernels: For the Security-minded

Within information security, hardening refers to the process of shoring up specific areas to mitigate potential exploits and vulnerabilities. The mainstream Linux kernel obviously tries to achieve this degree of security as much as reasonably possible, but some users may choose to constrain their kernel and its feature in order to trade them for increased security. The hardened Linux kernel provides this.

Typically, the hardened Linux kernel behaves very similarly to a typical kernel, though there are tradeoffs. For example, some applications may simply not work on a hardened kernel, and many pre-compiled kernel modules may need to use their DKMS equivalent in order to properly integrate with a hardened kernel. Nevertheless, for mission-critical applications where security is a priority, a hardened kernel is often the right choice.

Achieving Enlightenment with Zen

The zen kernel is one of the more widely supported community-based patch sets. While not a massive departure from the upstream Linux kernel, the zen (or Liquorix) kernel makes changes in order to supercharge performance. From the Liquorix web site:

Liquorix is a distro kernel replacement built using the best configuration and kernel sources for desktop, multimedia, and gaming workloads.

One change, for example, is to alter the aforementioned I/O scheduler to select the bfq algorithm by default, which optimizes for system responsiveness. Another set of changes compresses swap storage and uses an alternative caching algorithm to perform well under high memory load.

As one of the more well-established patch sets, there may already be a linux-zen package in your distribution of choice ready to use – however, the project documentation also provides a tutorial explaining how to use this kernel if you need additional options.

Pushing FPS with Gaming Kernels

The linux-tkg kernel is a specialized set of patches specifically designed to optimize gaming performance. Squeezing as many FPS as possible out of a graphics card can be a strenuous exercise, but changes in the tkg patch set can offer additional gains with focused changes to critical pieces of the Linux kernel.

For example, additional CPU schedulers have been added, like the Project C scheduler or the -ck patch set. Network algorithms like Cake are enabled by default, as are optimized compilation flags like -O2 to squeeze as much performance out of compiled code as possible.

The -tkg patch set is less ubiquitous than the LTS or Zen kernels, but the installation steps on GitHub provide a comprehensive guide detailing how to compile and install the custom kernel.

The XanMod patch set provides a similar set of features targeting users that require a low-latency, high-performance kernel suitable for use cases like gaming. In either case, the tkg and XanMod kernels offer a very specialized form of Linux kernel that can still function as a day-to-day operating system with additional benefits on gaming machines.

Real-time Kernels

The term "real-time computing" refers to the concept of a system that aims to guarantee that certain operations run an a given window of time. For example, a system that controls the brakes of a car or the wings of an aircraft cannot afford to be interrupted by a flush of pending writes to a disk or drawing images on a screen.

The Real Time Linux project provides a set of patches that offer this functionality. While these features are often very specialized and not typically suited for desktop or server applications, they can nevertheless be critically important in the right situation. For projects that have applications in particular industries, these real-time patches are a necessity.

The suite of -rt kernels have a much more targeted audience but are often more widespread than kernels optimized for gaming or low-latency due to their importance where they remain in-use. Your distribution's package manager may provide these types of kernels under names like linux-rt or linux-rt-lts.

Hybrid Kernels

There are many more specialized kernels out there that we explore, but we'd be here all day! With that said, there's even more ways to customize a kernel using the different types we've talked about already.

An interesting trait of many of these patches is that they're often combined into kernels that blend their features into one kernel source tree. For example, a critical piece of operational machining infrastructure may require the real-time capabilities of an -rt kernel but also benefit from the additional security of a hardened kernel. As long as different patches can be applied cleanly together, nothing stops you from mixing and matching kernels tailored to your exact needs.

Some distributions provide this simply by combining patch sets into the kernel's package name. For example, the Arch Linux User repository provides the linux-rt-lts package, which combines the real-time and long-term support kernel features. Ubuntu provides packages like linux-image-*-aws specialized with various changes to optimize the kernel for operation in AWS EC2.

Regardless of your chosen distribution, if you're interested in running a custom kernel, be sure to explore your distribution's packages for available kernels. They provide the simplest and quickest way to get started with a specialized version of the Linux kernel.

Go forth and Tinker!

This article has only scratched the surface of the Linux kernel landscape. While many of these kernels are widespread and commonly found in the majority of distribution repositories, some more esoteric patch sets exist that push the kernel even further. With an open-source kernel, you could even modify the kernel yourself to compile one custom to your own needs, but that's a guide for another time.

Until then, check out some of these resources for more information and find a kernel that you'd like to try for yourself!

• ArchWiki - Kernel
• Ubuntu kernels from Canonical
• The Linux Kernel Archives