20,000 Leagues Under Your Shell

Over time most Linux aficionados accrue a sparkling war chest full of hard-won tricks that can come in tremendously handy when a situation demands quick thinking at your terminal. I stashed away many of these bits of knowledge over the years and whenever I had the opportunity to watch over the shoulder of somebody exceptionally well-versed with Linux.

Today, you're pairing with me at the terminal. We're exploring the depths of Linux filesystem and shell tools and tricks.

Take a /proc with me

One of the most useful directories in a Linux system is /proc. From the man page for proc:

The proc filesystem is a pseudo-filesystem which provides an interface to kernel data structures.

When the man page says "pseudo-filesystem", it means that if you were to peek underneath your disk where you might expect to find bits representing a file like you would for a text file at /tmp/launch-codes.txt, there's nothing at /proc. It's present and alive on a running Linux system, but entirely absent if you were to pull the disk out and inspect it. /proc a control panel for your running kernel!

If you were to take a look in your own /proc right now, you might find a lot of directories like the following:

ls /proc

1
10
10021
10059
10144
...hundreds more files...

Each of those numbers represents a process ID, or PID – yes, the same PID that identifies the process for your browser or terminal program. In fact, you can interrogate lots of information about the process itself. For example, you might recall that process 1 on a Linux system is traditionally the top-level init process, which in most modern systems is systemd-based. Let's see the command that kicked off PID 1 on my system:

cat /proc/1/cmdline

/run/current-system/systemd/lib/systemd/systemd

cmdline is a file that tells us the command that kicked off process 1 - in this case, systemd itself.

There's a cmdline file in /proc that is particularly useful - /proc/cmdline, which actually shows you the arguments passed to your kernel itself at boot time. Mine is very wordy, but tells me the initrd that my system booted with, along with any other flags, which in my case is init and loglevel:

cat /proc/cmdline

initrd=\efi\nixos\hx5g5rmvq748m64r32yjmpjk3pmgqmr1-initrd-linux-5.17.11-initrd.efi init=/nix/store/9zvklk45yx41pak2hdxsxmmnq12n712k-nixos-system-diesel-22.05.20220604.d9794b0/init loglevel=4

My NixOS hostname is diesel. Please note that I do not put petroleum in my laptop.

/proc isn't just read-only, either. Like its man page says, /proc is an interface to the kernel, which includes interacting with the kernel itself. The /proc/sys directory holds a variety of knobs and dials, but I want to show you /proc/sys/vm, which lets us peek at the kernel's virtual memory. Want to get more adventurous?

Consider my machine's current memory usage.

free -h

              total        used        free      shared  buff/cache   available
Mem:           31Gi        22Gi       3.0Gi       4.4Gi       5.6Gi       3.6Gi
Swap:          31Gi       130Mi        31Gi

Nothing too unusual here – but what if I wanted to free up my memory aggressively? Most of the time, the kernel knows best when it comes to using memory for caching, but there are some situations in which you may want to clear any memory that is safe to clear – we don't want to break any running processes, just reclaim memory if possible.

It turns out that there's a file for that. We pipe an echo command into sudo tee because /proc/sys/vm is usually write-protected and only root can write to the file we're interested in.

echo 1 | sudo tee -a /proc/sys/vm/drop_caches

What this command does is effectively signal to the kernel, "please drop any caches in memory that you can afford to lose without breaking any running processes on my system." On my machine this opens up about 500M of memory:

              total        used        free      shared  buff/cache   available
Mem:           31Gi        22Gi       3.5Gi       4.4Gi       5.1Gi       3.6Gi
Swap:          31Gi       130Mi        31Gi

Cool! There are all sorts of useful file-like objects in /proc that can do interesting things like this. Feel free to open up man proc if you'd like to learn more.

/dev as a prehistoric curl

A character device like /dev/sda represents an attached disk, but there's another use for the /dev path: a little-known way to send network requests.

The path /dev/tcp isn't actually a file-like device exposed by the Linux kernel, but actually a feature of your chosen shell like bash. Shells can intercept operations on this path in order to open up low-level socket connections to remote endpoints like web servers listening on port 80.

To begin, open a new file descriptor connected to a file path in /dev/tcp that indicates the desired endpoint and port. Much like how file descriptor numbers 0, 1, and 2 represent stdin, stdout, and stderr, respectively, you can think of this new file descriptor 3 as representing a pipe to a remote network endpoint. We'll assume the use of bash from here on out.

exec 3<>/dev/tcp/httpbin.org/80

Next, send the plaintext form of a simple HTTP request to the open file descriptor. This GET request to /status/200 also requires the Host header to be set in order to be properly handled by most reverse proxies. Two newlines signal termination of the request:

echo -e "GET /status/200 HTTP/1.1\r\nHost: httpbin.org\r\n\r\n" >&3

Finally, a simple read operation retrieves the HTTP response:

cat <&3

You should see a response similar to the one below:

HTTP/1.1 200 OK
Date: Fri, 10 Jun 2022 21:39:43 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 0
Connection: keep-alive
Server: gunicorn/19.9.0
Access-Control-Allow-Origin: *
Access-Control-Allow-Credentials: true

Congratulations! You've just sent an HTTP request using nothing but your shell.

Swimming in /sys

There's one more root-level directory to explore after diving into /proc and /dev: the enigmatic /sys directory.

Like /proc and /dev, /sys is another file-like interface to low-level mechanisms that sit very close to the operating system. Unlike /proc - which is relatively process-focused - and /dev - which models block devices and more - /sys is a useful interface to many abstractions that the kernel models.

For example, take the directory /sys/class/net. Inside of this directory, you'll find a list of links that represent the network interfaces on your host. Here's what mine looks like:

ls /sys/class/net

enp0s20f0u6u4u1
lo
tailscale0
wlan0

As you can see, the active network connections that my system is managing include a wired interface (the interface that begins with en), the lo loopback interface, a Tailscale interface, and my wireless interface wlan0. Listing the contents of one of these directories reveals a long list of files, but let's look more closely at two files in particular for my wired network interface:

cat /sys/class/net/enp0s20f0u6u4u1/statistics/rx_bytes
cat /sys/class/net/enp0s20f0u6u4u1/statistics/tx_bytes

11281235262
274308842

Each of these files represents the number of received bytes and transmitted bytes, respectively. Check out how the numbers change if I use the same command a few seconds later:

cat /sys/class/net/enp0s20f0u6u4u1/statistics/rx_bytes
cat /sys/class/net/enp0s20f0u6u4u1/statistics/tx_bytes

11289633209
274760138

Bigger numbers! Apparently I'm making the most of my bandwidth. How is this useful?

Have you ever wondered how network usage widgets are written? Well, how about making your own?

Check out this small bash script that uses the aforementioned files in the statistics directory to derive a network activity rate.

interval=1
interface=$1

rx_bytes=$(cat /sys/class/net/$interface/statistics/rx_bytes)
tx_bytes=$(cat /sys/class/net/$interface/statistics/tx_bytes)

rx_bytes_rate=0
tx_bytes_rate=0

function fmt() {
  numfmt --to=iec-i --suffix=B $1
}

while true
do
        echo -en " $(fmt $tx_bytes_rate)/s ⬆ $(fmt $rx_bytes_rate)/s ⬇\t\r"
        sleep $interval
        old_rx_bytes=$rx_bytes
        old_tx_bytes=$tx_bytes
        rx_bytes=$(cat /sys/class/net/$interface/statistics/rx_bytes)
        tx_bytes=$(cat /sys/class/net/$interface/statistics/tx_bytes)
        tx_bytes_rate=$(( ($tx_bytes - $old_tx_bytes) / $interval ))
        rx_bytes_rate=$(( ($rx_bytes - $old_rx_bytes) / $interval ))
done

You can place this script somewhere in your $PATH, make it executable with chmod +x <script>, and try it out with script.sh <interface name>. Here's what the output looks like on my machine:

13KiB/s ⬆  379KiB/s ⬇

That's pretty cool! You can imagine some uses for this: for example, as a widget for a tool that can render command output or as a quick way to look at network activity for a particular network interface. In either case, the file-based interface to this data makes accessing and using it exceptionally easy.

Further Exploration

This was just a small dive into the types of information available to you as you look further into the capabilities of a modern Linux system. You can search for additional guides like this one or go straight to the source by reading man pages for entries like man hier in order to read the function and purpose of various directories in /.

Have fun exploring!