Since the early days of Unix, the shell has been part of the user's interface with the operating system. The first Unix shell (the ) had very limited features, mainly I/O redirection and command pipelines. Later shells expanded on that early shell and added more and more capabilities, which gave us powerful features that include word expansion, history substitution, loops and conditional expressions, among many others. Thompson shell Why This Tutorial Over the past 20 years, I've been using GNU/Linux as my main operating system. I've used many GNU/Linux shells, including but not limited to , , and . However, I've always been bugged by this question: Like, for example: bash ksh zsh what makes the shell tick? How does the shell parse my commands, convert them to executable instructions, and then perform these commands? How does the shell perform the different word expansion procedures, such as parameter expansion, command substitution, and arithmetic expansion? How does the shell implement I/O redirection? ... and so on. As most GNU/Linux shells are open-sourced, if you want to learn the inner workings of the shell, you can search online for the source code and start digging in (that's what I actually did). But this advice is actually easier said than done. For example, where exactly should you start reading the code from? Which source files contain the code that implements I/O redirection? Where can I find the code that parses user commands? I guess you got the point. This is why I’ve decided to write this tutorial, to help Linux users and programmers gain a better understanding of their shells. Together, we are going to implement a fully functional Linux shell, Along the way, we'll see how a Linux shell manages to parse and execute commands, loops, and conditional expressions by actually writing the C code that does the above tasks. We’ll talk about word expansions and I/O redirection, and we’ll see the code that performs features. from scratch. By the end of this tutorial, we’ll have a basic Linux shell, that will not do much for now, but which we’ll expand and improve in the next parts. At the end of this series, we’ll have a fully functional Linux shell that can parse and execute a fairly complex set of commands, loops, and expressions. What You Will Need In order to follow this tutorial, you will need the following: A working GNU/Linux system (I personally use and , but feel free to use your favorite Linux distribution). Ubuntu Fedora (the GNU Compiler Collection) to compile the code. GCC A text editor to write the code (I personally use , but you can use , , or any other editor as well). GEdit Vim Emacs How to program in C. I'm not going to dive into the details of installing the required software here. If you are not sure how to get your system running any of the above software packages, please refer to your Linux distribution's documentation and make sure you have everything set up before going further. Now let's get down to business. We’ll start by having a bird’s eye view of what constitutes a Linux shell. Components of a Linux Shell The shell is a complex piece of software that contains many different parts. The core part of any Linux shell is the or . This part serves two purposes: it reads and parses user commands, then it executes the parsed commands. You can think of the CLI itself as having two parts: a (or front-end), and an (or back-end). Command Line Interpreter, CLI parser executor The scans input and breaks it down to tokens. A consists of one or more characters (letters, digits, symbols), and represents a single unit of input. For example, a token can be a variable name, a keyword, a number, or an arithmetic operator. parser token The takes these tokens, groups them together, and creates a special structure we call the , or . You can think of the AST as a high level representation of the command line you gave to the shell. The parser takes the AST and passes it to the , which reads the AST and executes the parsed command. parser Abstract Syntax Tree AST executor Another part of the shell is the user interface, which usually operates when the shell is in the , for example, when you are entering commands at the shell prompt. Here the shell runs in a loop, which we know as the , or interactive mode Read-Eval-Print-Loop REPL. As the loop's name indicates, the shell reads input, parses and executes it, then loops to read the next command, and so on until you enter a command such as , , or . exit shutdown reboot Most shells implement a structure known as the , which the shell uses to store information about variables, along with their values and attributes. We'll implement the symbol table in part II of this tutorial. symbol table Linux shells also have a history facility, which allows the user to access the most recently entered commands, then edit and re-execute commands without much typing. A shell can also contain , which are a special set of commands that are implemented as part of the shell program itself. builtin utilities Builtin utilities include commonly used commands, such as , , and . We'll implement many of the builtin utilities as we move along with this tutorial. cd fg bg Now that we know the basic components of a typical Linux shell, let's start building our own shell. Our First Shell Our first version of the shell won't do anything fancy; it will just print a prompt string, read a line of input, then echo the input back to the screen. In subsequent parts of this tutorial, we’ll add the capability to parse and execute commands, loops, conditional expressions, and much more. Let's start by creating a directory for this project. I usually use for my new projects, but feel free to use whatever path you're comfortable with. ~/projects/ The first thing we'll do is to write our basic REPL loop. Create a file named (using ), then open it using your favorite text editor. Enter the following code in your file: main.c touch main.c main.c { *cmd; { print_prompt1(); cmd = read_cmd(); (!cmd) { (EXIT_SUCCESS); } (cmd[ ] == || (cmd, ) == ) { (cmd); ; } ( (cmd, ) == ) { (cmd); ; } ( , cmd); (cmd); } ( ); (EXIT_SUCCESS); } # include <stdio.h> # include <stdlib.h> # include <errno.h> # include <string.h> # include "shell.h" int main ( argc, **argv) int char char do if exit if 0 '\0' strcmp "\n" 0 free continue if strcmp "exit\n" 0 free break printf "%s\n" free while 1 exit Our function is quite simple, as it only needs to implement the REPL loop. We first prints the shell's prompt, then we read a command (for now, let's define a command as an input line ending with ). If there's an error reading the command, we exit the shell. If the command is empty (i.e. the user pressed without writing anything), we skip this input and continue with the loop. main() \n ENTER If the command is , we exit the shell. Otherwise, we echo back the command, free the memory we used to store the command, and continue with the loop. Pretty simple, isn't it? exit Our function calls two custom functions, and . The first function prints the prompt string, and the second one reads the next line of input. Let’s have a closer look at those two functions. main() print_prompt1() read_cmd() Printing Prompt Strings We said that the shell prints a prompt string before reading each command. In fact, there are five different types of prompt string: , , , , and . The zeroth string, , is only used by , so we won’t consider it here. The other four strings are printed at certain times, when the shell wants to convey certain messages to the user. PS0 PS1 PS2 PS3 PS4 PS0 bash In this section, we’ll talk about and . The rest will come later on when we discuss more advanced shell topics. PS1 PS2 Now create the source file and enter the following code: prompt.c { ( , ); } { ( , ); } # include <stdio.h> # include "shell.h" void print_prompt1 ( ) void fprintf stderr "$ " void print_prompt2 ( ) void fprintf stderr "> " The first function prints the or , which you usually see when the shell is waiting for you to enter a command. The second function prints the or , which is printed by the shell when you enter a multi-line command (more on this below). first prompt string, PS1 second prompt string, PS2 Next, let’s read some user input. Reading User Input Open the file and enter the following code at the end, right after the function: main.c main() { buf[ ]; *ptr = ; ptrlen = ; (fgets(buf, , )) { buflen = (buf); (!ptr) { ptr = (buflen+ ); } { *ptr2 = (ptr, ptrlen+buflen+ ); (ptr2) { ptr = ptr2; } { (ptr); ptr = ; } } (!ptr) { ( , , strerror(errno)); ; } (ptr+ptrlen, buf); (buf[buflen ] == ) { (buflen == || buf[buflen ] != ) { ptr; } ptr[ptrlen+buflen ] = ; buflen -= ; print_prompt2(); } ptrlen += buflen; } ptr; } * char read_cmd ( ) void char 1024 char NULL char 0 while 1024 stdin int strlen if malloc 1 else char realloc 1 if else free NULL if fprintf stderr "error: failed to alloc buffer: %s\n" return NULL strcpy if -1 '\n' if 1 -2 '\\' return -2 '\0' 2 return Here we read input from in 1024-byte chunks and store the input in a buffer. The first time we read input (the first chunk for the current command), we create our buffer using . For subsequent chunks, we extend the buffer using . We shouldn’t encounter any memory issues here, but if something wrong happens, we print an error message and return . If everything goes well, we copy the chunk of input we’ve just read from the user to our buffer, and we adjust our pointers accordingly. stdin malloc() realloc() NULL The final block of code is interesting. To understand why we need this block of code, let’s consider the following example. Let’s say you want to enter a really, long line of input: really echo "This is a very long line of input, one that needs to span two, three, or perhaps even more lines of input, so that we can feed it to the shell" This is a silly example, but it perfectly demonstrates what we’re talking about. To enter such a long command, we can write the whole thing in one line (as we did here), which is a cumbersome and ugly process. Or we can chop the line into smaller pieces and feed those pieces to the shell, one piece at a time: echo "This is a very long line of input, \ one that needs to span two, three, \ or perhaps even more lines of input, \ so that we can feed it to the shell" After typing the first line, and to let the shell know we didn’t finish our input, we terminate each line with a backslash character , followed by newline (I also indented the lines to make them more readable). We call this the newline character. When the shell sees the escaped newline, it knows it needs to discard the two characters and continue reading input. \\ escaping Now let’s go back to our function. We were discussing the last block of code, the one that reads: read_cmd() (buf[buflen ] == ) { (buflen == || buf[buflen ] != ) { ptr; } ptr[ptrlen+buflen ] = ; buflen -= ; print_prompt2(); } if -1 '\n' if 1 -2 '\\' return -2 '\0' 2 Here, we check to see if the input we’ve got in the buffer ends with and, if so, if the is by a backslash character . If the last is not escaped, the input line is complete and we return it to the function. Otherwise, we remove the two characters ( and ), print out , and continue reading input. \n \n escaped \\ \n main() \\ \n PS2 Compiling the Shell With the above code, our niche shell is almost ready to be compiled. We’ll just add a header file with our function prototypes, before we proceed to compile the shell. This step is optional, but it greatly improves our code readability, and prevents a few compiler warnings. Create the source file , and enter the following code: shell.h ; ; ; # SHELL_H ifndef # SHELL_H define void print_prompt1 ( ) void void print_prompt2 ( ) void * char read_cmd ( ) void # endif Now let’s compile the shell. Open your favorite terminal emulator (I test my command line projects using and , but you can as well use , other terminal emulators, or one of Linux’s ). Navigate to your source directory and make sure you have 3 files in there: GNOME Terminal Konsole XTerm virtual consoles Now compile the shell using the following command: gcc -o shell main.c prompt.c If everything goes well, should not output anything, and there should be an executable file named in the current directory: gcc shell Now invoke the shell by running , and try entering a few commands: ./shell In the first case, the shell prints , which defaults to and a space. We enter our command, , which the shell echoes back to us (we’ll extend our shell in part II to enable it to parse and execute this — and other—simple commands). PS1 $ echo Hello World In the second case, the shell again echoes our (slightly long) command. In the third case, we split the long command into 4 lines. Notice how every time we type a backslash followed by , the shell prints and continues to read input. After the last line is entered, the shell amalgamates all the lines, removing all escaped newline characters, and echoes the command back to us. ENTER PS2 To exit from the shell, type , followed by : exit ENTER And that’s it! We’ve just finished writing our very first Linux shell. Yay! What's Next Although our shell currently works, it doesn’t do anything useful. In the next part, we’ll fix our shell to make it able to parse and execute simple commands. Stay tuned!
![featured image - Let's Build a Linux Shell [Part I]](https://images.unsplash.com/photo-1518432031352-d6fc5c10da5a?ixlib=rb-1.2.1&q=80&fm=jpg&crop=entropy&cs=tinysrgb&w=1080&fit=max&ixid=eyJhcHBfaWQiOjEwMDk2Mn0&auto=format)
![Building a Linux Shell [Part V]](https://firebasestorage.googleapis.com/v0/b/hackernoon-app.appspot.com/o/images%2FRehfvhVduMecnl6qtW8ev7jN1H72-8ga231o.jpeg?alt=media&token=033fd9d0-825f-48e9-98cb-48822ad0ff7c&auto=format&fit=max&w=3840)
