[Utility Post] STDOUT, STDERR, & Redirection for the Perplexed & Curious

(Psst — if you’re busy and want a cheat-sheet see here: ) https://github.com/valgaze/redirection-fun/blob/master/cheatsheet.md Put simply, “redirection” is a method to save output and errors to a file (or suppress them entirely) instead of their default behavior of displaying on the screen. This can be useful because it may sometimes be desirable to store output/error logs for later analysis instead of letting them race by on the terminal. Redirection is also useful when using long-running or noisy . tools If you want to run npm install, for example, and save the output to one file and any errors to a different file, this is all it takes: npm install >npm-log.txt 2>npm-errors.txt [If you’re curious about redirection & npm, see directory of the companion repo: ] /example https://github.com/valgaze/redirection-fun Or imagine there exists some tool called “myapp.” If you knew nothing about “myapp” — not knowing its purpose, functionality or output — is there anything you can say with reasonable certainty about the following command? $ /dev/bin/myapp >logfile.log 2>&1 Probably not much, but if myapp behaves like most other command line programs (and you’re familiar with redirection!), you could at least say the following: Its Standard Output (if any) will be written to logfile.log Its Standard Error (if any) will also be written to logfile.log (mixed in with Standard Output) This note will explain what that means, why this is the case, and detail some other examples of how to perform “redirection” in the terminal. If you want to follow along with the examples, type the following in your command prompt: git clone redirection-fun && cd $_ https://github.com/valgaze/redirection-fun Streams & Redirection Whenever you run a command in the terminal, you have access to three ‘streams’: Standard Input (stdin), Standard Output (stdout), and Standard Error (stderr) In short, whenever you see “good news” from running a command that is probably coming from its stdout (Standard Output) stream and whenever you see any “bad news” (errors or warnings) odds are it is coming from stderr (Standard Error.) By default these so-called “standard streams” will always display right in the terminal once a command is run. But since the streams are standardized and plaintext (ie human-readable) you can in fact “redirect” them elsewhere using a pretty simple but powerful syntax. For instance, when you use the you will see its output (ie a of files) display right in your terminal as expected: “ls” command listing 😎 ~/redirection-fun ls 1.json 3.json 5.json README.md2.json 4.json 6.json example That’s because the stream (Standard Output) is a list of files and the stdout stream selects the terminal as its destination by default. STDOUT If we use redirection, however, instead of displaying in the terminal the output can instead be to a file_:_ redirected 😎 ~/redirection-fun ls 1>output.txt If you inspect output.txt you’ll discover that it now contains that Standard Output from the ls command we saw earlier: 😎 ~/redirection-fun cat output.txt1.json2.json3.json4.json5.json6.jsonREADME.mdexampleoutput.txt In other words, rather than display in the terminal, stdout was “redirected” to output.txt The procedure is the same for errors. If you type the ls command and ask it to list files in a directory that does not exist (here a non-existent directory called “bongo”) you will get an error: 😎 ~/redirection-fun ls bongo/ls: bongo/: No such file or directory The error above is from STDERR and like STDOUT by default its destination is the terminal. We can use redirection to save it to a file named “errors.txt” instead: 😎 ~/redirection-fun ls bongo/ 2>errors.txt If you inspect errors.txt, guess what you’ll find? 😎 ~/redirection-fun cat errors.txtls: bongo/: No such file or directory This is what redirection is all about — redirecting data from its default destination (the terminal) and moving it somewhere else. As you’ll see below, you can send the data from stdout and stderr to a single file, separate files, or discard it entirely. Standard Output If you’re following along at home ( ) try the following inside the project’s main directory: https://github.com/valgaze/redirection-fun , 😎 ~/redirection-fun ls 1>output.txt You should see no output on your screen and if you inspect output.txt it will be filled with the stdout from the ls command (ie what would have displayed in your terminal.) The special number 1 is called a and here it represents Standard Output and can be safely dropped when referring to stdout. All of the following commands are identical to : Important: File Descriptor ls 1>output.txt ls 1>output.txtls 1> output.txtls > output.txtls >output.txt For the rest of this note, we’ll follow the syntax of last item in the list above (no file-descriptor and no space for stdout.) The command above will create the file output.txt if it does not already exist and will overwrite it if it does exist. Note that if you are redirecting to a file to a specific subdirectory (ex ) that subdirectory must already exist otherwise the command will fail. ls > /usr/abcd/output.txt If you want to to an existing file (not overwrite) stdout, you would do the following (two arrows): append 😎 ~/redirection-fun ls >>output.txt If you inspect output.txt you will see that since we ran the command twice, the standard output is doubled: 😎 ~/redirection-fun cat output.txt1.json2.json3.json4.json5.json6.jsonREADME.mdexampleoutput.txt1.json2.json3.json4.json5.json6.jsonREADME.mdexampleoutput.txt We know how to write stdout to a file and append it to an existing file but what if we want to get rid of stdout entirely? How does one run a command completely free of any (standard) output? To discard all stdout data, you should redirect the command to the located on the path “null” device stream /dev/null 😎 ~/redirection-fun ls >/dev/null For an “ ” command like ls, redirecting to /dev/null does not make a lot of sense but for other commands (especially those with long-running and/or noisy output) redirecting to /dev/null is the best way to discard stream data. informational To recap, all you need to know to redirect stdout is: ls >/dev/tty # redirects to terminal, default behavior ls >output.txt # redirects stdout to a file, here output.txt ls >>output.txt # appends stdout to a file, here output.txt ls >/dev/null # discards stdout entirely by redirecting it to /dev/null Standard Error Standard Error (stderr) is similar to stdout but if stdout is most of the “good news” then stderr is all of the “bad news.” Most of the syntax is identical, however, the only change is that stderr’s File Descriptor (the number 2) cannot be dropped. To redirect errors to a file named errors.txt, simply run: 😎 ~/redirection-fun ls bongo 2>errors.txt If you inspect errors.txt you’ll get your error: 😎 ~/redirection-fun cat errors.txtls: bongo: No such file or directory It’s the same story as appending to an existing file: 😎 ~/redirection-fun ls bongo 2>>errors.txt If you inspect errors.txt after appending you’ll see multiple entries as expected: 😎 ~/redirection-fun cat errors.txtls: bongo: No such file or directoryls: bongo: No such file or directory Just like with stdout if you want to discard stderr entirely (neither display on screen nor write to a file), simply redirect stderr to the null device: 😎 ~/redirection-fun ls bongo 2>/dev/null Redirecting Standard Error, besides the mandatory File Descriptor (ie the number 2), will look very similar to redirecting Standard Output: ls 2>stderr.txt # Write stderr to file ls 2>>stderr.txt # Append errors to file ls 2>/dev/null # Discard stderr entirely Combining Stdout & Standard Error Oftentimes you will want stdout and stderr to be captured at the same time. Simple case: separate files (overwrite) 😎 ~/redirection-fun ls >output.txt 2>errors.txt Here stdout is redirected to output.txt and stderr is redirected to errors.txt Simple case: separate files (append, don’t overwrite) 😎 ~/redirection-fun ls >>output.txt 2>>errors.txt Just like above, stdout and stderr are to end of the their new destination files. appended Trickier case: SAME file (overwrite) If you want both stdout AND stderr to share the same file there are a number of approaches. One approach involves “duplicating” the stream using “&num” syntax. You first redirect stdout and then redirect stderr to the same destination by using the & symbol and the number to instruct stderr to go wherever stdout is going, for instance: 😎 ~/redirection-fun ls >shared-output.txt 2>&1 Here stdout (1) is redirected to and stderr (2) is redirected to wherever stdout is going (in this case output.txt) output.txt Note that the following will have an identical outcome to above: 😎 ~/redirection-fun ls 2>shared-output.txt >&2 Here stderr (2) is first redirected to first and then stdout (1) is redirected to stderr’s destination so it too goes to shared-output.txt shared-output.txt If you want to stdout and stderr to the same file using this syntax you would perform either of the following: append 😎 ~/redirection-fun ls >>shared-output.txt 2>&1 😎 ~/redirection-fun ls 2>>shared-output.txt >&2 In the first command, stdout (1) is being redirected in append mode (two arrows) to and stderr (2) follows stdout so it too is being redirected to . In the second command the logic is exactly the same, however, stdout (1) follows stderr’s (2) lead instead. shared-output.txt shared-output.txt A “portable” way to redirect both stdout and sterr to the null device (ie put them in a trash can), you would write something like the following: 😎 ~/redirection-fun ls >/dev/null 2>&1 Danger: Order Matters! The order matters when when combining stdout and stderr to the same destination order. For example, if we intended both stdout and stderr to both share a file named , the following commands would fail: shared-output-2.txt 😎 ~/redirection-fun ls >&2 2>shared-output-2.txt # This first command would fail and in fact displays stdout on the screen when run. The reason is that since order matters, stdout (1) copies stderr’s (the terminal or /dev/tty) and then stderr (2) is redirected to . Stdout was copying stderr's first destination (the terminal, it's default) and not it's final redirected destination (the file shared-output2.txt) initial default destination shared-output-2.txt 😎 ~/redirection-fun ls 2>&1 >shared-output-2.txt # This command would redirect stderr to . Going in order, stderr (2) is redirected to stdout's initial destination (/dev/tty or terminal) and only later is stdout (1) redirected to shared-output-2.txt not shared-output-2.txt To recap, to redirect both stdout and stderr streams… Different destination: 😎 ~/redirection-fun ls >output.txt 2>errors.txt😎 ~/redirection-fun ls >>output.txt 2>>errors.txt # (no overwrite) Same destination: 😎 ~/redirection-fun ls >shared-output.txt 2>&1😎 ~/redirection-fun ls 2>shared-output.txt 1>&2😎 ~/redirection-fun ls >>shared-output.txt 2>&1 # (no overwrite)😎 ~/redirection-fun ls >/dev/null 2>&1 Standard Input (stdin) and piping Standard Input (stdin) Standard Input (stdin), like stdout and stderr, is an abstraction — the details of how the data is generated or where it comes from does not matter. For example, usually the stdin for a program would come from a source like a keyboard, but redirecting stdin makes it possible for a command’s input to come from a file instead: command files.txt # stdout to files.txtgrep json < files.txt # stdin to grep is from files.txt Here lists the current files/folders and instead of stdout displaying on the terminal, stdout is redirected to files.txt. In the next command we run and we redirect stdin to grep. We didn’t redirect ’s stdout so when it runs it will display a list of files (if any exist) with json in their filename in the terminal. ls grep json grep Piping Piping is very similar to redirecting to stdout but instead of writing to a file, piping only redirects stdout as the stdin to another program. Instead of writing above with two commands, we accomplish the same thing by piping: ls | grep json 😎 ~/redirection-fun ls grep | json1.json2.json3.json4.json5.json6.json The command lists files and then its stdout is then “piped” over as stdin whose stdout is not being redirected so it displays in the terminal. ls grep json Fun Example: A bootable Live-CD from a tweet Security engineer Alok Menghrajan crafted a command small enough to squeeze into 280 characters that generates a bootable live-cd for a retro computer game: The command itself a bit messy but for a moment if you take for granted , it’s just piping and redirecting: the Perl command output is piped to a command to encode it in base64 and then the STDOUT of thatbase64 operation is redirected to the file cd.iso— pretty nifty: Alok’s brilliant Perl trick $ perl -E "ALOKS_PERL_TRICK" | base 64 -D >cd.iso That should do it, happy redirecting! Further Reading http://www.linfo.org/standard_output.html http://www.linfo.org/output_redirection_operator.html http://wiki.bash-hackers.org/howto/redirection_tutorial https://ryanstutorials.net/linuxtutorial/piping.php https://www.networkworld.com/article/3025497/linux/sending-data-into-the-void-with-dev-null.html http://www.informit.com/articles/article.aspx?p=2854374&seqNum=5 https://www.quaxio.com/bootable_cd_retro_game_tweet/