May 3, 2004

CLI magic: need redirection?

Author: Joe Barr

OK, noobies, wake up from that GUI-induced stupor you're in and follow me. This week's column is going to take you in new directions -- or should I say redirections -- on the command line interface. You're probably already familiar with standard transmissions, Standard Oil, and standard rates. For the momement, forget all about those standards and focus on these: STDIN, STDOUT, and STDERR.

You may recall from an earlier column about files and streams that by default the STDIN (standard input) stream comes from your keyboard, and STDOUT (standard output) goes to your terminal. That's where the process reading STDIN and writing STDOUT normally puts STDERR (standard error) as well. Let's take a look at a few ways we can redirect those standard streams.

Less than:

The left angle bracket, or "less than" symbol, shown above is used to tell a process to look elsewhere for its standard input. Where should it look? To the file name following the symbol. You can insert a space or not between the "less than" symbol and the file name, depending on how you feel. It doesn't matter which form you use, the result is the same.

For example:

grep -i needle <haystack.txt


grep -i needle

both produce the same results.

More than: >

The right angle bracket, or "more than" symbol, shown above is used to tell a process to redirect its standard output from the console terminal to the file name which follows it. This usage doesn't give a whit about whether a file by the name already exists or not: if one does, it is overwritten. Just as with the "less than" symbol, it doesn't matter whether you leave a space between the symbol and the file name or not.

The following example extends our previous grep command to redirect the results to a file named "search.results".

grep -i needle search.results

More more than: >>

Using two "more than" symbols instead of just one prevents previously existing file data from being overwritten. Instead, if a file with the same name specified after the two "more thans" already exists, standard output is appended to it. Like this:

grep -i spindle >search.results

What about STDERR?

Redirecting standard error output from a process is a little bit trickier. To do that, you have to specify its file descriptor number. But don't worry, that's as easy as 1-2-3. Except starting from zero instead of one.

When a process is executed, standard input is assigned as file descriptor 0, standard output as file descriptor 1, and standard error as file descriptor 2. In order to redirect standard error, we use the same symbol ("more than" or double "more thans") used above to redirect standard output plus its file descriptor number.

Let's try it and see what happens. We'll change the name of the file we're searching from "haystack.txt" to "nofile.txt", and add the redirection for standard error, aka FD 2.

grep -i spindle >search.results 2>error.log

Whoops. That didn't look. I got the following error message on my screen:

bash: nofile.txt: No such file or directory

Aha! I see the problem. The Bash shell reported the error as it evaluated the command line, so our process (grep) never got a chance to run. We need to find an error that will get by Bash but trigger an alarm in grep. Let's try using an invalid option, like this:

grep -J spindle >search.results 2>error.log

That time it worked just as planned. Note that instead of specifying a file name, we can also just tell the process to put STDERR together with STDOUT. Like this:

grep -J spindle >search.results 2>&1

Replacing the file name with "&1" did the trick.

Put this in your pipe and smoke it

I've been putting this last one off. It's probably the most common means of redirection, but we've already talked about it a little bit in a previous column. It's the elegant and powerful "|" operator, which allows you to "pipe" the standard output from one process to the standard input of another process.

Let's say you have a of friends and acquaintances named Bob, so you know the results of a grep for Bob on phones.txt will produce a lot of matches that you will have to scroll through in order to find the right Bob. Redirecting grep's standard output to less will solve the problem nicely. It's as easy as:

grep -i Bob

One word of caution: don't be like me and confuse command line options with standard input! When in doubt, use man to see which is which for the program in question.

Click Here!