February 13, 2006

CLI Magic: MultiTail follows files in style

Author: Joe 'Zonker' Brockmeier

Troubleshooting often involves having to watch logfiles in real time. That means using tail or a similar utility to see new messages that are added to a logfile by Apache, MySQL, X.org, or whatever program you're trying to deal with at the time. While tail is usually readily available on *nix systems, I prefer to use MultiTail whenever possible. It has some features that you won't find in tail, such as filtering and a color display, and MultiTail allows you to follow the output from a command as easily as following a logfile.

If MultiTail isn't already installed on your system, the MultiTail download page has source code and packages for Red Hat, Fedora Core, Slackware, Solaris, and Mac OS X 10.2 or higher. If you're a Debian, Ubuntu, or SUSE user, you should be able to find packages via your distro's package repositories or CDs. For Debian and Ubuntu users, a simple apt-get install multitail should install the package.

To start with the software, type multitail at the console or in an xterm. You don't need to specify a filename or filenames to run MultiTail, though you can if you wish to. MultiTail is an interactive program that lets you specify filenames on the fly.

When you first run MultiTail with no arguments, you'll see a message indicating the version of MultiTail, inviting you to press F1 for help. Scroll through the help options to get a feel for the commands you can run.

Let's start by specifying a file. Press a and MultiTail will prompt you to choose f to add a window for a file or c to add a window to follow a command.

Let's start with a file. Enter f and pick any logfile, such as /var/log/messages. You'll notice that MultiTail allows you to complete filenames by using the Tab key -- just as if you were working at the shell -- and it will show you a list of one or more possible filenames if you enter the first few characters and press Tab.

After you've chosen a filename, MultiTail will ask if you want to follow the file, and what type of colors you'd like to use. Choices include syslog, misc, field, none, and Scheme. The Scheme option offers color schemes for Apache logs, Apache error logs, system logs, and postfix logs. I've found the syslog and Apache color schemes to be useful; you might want to test them out for yourself and see which ones you prefer. You can define new color schemes in /etc/multitail.conf. See the introduction to MultiTail for more on writing your own color scheme.

Once you've chosen your colors, MultiTail will display the file or command output. Now you can further tweak the display to narrow down results. Let's say you want a visual clue when an error message is detected in the logfile. With MultiTail open, press e. MultiTail will prompt you to add, edit, or delete a regular expression. Select add, and enter the regular expression you'd like to match, and whether you'd like to negate the regular expression.

Choosing "negate" can be useful, for example, for weeding out non-error messages in logfiles. After that, you'll be able to choose the usage of the expression. For example, if you choose "match," then MultiTail will display lines that match your regular expression -- and no others. If you choose colorize, MultiTail will display everything, but it will display lines that match in a different color. Once you've made your selection, press q to exit the regular expression screen.

If the regular expression matches, you should see an immediate change in MultiTail's display -- it applies the regular expression to all lines, not just new lines.

If the top/bottom display doesn't work for you, that's no problem -- MultiTail also has a side-by-side mode that you can invoke using the -s option. For example, multitail -s /var/log/logfile1 /var/log/logfile2 will display logfile1 in the left side of the display, and logfile2 in the right side. Side-by-side displays can be easier to read if you have a widescreen monitor and maximize your terminal emulator window.

If you'd like to see how long MultiTail has been running, or how much memory it has been using, press t to bring up its statistics window.

If you want to follow the output of a command, that's simple as well. While in MultiTail, press a, and MultiTail will ask if you want to add to the existing window. I usually choose n so that I can keep the displays distinct. Then, select c for command and specify the command that you want to run. One good command with which to test MultiTail out would be ping. Enter ping hostname. MultiTail will prompt for the colors you'd like to use, and then the interval to repeat the command. The interval is useful for programs that run once and exit, without giving you the option of setting an interval.

Once you've gone through all the options, MultiTail will start following the output of the command until you delete the window or quit MultiTail altogether.

If you need to go back through the output, select b while in MultiTail. The program will ask which window you'd like to scroll back through, then display that window in a sub-window that allows you to scroll backward through the file or command output. By default, MultiTail buffers 100 lines, but you can modify this with the m command or set the value at startup with the -m option. Set the buffer to 0 if you want the buffer to be unlimited in size.

Once you've gotten MultiTail tweaked just the way you like it, you can avoid having to spend time resetting all of the options again. While in MultiTail, press w to write a shell script that lets you start MultiTail with the same settings. Pretty handy stuff, no?

MultiTail is a very powerful program, and it's far too full-featured to cover every possible use here. For example, it's possible to specify MultiTail's parameters on the command line rather than using the interactive display. Check the MultiTail man page for the command line options. Also, be sure to check the examples page for some suggested uses of MultiTail.

Most importantly, just fire up MultiTail when you have some time and poke around to see what you can do with it. After a little while, I'll bet you'll find it as useful as I do.

Click Here!