May 12, 2006

My sysadmin toolbox

Author: David Martinjak

I started running Linux while pursuing my undergraduate degree in college. My first career position using Linux was as a security engineer building and administering systems in an enterprise environment for an insurance corporation. Since then, I've constructed a slightly unorthodox toolbox for both home and work-related operations, with security being a primary focus.


I see a lot of people criticize others for using Telnet, but then proceed to use netcat. Why use plaintext? Cryptcat is a great open source utility available under the GNU General Public License (GPL) that can read and write data across TCP and UDP network connections using Twofish encryption. It has ports for Windows NT, BSD, and Linux.

Cryptcat comes in handy when I am working on a system that does not have SSH on it. I can open up two shells on my workstation with Screen or Konsole and begin listening with Cryptcat. Then I can log in to the remote system and run an instance of Cryptcat there, but pipe it through bash, then back out to Cryptcat. The end result is a lower-security makeshift SSH. Here is an example:

workstation$ cryptcat -vv -l -p 5555 -k mysecretkey
workstation-out$ cryptcat -vv -l -p 4444 -k mysecretkey
server$ sleep 300 | cryptcat workstation 5555 -k mysecretkey | /bin/bash | cryptcat workstation 4444 -k mysecretkey

When I go back to my workstation after running those commands, I can just type commands into the workstation prompt and see the output from the server on the workstation-out shell. One note is that cryptcat defaults the key to "metallica" if you don't pass it a secret key with the -k option.

Of course, you'll find SSH on most systems these days -- but Cryptcat is every bit as useful as netcat, with the added bonus of encryption thrown.

Bash completion

Bash is my shell of choice, in part because its programmable completion puts bash ahead of other shells that I've used. Why should tab completion stop where the $PATH ends? The default bash_completion file should suffice for most users. You can, however, build your own custom completion file to reduce your keystrokes. Run the following command to employ bash completion each time a shell is spawned:

echo ". /etc/bash_completion" >> ~/.bashrc

One thing that bash completion can fill in for you is a host name or IP address. For example, we all know that Tab completes filenames and command names -- but did you know you could use Esc-@ to complete hostnames?

Sometimes our admins used obscure names or names that were similar to each other for our systems at work. Other times I worked on systems and only knew their IP addresses. A week or two later, I would be trying to remember what the name of the system was or which IP I need to connect to. Fortunately I was able to add host name and IP address completion for Cryptcat into bash completion:

sudo echo "complete -F _known_hosts cryptcat" >> /etc/bash_completion


Everyone knows OpenSSH, and most people are probably familiar with the brute force attacks that target systems running OpenSSH. You can take several approaches to dealing with this situation, and many are covered in a thread over at LinuxQuestions.

I take several steps to avoid issues with the SSH brute force attacks, and have modified the sshd_config for this:

PermitRootLogin no
Port 1111
PermitEmptyPasswords no
PasswordAuthentication no
PubkeyAuthentication yes
Protocol 2

These changes restrict access via SSH. The first line directs SSH to listen only on a given interface, and only accept connections there. Changing the listening port (when possible) is probably the easiest way to avoid brute force attacks on SSH. Changing the port that SSH listens on won't prevent an attacker from finding SSH, but it will render most bots and worms useless. Let them pound on port 22 all they want -- nothing's there.

Setting the PasswordAuthentication to no is a significant factor when discussing access control. OpenSSH can also be configured to force user authentication through public/private key pairs, which helps to limit the potential field of attackers.


One of my favorite quotes from Linus Torvalds is "intelligence is the ability to avoid doing work, yet getting the work done." I don't claim to have a great level of intelligence, but I do enjoy getting work done while avoiding doing work.

One of the ways I do this is to patch all of my kernels with grsecurity/PaX, which is the best safety net I've found in protecting my systems from overflows. You can find a good start guide on the grsecurity site. Once you've downloaded the grsecurity patch and corresponding kernel source, all you have to do is patch, reconfigure, and compile your kernel. To apply the patch, cd to the source directory, make sure that you have the grsecurity patch file and the kernel source, and run the command:

# patch -p0 > grsecurity-2.1.9-

You can then configure your kernel:

# cd linux- ; make menuconfig

Enable most of the grsecurity and PaX options in the configuration, but read the help for each item if you have questions. One note is that you cannot enable the option for Disable privileged I/O if you want to run X Windows. You can test your resistance to overflows using the paxtest utility.

I also recommend libsafe to stop return-to-libc attacks, but it is maintained separately from grsecurity/PaX.


Vim is my editor of choice simply because I've used vi variants since I started using Linux. Its consistency and standardization make Vim straightforward to use across different platforms and distributions.

Just the other day, one of the developers had a mess of ColdFusion code to modify. We were able to make all of the changes in about 30 seconds using a substitution in Vim. It probably would have taken him at least an hour to change each of these lines by hand otherwise. Here's what we did:

:%s/^.\+"\(Crew_Special_signup\..\+\)".\+$/<cfif findNoCase("\1",form\.walkerSelected)>, \1/g

Fairly simple regular expression, and it worked great.


Version control isn't just for developers. I maintain all sorts of things in version control: configuration files, proposals, even my résumé. Learning or migrating to Subversion is simple, and you can choose from among several GUI clients for it. Even beginners can have their project maintained in a repository in minutes. Subversion really does a great job, and comes with thorough documentation.

I have a small script to create a repository and check out a working copy for a new project that you might find useful:

# usage: mkproject newprojectname
mkdir -p $1/branches $1/tags $1/trunk                         # create initial directory
svnadmin create /path/to/repos/$1                             # create repository
svn import $1 file:///path/to/repos/$1 -m "Initial import"    # import the initial directory
rm -rf $1                                                     # remove the initial directory
svn checkout file:///path/to/repos/$1/trunk ~/$1              # checkout a working directory


Nagios is a monitoring program that helps me keep tabs on systems and services on my network. It is modular and easy to write plugins for. It has many configurable options; you can even set it to page or call you with alerts. Nagios also scales to enterprise production environments; I ran it in a Fortune 500 company.

The following plugin example is a bit unorthodox, especially for being in a Fortune 500 company, but it displays the flexibility that Nagios provides and the simplicity of writing plugins.

Phrack was my favorite zine and that of a few other security engineers. Unfortunately the releases were spread out, and we got tired of checking the Phrack site to see if the newest issue was available. So one of the engineers wrote a bash script Nagios plugin to page us once the new issue was released. Here's the check_phrack Nagios plugin (for issue 61):

# check for Phrack 61
. /usr/local/nagios/libexec/

if [ -r "/tmp/.got_phrack" ]; then
       if [ ! -r "/tmp/phrack61.tar.gz" ]; then
               /usr/bin/wget -q -b -o /tmp/phrack61.tar.gz
       echo "Got Phrack!"
       exit $STATE_OK
CHECK=`/usr/local/bin/lynx --source |grep "Phrack 61 download"| grep 0000`
if [ -z "$CHECK" ];then
       touch /tmp/.got_phrack
       echo "Come and get it!"
       exit $STATE_CRITICAL
echo "Keep waiting :-("
exit $STATE_OK

Nagios would systematically hit the Phrack site for us. If the new Phrack was available, Nagios would fire off a page. If we didn't have the new issue, the check_phrack plugin would download it, and change the Nagios state back to OK from critical. Credits to Imhotep of AU Crew.


Perl is a timesaver and lifesaver for me. I've been able to complete huge data tasks in minutes thanks to Perl and Perl modules found on the Comprehensive Perl Archive Network (CPAN). I've completed even large programming and debugging tasks more quickly and efficiently because I could use Perl to test and write the code. There is so much you can do with Perl that it is at least worth learning the basics.

Remember the programmer from the Vim example? What if he had needed to make the same changes across several files? No problem, Perl would have handled it with the same ease:

perl -pi -e s/^.+"(Crew_Special_signup\..+)".+$/<cfif findNoCase("$1",form\.walkerSelected)>, $1/g `find . -name "*.cfm"`

Now if only we could get the developers to switch from ColdFusion to Perl!


This one is a bit superfluous, but I will admit to keeping an extra screen open with wuzzah -a running. Wuzzah is a neat little program that lists when users have logged on or off and where their connection is from. It's a nice little application to have, sort of like icing on the cake.

Of course there are the obligatory utilities (eg: nmap, tcpdump, screen, grep, etc.), but I wanted to highlight a few tools that are less commonly mentioned and that I use on a daily basis.

Let us know about your most valuable utilities and how you use them. There need not be 10 of them, nor do they need to be in order, and if we publish your work, we'll pay you $100.

Click Here!