March 3, 2006

My sysadmin toolbox

Author: Matt Darby

I preside more than 10 Slackware and Debian servers at a small engineering firm, serving 50 in-house and several external employees. Normal IT services are rendered through the use of Apache, MySQL, PHP, BIND, Squid, Netfilter, Courier IMAP, Postfix, VsFTPd, and Samba. Here's a list of my essential tools.

PuTTy

PuTTy is a free SSH client that's indispensable if you work on Linux boxes from a Windows desktop. It has full support for shared key authentication, X11 forwarding, session saving, and a ton of other great options. Some tricks I use to make PuTTy even greater:

After you create and save a session, create a shortcut to PuTTy and view its properties. Append "-load session name" to the end of the "Target" box to force PuTTy to automatically start that session when opened.

If you use the program screen in PuTTy, you need to edit your keyboard settings to emulate XTerm:

Go to Terminal -> Keyboard and select XTerm R6. Set both values under Application keypad settings to Application to ensure that your keys are set properly.

Under Connection you can set the Auto-login username field to automatically log in as a certain user (no surprise there, but very handy).

Under the Connection -> SSH menu you can send a remote command to the session after you login.

If you want to use private key authentication, and already have your keys, go to Connection -> SSH -> Auth to tell PuTTy where your public key is located.

I have all of these tricks set up on my work desktop. I can click a PuTTy shortcut on my Quick Launch toolbar, and PuTTy will automatically log in to my main server and start a screen session that has windows to all my other servers mapped to my function keys. If I need to work on a different server, I simply press its corresponding function key and I'm there.

GNU Screen

Screen allows a user to attach, detach, reconnect, and share terminal sessions. You can create any arbitrary number of screens automatically, and each of these screens can be connected automatically to any number of servers, regardless of location. I can work all day in a screen session, detach, go home, reattach and continue working. I can even reattach with my Treo 650.

Screen has a simple configuration file with some powerful features. Here is my personal ~/.screenrc.

GNU Emacs

Emacs is the text editor that thinks it's an operating system. You can do almost anything from Emacs, including compiling source code and running the resulting binary. The Bash shell uses Emacs editing mode by default, so once you learn the keystrokes in Emacs, your command line will become a faster, friendlier place to work in. As an example, Ctrl-A will move your cursor to the beginning of your command line, and Ctrl-E will take it to the end. Ctrl-K will delete from the cursor to the end of the command line. There are literally hundreds of great time-saving keystrokes and modes in Emacs.

Here's my ~/.emacs file.

SSH

SSH allows you to do some elegant things. With the usage of authorized keys, you can log in to your server without passwords. This may sound odd, but it's actually more secure than using passwords. This method also eliminates all of the unauthorized login attempts that your server is exposed to on the Internet. With SSH you can also automatically login to a remote server, gzip a series of files or folders, and transfer the resulting tarball to your local machine without creating archives on the remote machine! I use this script to backup several key folders from around my Samba shares:

#!/bin/bash

filename=Backup-`date +%Y%m%d`.tar.gz
ssh user@server "tar czpf - /path/to/files/" | cat > /path/to/backup/repository
/$filename

System Rescue CD

The System Rescue CD is a live CD based on Gentoo that allows you to boot an ailing server from CD. After booting, you can mount your normal root partition, chroot into it, and fix any errors. Exit, unmount, reboot, and you're back in business!

Bash completion

Bash Completion is a great project that further extends the completion features of the Bash shell. Forget a switch for tar or CVS? Simply type in the program name and press Tab to display all the available options. To use it, untar the bash_completion script to /etc and add . /etc/bash_completion to your ~/.bashrc file. Reload your bash settings with . ~/.bashrc.

GNU Readline

The GNU Readline Library is a great way to further customize your command line environment. You can bind keys to certain functions, change your completion options (such as what and how many options to show when a completion attempt is ambiguous), and set your editing mode to use Emacs or vi key mappings.

To modify the default readline behavior you need to edit your .inputrc file in your home directory or in /etc/inputrc. Here are the directives I find most useful.

.bashrc

Editing the ~/.bashrc file to your liking is a great way to feel comfortable at your command line. You can set command abbreviations, functions, and environment variables in this simple text file. Creating aliases to complex commands can speed up your work. After you add functions to your own ~/.bashrc, execute it to get the system to recognize the new fuctions.

This function, for instance, automatically creates a key pair for you (if you do not already have them), and inserts your public key into a remote server's authorized key file, allowing you to login to that remote server using authorized keys.

#Usage: put_key servername
function put_key(){
   if [ ! -s ~/.ssh/id_dsa.pub  ]
      then
         ssh-keygen -t dsa
   fi
   cat ~/.ssh/id_dsa.pub | ssh $1 'sh -c "cat - >> ~/.ssh/authorized_keys2"'
}

Here is an easy way to push a file to several remote servers without installing any software packages:

#Usage: servers /path/to/local/file /path/to/remote/location/file
work_server[0]="server1"
work_server[1]="server2"
work_server[2]="server3"

function servers(){
  for host in "${work_server[@]}"
    do
      scp $1 root@$host:$2
  done
}

This function gives you a nicely formatted display of a machine's important statistics, such as memory used, load, disk usage, and who is currently logged into the server:

#Usage: ii
RED='\e[1;31m'
BLUE='\e[1;34m'
CYAN='\e[1;36m'
NC='\e[0m'

function ii(){
    clear
    echo -e "\nYou are logged on ${RED}$HOSTNAME"
    echo -e "\nAdditional information:$NC " ; uname -a
    echo -e "\n${RED}Users logged on:$NC " ; w -h
    echo -e "\n${RED}Current date :$NC " ; date
    echo -e "\n${RED}Machine stats :$NC " ; uptime
    echo -e "\n${RED}Memory stats :$NC " ; free -m
    echo -e "\n${RED}Disk usage :$NC " ; df -lh
    echo -e "\n${RED}Local IP Address :$NC" ; /sbin/ifconfig eth0 | awk '/inet/
{ print $2 } ' | sed -e s/addr://
    echo -e
"----------------------------------------------------------------------\n"
}

Rsync

Keeping two directories in sync securely, along with file permissions, owner, and group settings, is easy with rsync. The rsync utility also supports compression and SSH tunneling. One method I use to keep home directories synced to my backup server is:

#!/bin/bash

bad_files='\(.*\.\(log\|bak\|ac$\|dwl\)\|"zzz*"\|"~*.*"\)'

cd /home/
rsync -avz -e /usr/bin/ssh --delete --stats --exclude $bad_files . root@server:/mnt/backup/homes/

The -avz options respectively archive synced files (preseving permissions, etc.), print what is being synced, and compress the files during the syncing process.

The --delete option forces deletion of file that are found only on the system that's being synced from. This keeps the files in true sync.

The --stats option prints a short message at the end of the syncing process showing you how many bytes were transferred and how long the process ran.

The --exclude $bad_files option excludes any files that match the regular expression I've specified; in this case it doesn't process *.log, *.bak, *.*ac, *.dwl files, or any files that start with "zzz" or "~".

I hope some of my tricks help lighten your daily grind at the keyboard. I look forward to your comments and hope to learn something new myself.

Let us know about your most valuable utilities, and if we publish your work, we'll pay you $100.

Click Here!