April 7, 2006

My sysadmin toolbox

Author: Vaida Bogdan

I use FreeBSD and Linux on more than 15 servers at work. Here are 10 of the tools I find most useful.

GNU Screen

GNU Screen is a multi-screen window manager that can turn a single remote connection into a universe of possibilities. I can read a logfile, connect to my other servers, and read my favorite BOFH episode, all over one SSH connection.

One of Screen's main features is its ability to detach my session when I leave work and re-attach it the next morning. As long as the server isn't restarted I can leave tail -f /var/log/messages running forever.

In my network deployment I have a secure box that's the only server that allows remote shell access (IP-based). After I log in, I always resume my screen session by executing screen -Dr.

In this session I have seven windows open. Windows one and two are my work area. Sessions three to six are remote SSH connections to four of my servers, which I can access by pressing Ctrl-a 3 through Ctrl-a 6. The screen session named zero runs the aforementioned tail command.

If I want to lock my terminal, I can type Ctrl-a x. The password can be my username's password or a password I set in my .screenrc.

Screen also allows me to copy and paste between windows. Ctrl-a [ enters the copy/scrollback mode. I go to the first character I want to copy and press Enter, go to the last character and press Enter again, and, finally, I switch to the window I want to paste to, and type Ctrl-a ]. VoilĂ ! Copied and pasted without a mouse.

Another interesting feature is Screen's ability to work with another person. Run chmod +s 'which screen', and in your screen session type Ctrl-a : multiuser on and Ctrl-a : acladd connectinguser. Now the connecting user can attach to your screen session with screen -x yourusername/. Anything you type will be seen on both of your terminals, and vice versa.

Screen lets you print the current screen, log a session, and even split windows.

Duplicity, ssync, and FUSE

Technically speaking, this is three utilities, but Duplicity, ssync and Filesystem in Userspace (FUSE) make a great combo.

With Duplicity you can do encrypted backups of your filesystem to a remote server. I use it to backup my home directory to a server in another town. I do incremental backups daily and full backups monthly. My script looks like this:


----------- duplicity_backup.sh -----------
#!/bin/sh
duplicity -i --include /etc --include /boot/loader.conf --include /root --include /usr/local/etc --exclude '**' / scp://username@backupdomain/BACKUP/backup_system
duplicity -i /home/username scp://username@backupdomain/BACKUP/backup_home
-------------------------------------------

The command duplicity --verify scp://username@backupdomain/BACKUP/backup_home /home/username compares the system with the backed up files to check what has changed since the last backup.

To restore a file as it was four days ago you would run:

duplicity -t 4D --file-to-restore personal/project.c scp://username@backupdomain/BACKUP/backup_home /home/username/personal/project.c

I also use ssync and FUSE to synchronize my workstation with my notebook (in one direction: notebook -> workstation). I mount the remote directory to /tmp with fusefs-sshfs and then I sync it with my home:

mkdir /tmp 2>/dev/null
mount_fusefs auto /tmp/username sshfs username@workstation: && \
ssync -f /home/username -t /tmp/username 1>>backup_fusefs.log 2>&1

Unison is a similar application, with which you can mirror two filesystems bidirectionally.

GEOM Based Disk Encryption

I use GEOM Based Disk Encryption (GBDE) to transparently encrypt my home partition for security reasons.

The FreeBSD handbook contains a step-by-step guide on how to encrypt a local hard drive, and you can find a script that uses GBDE on a memory disk here. You can also try GELI as an alternative to GBDE.

FreeBSD's Mandatory Access Control

FreeBSD's Mandatory Access Control is imported from the TrustedBSD project. Mandatory Access Control is still under development, but it has the potential to be a useful tool in every FreeBSD deployment.

I use the mac_chkexec module to prevent a file from being executed if it doesn't match a specific checksum. I run the module in learning mode first, so that it computes the checksum of my system binaries, and then enforce the policy with a sysctl. As a result, none of my users are able to execute a file that wasn't executed while in learning mode or whose contents have changed.

Mac_seeotheruids prevents users from seeing other users' processes, which is similar to the security.bsd.see_other_uids sysctl, and mac_bsdextended is a kernel module that implements a system firewall policy. For example, mac_bsdextended can be used to ensure that users can't see other users' files.

Other interesting modules include the the Multi-Level Security confidentiality policy (mac_mls) and the Low-watermark Mandatory Access Control data integrity policy (mac_lomac) module.

Pkg_cutleaves

I use the pkg_cutleaves script to find installed "leaf" packages -- that is, packages not referenced by other installed packages -- to trim unwanted packages from my system. I have a file, /usr/local/etc/pkg_leaves.exclude, in which I list all the packages I need, and pkg_cutleaves displays a list of "I don't need you" candidates, thus helping to keep my system clean.

Birthday

The birthday utility is not sysadmin-specific, but it's a simple program that keeps track of important dates, and it has saved me a few times.

My $HOME/.birthday contains fields like:

friend =dd/mm/yyyy bd
wine festival =1/10 to 3/10 ev

The first field is the event name, with the date or dates after the equals sign, then the type of reminder. For example, bd is a birthday, ev is an event.

Then, I add a few lines to my .bashrc to make sure I am notified:

if [ ! "$SSH_CONNECTION" ]; then
     echo
     birthday
     echo
fi

Now, every time I start a terminal, I see important dates two weeks ahead displayed at the top of my screen.

Figaro's Password Manager

Figaro's Password Manager (FPM) is a lightweight password manager and password generator. After you type your master password, double-clicking a link in the FPM GUI will launch your browser, or gnome-terminal with SSH, or any other program. It also copies your username to the clipboard and the password to the primary selection, to make it easy to log into whatever service you're using. You can then paste your username with Shift-Insert and your password with the middle mouse button. I find this tool useful on my notebook, and feel safe using passwords even if I'm not alone, since no one can see me typing a password on the keyboard.

Graphviz

Though I don't use it very often, I find Graphviz useful to design network diagrams by structuring the information instead of drawing it. As a result I don't need talent as a graphical designer, just my abstraction skills.

This is a good example of using Graphwiz to create a network diagram. The image generated by Graphviz is here.

Wmweather

I use wmweather, a dock application for Window Maker, to monitor the weather in my city. It sits in my dock and displays temperature, humidity, wind speed, and so forth.

Portaudit

Portaudit checks my installed FreeBSD ports periodically against a database containing published vulnerabilities, and warns me in my daily security report if they are exploitable. By modifying some rc scripts, I can automagically update any vulnerable package the day the vulnerability is published.

Let us know about your most valuable utilities and how you use them. There need not be 10 of them, nor do they need to be in order, and if we publish your work, we'll pay you $100.

Click Here!