February 17, 2006

My sysadmin toolbox

Author: Brian Jones

I administer servers and networking equipment in a small to mid-size
heterogeneous (but 99.44% *nix) environment. I've worked on projects ranging
from NIS-to-LDAP migration for authentication, to the deployment of a 164-CPU
Beowulf cluster, to writing an extension to a large OO-PHP application. I'm a
generalist, but my favorite areas are LDAP (and authentication in general),
database design and administration, and automation (a broad category that
includes writing tools and code to glue services together). Here are my
favorite tools.


Minicom allows me to
use my laptop as a serial console. I can walk up to any server (x86, SPARC,
whatever), plug into the serial port on the back, fire up minicom in an xterm
window, and log in.


Some readers are going to roast me for using Telnet instead of netcat, but
in the cold, cruel real world in which I live, I do not always have a choice as
to the installed software. If I'm consulting, I'm not going to tell a client
that I need netcat to do what Telnet (which is already installed) can do just
fine. If you want to test connectivity to your mail server, you don't have to
have a fat mail client (or a mail account for that matter). Just telnet to port
25 of the mail server.


I might use SSH more than any other tool except for the shell, which I use
to launch it, and the terminal, which I open to get to the shell. What I like
best about SSH is that, using shared keys, I can mostly do away with sending
any kind of password over the wire. Also, there are services in my environment
that require the use of a GUI interface -- I'm not learning every
Veritas command line tool, thanks -- and with SSH, I can tunnel an X session
back to my workstation.


I don't run X on a server unless it's a requirement of a service or
application it provides. However, I do almost all of my administration tasks
from my workstation, which runs dual screens and X.

People who say you can get the same functionality from using screen and
VTsplit and the like have probably never spent enough time in an X environment
to know the beauty of dual screens, where you can quite easily keep track of
several different service or host monitors, email, my admin group's Jabber chat
room, several documentation sources, Slashdot, Gmail, and lots of extra
niceties like the weather and package update notifications via alert applets in
my taskbar.

I don't care what window environment I'm in. I use fluxbox, KDE, GNOME, and
Xfce depending on the machine and my mood.


ClusterIt is actually a
suite of about a dozen or so tools that come packaged together. They're meant
for use in Beowulf-type environments, but oddly enough, I don't use this suite
on the cluster I maintain. I use the tools on collections of machines that are
identically configured, such as our Linux development machines, a 30-node
teaching lab running Fedora, or a collection of Solaris service machines.

Want to know which machines a particular user is logged in on? Try running
the ClusterIt utility dsh "w|grep username". This doesn't
even begin to describe the power this suite of tools gives you.


If I'm doing anything on a remote host that requires an editor, I use vi or
Vim. This is probably 85% of my file editing. However, if I'm writing code on
my local workstation or on my laptop for deployment later elsewhere, I use jEdit these days. It's a nice, unassuming,
unpretentious editor that lets me do just about whatever I want. It gets code
blocks out of my way when I'm not reading them, it has a keyboard shortcut
config tool that lets me emulate Vim moves I can't live without, it supports
more languages than I'll ever use, and it lets me alter every language element
in terms of how it's colored or highlighted.

jEdit makes managing multiple open files easy, and there's probably a plugin
for anything I haven't thought of using it for yet. If you've been using
Bluefish or Quanta, you might find that jEdit fits your brain a little


I've written
about xmeter before
, and it's still a tool I rely on every day. It's a
graphical utility used to monitor statistics collected by remote hosts' rstatd
daemons. I use it to do simple load monitoring of just about all of the hosts
in my environment. You can set xmeter thresholds, so if, say, the load on any
server goes through the roof, the monitoring graph changes color from green to
bright red.

Xmeter is, to my knowledge, not maintained, but you can grab a zip file of
the source and man page from my


This is one of those tools that you'll never stop using. It's not useful
only if you don't know something; often it's useful to give yourself a sanity
check. If I find myself in a machine room, and it's after midnight, and I have
to run any command starting with "ch" on any system, I check its man page just
to make sure.

Forget the -h on a call to chown on a Solaris
system using non-GNU tools, and you could inadvertently change the ownership of
your entire directory hierarchy. Who ever remembers that you need a
; at the end of the argument to the -exec flag of the
find command?

The man command is the first thing I consult when I get a goofy error from a
command, but it's also useful for figuring out what a particular function call
does or the syntax of various configuration files.


Ping is another tool that is installed by default on just about every machine
I've ever logged into. If I'm logged into a Solaris 7 box, and I need to see if
another machine is up and running and connected to the network, I'm running the
ping command, because that's all that is likely to be there. Sure, I'd rather
use something like hping, which supports far
more than just simple ICMP, or even fping,
which at least makes pinging a collection of hosts a little more friendly, but
I don't often find these tools installed at client sites, and I don't even have
them installed ubiquitously in my own environment.

Network grep (ngrep)

I don't trust apps that say they're using encryption -- I test. If I'm at a
site where I'm responsible for the security of an application, and ngrep is not installed, I
will request that it be installed. If they say no, then I request that
I be able to mount a CD or USB device with a binary on it I can use.

Ngrep is "network grep". It'll capture packets that contain data matching a
given pattern. If I want to test for encryption, I launch ngrep on the server
with a command like ngrep badpass port 389. This will
look for any packets whose data contains the string badpass.

If ngrep finds one, then clearly the data was not encrypted, and something
is going wrong. You can also use ngrep for basic tcpdump-like functionality,
and can tell it to inspect files instead of traffic from a device.

Let us know about your most
valuable utilities (there need not be 10 of them), and if we publish your work, we'll pay you

Click Here!