Recently I built some storage shelves in my basement. One step of
the process required a chisel. My accomplice suggested that I just
use a screw driver since it was more handy at the time. I responded
that I wanted to use a chisel since we had one and because I believe
in using the right tool for the job if it is available. I have the
same philosophy at work. I like to use the right tool for the job if
it is available. In ten years of managing networks I have picked up a
few favorite tools and techniques that each has its place in my
network administrator’s toolbox. Another administrator would likely
prefer a different set of tools but these are the ones that I like.
Likewise, a system administrator would likely not call these tools
her favorite because they are not the right tools for that job. The
tools described below or ones like them will be useful to any network
administrator and they are all open source.Minicom
For configuring routers, switches, and other appliances via a
console/serial connection, minicom is my tool of choice in Linux. If
it is not included in your Linux installation you should be able to
find it on the distribution CDs. Minicom is simple to use and works
for its intended purpose. The first small hurdle to clear in using
minicom is figuring out which device is the serial port on the
computer. My first experience using minicom was not very good but it
was not minicom’s fault. After figuring out which device was the
serial port on my computer I thought that I was ready to go but then
struggled for an hour or so to get minicom working. In the end, the
problem was not minicom but a flaky console port on the router.
Minicom was working properly the whole time. The lesson I learned
from that experience is that minicom is simple and works. If you
spend longer than 5 minutes to get it working that is too long and
the problem is likely not minicom or your configuration.
IPTABLES
Most people would not consider iptables
to be a network tool but I do. It has its place in my toolbox. It is
a firewalling, address translating, frame counting powerhouse that
comes installed with many distributions of Linux. Iptables is a tool
that can be used for multiple purposes. Firstly, its intended purpose
is as a firewall. Secondly, it can be used to perform network address
translation. Thirdly, it can be used to manage or account for traffic
that passes through it.
I’ve used iptables in a couple of ways that others might find
interesting. The destination NAT feature helped me to simulate a
large network (over 16 million nodes) without having that many
physical devices. This same feature will even do round-robin load
balancing. On another occasion I was curious about a certain
interaction between two systems on my network so I configured
iptables simply to count the number of frames and bytes associated
with that interaction. The documentation
section of the iptables website is very complete and has many good
HOWTOs.
Expect
Next in my toolbox is Expect
which is a scripting language that automates interactive sessions
like telnet. There is enough capability in the language that its
scripts can do most everything a network administrator would find
useful. The scripts basically come in the form of “expect
something” (like a router prompt) then “send something”
(like “config t”). My primary use for expect is to
automate telnet sessions. It takes some time up front to figure out
what to “expect” and what to “send,” but the
results are worth it. If I need to change the same configuration
parameter on each of a dozen routers an expect script makes it much
easier than manually connecting to each router and typing in the
changes. Another way expect can be useful is to abstract the
interface to a network device. For instance, I have created web pages
that kick off an expect script to change a specific parameter or
group of parameters on a router or switch. This enables others to
make appropriate controlled changes to the network without having
access to the command-line on routers and switches.
Multi-Router Traffic Grapher (MRTG)
Multi-Router
Traffic Grapher (MRTG) is my tool of choice for tracking and
graphing things on my network. MRTG collects data and generates
graphs and associated HTML for presenting the results via a web
server. The only catch is that you need to know what it is you want
to graph. Most commonly MRTG is used to graph values retrieved via
SNMP. However, MRTG can also accept values from any external script
or program. It runs as a cron job at the interval you specify. Every
time MRTG runs it reads its configuration file, retrieves the data
specified, stores the data, and graphs it. The format of the output
is fairly configurable. The MRTG website has great documentation and
there are many examples and resources on the web for figuring out
exactly how to configure MRTG to graph just about anything in the way
you want.
Currently I am using an MRTG
95th percentile add-on by Sean Adams to monitor my company’s
bandwidth usage. We pay a base rate for up to 10 Mbps and for each
additional Mbps that we burst above it. In their generosity our ISP
ignores the 5% of the month with the highest utilization and only
bills us based on the remaining 95%. Sean’s add-on tracks 30 days of
usage, chops off the top 5%, and reports the 95th percentile which
provides confirmation of what our service provider is charging us.
The next thing I want to do with MRTG is to configure thresholds and
have it alert us when we exceed our 10 Mbps base bandwidth.
Big Brother
Big Brother is watching over
your network and systems, or could be. Some people call Big Brother a
“ping and page” application and they are right. It can
monitor any system or service for reachability. Big Brother is like
MRTG in many ways: it runs periodically to check your systems and
services, it presents the results via a web server, the hard part is
knowing what you want to monitor, and examples and resources are
plentiful on the web. The alerting mechanism is very flexible.
Knowing that Big Brother is watching allows many network
administrators to sleep well at night.
Nslookup and Dig
Nslookup and dig are pretty much standard command-line tools
installed or available for most linux distributions. In fact I cannot
remember ever installing either program individually but they are
always there. These tools allow you to interact manually with the
domain name system (DNS) to see if name resolution is working
properly. Every computer on the Internet has an IP address. DNS is
the way that friendly names (www.somedomain.com) are mapped to
functional IP addresses. Traffic is routed through the Internet based
on IP addresses (not names). So, if a web browser cannot resolve the
friendly name to an IP address it will not be able to talk to the web
server, no matter how friendly. Since many Internet connections begin
with a DNS query, nslookup and dig are invaluable tools in
troubleshooting connection problems.
Some times certain mail messages on our mail server get stuck in
the outbound queue. Someone sends a message to Joe@somedomain.com and
the mail server doesn’t know where to send it. Before the mail server
can send an e-mail it has to know the IP address of the mail
exchanger for the destination domain, in this case “somedomain.com.”
In order to figure out why the mail server cannot find where to send
Joe’s message, we could use nslookup or dig to do the same kind of
lookup that the mail server would do. A special kind of query called
an “mx” query would tell us the name and IP address of the
e-mail server at somedomain.com. Depending on the response, further
troubleshooting steps would be taken. For more information on DNS do
a search for “DNS tutorial” at your favorite search engine.
For more detailed information on how to use nslookup and dig, the
first (and probably best) place to look is in the “man”
pages.
Manual SMTP Session
This one is more of a technique than a tool but helpful
nonetheless for its intended purpose. Mail flows through the Internet
using the Simple Mail Transport Protocol (SMTP). For troubleshooting
Internet e-mail, a manual SMTP session is the best way to isolate the
problem because you get to see exactly what is going on. The first
step is to use nslookup or dig to find out the name and IP address of
a mail exchanger for the destination domain. Pretend for a minute
that the “mx” query mentioned above for somedomain.com
came back with a name of mailhub.somedomain.com and an IP address of
A.B.C.D. IP addresses are really numbers (not letters) but you
understand the idea. From there a network administrator would telnet
to port 25 on the server at A.B.C.D. If everything is working the
administrator would see a “220” message, the friendly
name of the mail exchanger, whether SMTP or ESMTP is supported, and
the name and version of mail exchanger software running on the
server. The session below illustrates how to send a mail message
manually. I know the thought would never crossed your mind, but
please note that this is for educational purposes only and not for
sending funny e-mail messages to your friends from famous people. The
parts in bold are what you would need to type.
root@Linux:/ > telnet mailhub.somedomain.com 25 Trying A.B.C.D... Connected to mailhub.somedomain.com. Escape character is '^]'. 220 mailhub.somedomain.com ESMTP Exim 4.04 Wed, 13 Nov 2002 16:27:42 -0700 helo someotherdomain.com 250 mailhub.somedomain.com Hello someotherdomain.com [E.F.G.H] mail from:<Jim@someotherdomain.com> 250 OK rcpt to:<Joe@somedomain.com> 250 Accepted data 354 Enter message, ending with "." on a line by itself Subject:Test MessageThis is a test message.. 250 OK id=18C6vd-0005Ey-00 quit 221 mailhub.somedomain.com closing connection Connection closed by foreign host root@Linux:/ >
Joe@somedomain.com should get an e-mail from Jim@someotherdomain.com
or you will have to do more troubleshooting to isolate the problem.
Telnet
Similar to the manual SMTP session above is a telnet session. How
could telnet be helpful in diagnosing a network problem? Well, by
default telnet connects to port 23 on a telnet server. “Port”
refers to a software port or handle and not a physical port on a
network. Each service on a server listens on a unique port so that
the server can know which software to use to handle each connection.
For instance, in the example above the mail exchanger is listening on
port 25. Web servers generally listen on port 80. Telnet servers
generally listen on port 23. The port numbers do not matter as long
as we understand that there are unique ports for each service and as
long as the client knows which port it needs to talk to. Telnet can
be used to confirm that a network path to a port is working. When a
user says I cannot get to the site at www.somesite.com because the
firewall is blocking me an administrator could telnet to port 80 on
www.somesite.com’s web server. The administrator would probably see
something like this:
root@Linux:/ > telnet www.somesite.com 80 Trying A.B.C.D... Connected to www.somesite.com. Escape character is '^]'. _
Based on the “connected” message, the administrator would
be able to tell the user that the connection has been established to
the web server. Some times pressing <ENTER> a couple times will
evoke a response from the server on the other end. Other servers like
SSH, telnet, and SMTP respond immediately with some indication that a
connection has been established. This is a helpful technique for
troubleshooting connections through firewalls and for seeing if there
is a network problem or a system problem.
Ethereal
Ethereal is the hammer in
my toolbox. If nothing else works I go for the hammer. It is a
full-featured open source sniffer available for *nix and Win32. A
sniffer is a piece of software that captures traffic from the network
and decodes the traffic to make it more readable for analysis. Most
quality sniffers, like Ethereal, can classify the captured frames as
belonging to a particular protocol (like HTTP or SMTP) and specify
the interactions of the protocol. Every network administrator should
have a sniffer is his toolkit and learn how to use it. Ethereal is
the sniffer I choose because the price is right and for the things I
do it is as capable as any other commercial sniffer I have used.
I usually bring out Ethereal for the trickiest of network
problems. If everything seems to be working on the network (name
resolution is working and pings work) and you’ve used some of the
other tools described above but an application still isn’t happy
break out the sniffer. If you plug a sniffer into a switch the only
traffic it will see is broadcast traffic and traffic to/from the
workstation where the sniffer is running. So, if you want to sniff
traffic to/from a particular server you will have to insert a hub
into the network temporarily or configure the switch to mirror
traffic from the interesting port to the port where the sniffer is.
Interpreting a sniffer trace is not a trivial task but with practice
and study, reading sniffer traces becomes easier. The Ethereal
User’s Guide is probably the best place to start.
Other open source sniffers are tcpdump and windump. Both are
command-line tools for capturing or viewing traffic on *nix and
Win32, respectively. They do the same thing as Ethereal but are just
not as pretty and do not break out the protocol data. I use tcpdump
and windump for quick captures when I don’t need to get into the
traffic too deeply to see what is going on. The output of tcpdump and
windump are readable in Ethereal.
Recap
Using the right tool for the job can make things much easier on a
network administrator and there are lots of great open source tools
available for managing networks. The list above represents a few of
my favorites and I hope you find them useful too.
Category:
- Linux
