Author: Brian Jones
in a large data center or a small home network, is to script yourself out of a
job. The more you can automate tasks, the more time you can spend researching
new technologies, deploying new services, and playing Quake! Experienced
administrators generally have at least three scripting languages under their
belts for regular use, and probably have books that cover one or two more for
occasional use. Beginners, however, sometimes have a hard time deciding which
tools to devote their learning cycles to. This article aims to help those users
come to grips with the many options available.
It’s extremely difficult to talk about scripting languages without setting off
a religious flamewar. What I’m presenting here are languages which are 1)
widely used, and therefore beneficial to know, if only to parse the work of
others; 2) generally available in a majority of the production
environments you’re likely to enter; and 3) necessities on any résumé that
touts the holder as a systems administrator. Flamers, please stand down.
Scripting with the Unix shells
The first scripting language many administrators learn is the Unix shell. For beginners, the nice thing about the shell is that you can work
interactively at a command prompt to get used to how the shell operates, what
it likes, and what it tends to choke on. You can get to know the error messages
and quirks of your chosen shell, all while doing useful things on a day-to-day
basis. For example, even a bash shell beginner probably knows the ls, pwd, and echo commands. Early bash scripts will probably be simple lists of shell
commands, like this:
#!/bin/bash
CWD=`pwd`
echo "The contents of $CWD are:"
ls
Shell is a great tool because it naturally lends itself to the “learn it as you
need it” mantra common even those who have become quite advanced scripters.
Another attractive feature of learning shell scripting is that you’re bound to
want to do more advanced things with your scripts than you would generally do
on the command line — but usually whatever you do in a script can be
done on the command line. In other words, the more you script with the
shell, the more efficient you become at using the shell, whether it’s being
used interactively or in a script. Eventually, you’ll get to a point where you
write shell scripts only for repetitive tasks, because you can do most
of the other one-off craziness with a cool one-liner. And those one liners can
be compiled together later to make shorter, more efficient scripts!
Many of the shells you’ll encounter as an administrator can be used to do
fairly advanced scripting. For example, bash, which is the default shell for
all of the Linux distros I’ve used, supports looping constructs like while,
until, for, and case; some variable-type declaration options; and you
can even get into forking, signals, and interprocess communication to some
extent. So why would anyone ever use anything but shell? They’d have to be
crazy, right?
Along came Perl
While shell scripting is a crucial requirement for anyone who
wishes to call himself a systems administrator, the shell is not the best tool for every job. The most
glaring problem with the shell is that every shell may not be available on
every system you need to work with. Shell scripts can also get messy and bulky to
accomplish more advanced tasks. As your needs become more
complicated, your choices are, generally speaking: 1) Learn every flag to all
of the thousands of commands available to your shell, and understand how every
single one of them handles input and output, etc., and keep writing shell code
— assuming all of your systems have a shell (not to mention the same versions
of the thousands of commands); or 2) start learning some Perl!
Most administrators find Perl to be every bit as valuable as the shell itself. The first thing that Perl has over shell is that there’s only one Perl. You
don’t have to concern yourself with shell variants, whether or not tar takes the z flag on one system or another, and if grep is a link or alias for egrep or fgrep. Perl is pretty much Perl, no matter where you go — and you can go pretty far with it, since there are versions available for every conceivable Unix variant, including Mac OS X, as well as Windows. You’d have a hard time finding a platform Perl doesn’t run under, as it’s been ported to seemingly every imaginable platform, from Acorn to z/OS. So your scripts can be
put on a “pack and go” CD, which can go anywhere and solve almost any issue.
The mantra of the Perl community is “TMTOWTDI,” which stands for “There’s More
Than One Way To Do It.” While the ability to perform a given task in at least
ten different ways can be daunting for new users, they quickly learn that they need to know only one of those ways to get their job done.
As an administrator, my experience has been that the key to Perl lies in
the rich assortment of available modules. Modules are created by developers in
order to make one job or another easier with Perl by creating a sort of
black-box collection of functions ready for the user to call. Perusing
the module list takes quite some time, and leaves one struggling to
find something that can’t be done with Perl and the right module.
Modules make tasks much, much easier. For example, I do a lot of work with
LDAP servers. Without Perl’s Net::LDAP module, I’d have to write a
specialized client tool using Perl’s socket modules, deal with low-level
network stream I/O, have a good understanding of the RFCs relating to the LDAP
protocol so I knew how to structure the data sent to the server … it would be
a mess. With one additional module, I just make a function call, pass it the
name of the server and a search term, and I get the information sent to
standard output (or wherever, provided I pass “wherever” to the function).
Here’s a small piece of a simple Perl script, which grabs the numeric UID and
GID of a given user from an LDAP directory and prints it to the screen:
#!/usr/bin/perl
use Net::LDAP;
$svr = "myserver.mydomain.com";
$port = 389;
$base = "o=MyOrg ,c=US";
$filter = "(uid=$ARGV[0])";
$conn = Net::LDAP->new($svr, port => $port) || die "Couldn't connect:
$!n";
$conn->bind || die "Couldn't bind: $!n";
$search = $conn->search(base =>$base , filter => $filter);
@results = $search->entries;
foreach $entry(@results) {
$gid = $entry->get_value("gidnumber") || die "GID Number problem - GID($gid)n";
$uid = $entry->get_value("uidnumber") || die "UID Number problem - UID($uid)n";
}
print "$gid .... $uid n";
As if the language wasn’t extensible and flexible enough, Perl even allows you to
include inline C code in your programs. In the unlikely event that even Perl can’t efficiently do what you want, you can get even closer to the metal with C.
The bottom line is that, while there are tons of languages out there that
are great at a lot of things administrators need, none of them is as
ubiquitous, powerful, or mature as Perl. Yes, PHP has some built in string
manipulation functions that Perl requires more manual work to do. True, what
takes two lines of ugly Perl regular expression syntax could likely be done
using only a colon in Python. But on the whole, as far as administration tasks
go, Perl is still the one single scripting language to rule them all.
You ought to know awk
I’m happy to take the heat for recommending awk to up-and-coming
administrators. awk is a quick way to work with things like
the /etc/passwd and /etc/shadow files, which have predictable “fields” delimited
by the colon character. This lends itself nicely to using awk from the
command line. For example, cat /etc/passwd | awk -F: '{print $1, $3, prints the first, third, and seventh fields of each line of the
$7}'
/etc/passwd file.
awk is great for adding a lot of power to a one-liner without remembering a lot of syntax. However, you can also use awk to write full-fledged scripts. It has support for all of the basic functionality you’ll need, including test comparisons, basic looping structures, and much more. In fact, I know one administrator who rarely uses Perl because he has such a thorough grasp of awk. I’ve seen 350-line awk scripts that do quite a lot! However, in the context of administrative duties, awk can give you an efficient way to pull out fields of a file or some form of standard input, manipulate that information, and write it back out. Being that much of the information of
interest to admins is stored or output in some predictable, delimited format,
it is ready-made for awk.
Here’s a really cheesy “awk script” (it’s really a lot of lines of the awk command embedded in a bash script, but you get the point) which I use to pipe NIS map output into, and get proper LDIF (LDAP Data Interchange Format) out of. I found the standard (and far more complex) LDAP migration tools to be far less
useful than this script:
#!/bin/bash
awk -F: '{
print "dn: cn="$1",ou=People,o=MyOrg,c=US"
print "objectClass: top"
print "objectClass: person"
print "objectClass: organizationalPerson"
print "objectClass: inetOrgPerson"
print "objectClass: posixAccount"
print "objectClass: inetLocalMailRecipient"
print "objectClass: shadowAccount"
print "uid: "$1
gfields = split($5,gecos,",")
namefield = split(gecos[1], fullname, " ")
print "sn: " fullname[namefield]
print "givenName: "fullname[1]
print "cn: " $1
print "userPassword: {crypt}"$2
print "loginShell: "$7
print "uidNumber: "$3
print "gidNumber: "$4
print "homeDirectory: "$6
print "gecos: "$5
print "mail: "$1"@myorg.com"
if (gecos[3] != "" && gecos[3] != "NONE")
print "roomNumber: " gecos[3]
if (gecos[4] != "" && gecos[4] !~ /.*xx.*/ && gecos[4] !=
"NONE")
print "telephoneNumber: "gecos[4]
print "displayName: " gecos[1]
print ""
}'
This is by no means what anyone would consider “good” or “proper” awk. I’ve used it more as duct tape here. You don’t need to understand that whole script — it’s just an example to see what awk generally looks like. This
script gets a NIS passwd map piped into it, and has output redirected to a file
that is ready for importation into an LDAP server. As you can probably gather
if you saw the earlier command-line awk example, this chunk takes
each line of my NIS map and divides it into fields separated by colons. I can then print the fields wherever and however I want by
addressing them by their position in the line, preceded by a dollar sign. In this case,
I know exactly what I want my data to look like, I know what it looks like now,
and the standard migration tools were rather rigid for my needs.
In conclusion
I hope this information on scripting helps a few lost souls confused by all of the options available to them. In future columns I’ll revisit scripting
languages, both to give you a more thorough overview of
their use, and to help you do more useful things with them.
Category:
- Linux
