November 10, 2004

SysAdmin to SysAdmin: SysAdmins in academia

Author: Preston St. Pierre

The job of a system and network administrator in academia differs wildly from the job in a corporate environment. Over the past few weeks I have contacted other administrators who have worked in both environments in an effort to make a comparison between the
two. I've received a lot of feedback from admins all over the country. I hope
that you find the results as interesting as I did.

After reading the email I received, I found that the basis for comparison revolved around three main areas, which I've
broken down below.

How decisions are made

A large part of the job of a system administrator in any environment
involves either making quick decisions in order to put out a fire, or trying to convince others in your group or department to go in
a certain direction when making higher-level, long-term decisions.
However, that's where the similarities end.

In my reading, and in my personal experience, academia has a far greater
tendency to be what I'd like to call a "stateless meritocracy" than does
a corporation. A stateless meritocracy, as I'm defining it here, follows
these basic rules:

  • Decisions are based on input from multiple parties or individuals.
  • Decisions are based on empirical or demonstrable evidence that a
    particular tool or direction has the most favorable result.
  • The decision is made with little or no regard to earlier mistakes of
    the person putting forth evidence toward the making of the current
    decision.

In other words, if you previously made a decision that didn't work out, that is largely irrelevant in the context of what's
happening now. If your way looks like the right way, that's
probably the way the group will tend to go. Of course, long strings of
poor choices should be avoided at all costs. Also remember that this is a tendency in academia, not an absolute
truth. There are academic environments that make decisions with almost no input from anyone, but this would appear to be (in my informal
survey) more the exception than the rule.

In the corporate world, there tend to be more layers of administration, separate internal groups affected by decisions, separate user communities with (sometimes) separate departmental infrastructures, and
even separate policies concerning things like security and quality assurance, such that
the job can be like building out a single solution for more than one company!

As a result of this complexity, decisions tend to involve more people
in more meetings to discuss every aspect of a solution, from how funds
are appropriated for purchase to the system calls it uses to perform a certain function. In academia, these discussions take place,
but it's usually a couple of emails and a 20-minute meeting instead of
several lunch hours spent at a large conference table.

Finally, corporate environments tend to be more track-record-oriented than our semi-utopian stateless meritocracy. Earlier bad
decisions count against you. In some environments it gets to the point where it seems like decisions are made based solely on the track record
of the group or individual putting forth the idea. Maybe this can be called a "stateful meritocracy," in that the merit of an idea is based
on the results of earlier decisions made by the group or individual.

How projects are organized

Project organization and
management in corporate and academic environments is the most fun to compare in some ways; in other ways, it's a little scary.

In a corporate environment, projects are assigned to teams that generally include one or more project managers. Project managers have
several duties, not the least of which is coming up with a plan for the
completion of the project, and implementing that plan by assigning
deliverables to the specialists with well-defined deadlines. However,
what many consider the absolutely crucial job of a project manager is
managing the expectations of the client. In fact, one company I worked for actually had a position called a "relationship manager" whose job was to collect data from the project teams and then
spin that into something digestible for the client.

Specialists are people who will be delivering on their assigned tasks and reporting their status to their project manager on a regular
basis. Each individual specialist manages the expectation of the project
manager on their own behalf and to their own benefit.

What's interesting about this is that the flow of data about the real
status of the project is bottom-up -- from the specialists, to the
project managers, to the clients. Meanwhile, the data concerning the
goal of the project flows in the exact opposite direction, again using
the project manager as the middleman.

As both project manager and specialist, I have always found this to be an ineffective way to manage a project. However, when you have a job to do, you're likely to do it in
the canonical way. That's what I did -- and I always assumed that someday I'd see why it was done that way. I never have. Academia has taught me that I was not a dim-wit after all!

Projects in academia are far simpler by comparison: those who need a
technical solution to a problem tell a team of technical specialists
(possibly accompanied by a manager) what they want. The
specialists begin probing to better understand the problem. They eventually figure out what the end user actually needs, and they figure out the resources and time needed to perform the task. They then split up the work into component parts and build out the solution.

This simplicity and relative informality makes working on projects in academia much more enjoyable, and less stressful -- once you get used
to it. Coming from a corporate environment, this aspect of academia flat out scared me, because while accountability is well known in
this scenario, the relative lack of planning and formalized deliverables and deadlines means that the expectations are not as well defined.

How solutions are implemented

This seems to be the largest difference between the sandals and the
suits. If you have worked only in corporate, academia will scare the pants off you, and if you've only worked in academia, corporate
will, well, scare the pants off you.

In a corporate environment, a project is developed in a testing environment that closely, if not exactly, emulates production. If it
works there, that's great! It doesn't mean you're going to stroll into
the production datacenter and start logging in as root to roll out your
brainchild. What it means, a lot of the time, is that you've earned the
right to help put together "change management documentation." Change
management documentation is a collection of documents that, very slowly,
very deliberately, in a step-by-step manner, with screen shots, tells
somebody else how to do the production rollout, and how to respond to
any conceivable mishap along the way. You hand this over to some folks
in the datacenter, and wait by your pager on the day of the rollout in
case something goes wrong that isn't in your documentation.

In academia, what is more likely to happen is you find a machine
suitable for the service you're implementing. You set up the machine on
a private subnet and bang away on it, using as many clients in as many use case scenarios as you and your group can think of. If it works, you can then (if appropriate) invite some of the more risk-averse users to help test this alpha service. Once that works, you can do a beta release of
the service, which consists of having one or two clients completely dependent on the service, and others using it on and off. This grows over the course of a couple of weeks until you're ready for production.
"Ready for production" means that the people who set this up are willing
to bet their Nerf arsenal that this thing is going to work without a hitch.

Again, coming from corporate, this is scary. Again, accountability and expectations are a concern. Again, it is far less formal. Maybe the
biggest difference on a technical level is that "production rollout" in
some cases means changing a DNS record, moving the machine to the public
service subnet, or removing records from a hosts.deny file or iptables script.

In conclusion

In the end, the core difference between corporate and academia is that, while a Web site becoming unavailable for two hours in one day is
inconvenient, undesirable, and stressful for administrators in academia, it is not likely to cost the university millions of dollars. There are
many Web sites in corporate America that cost thousands of dollars per second of downtime. Web sites are only one example.

That fact changes the cultures of the two communities as well. In
corporate, administrators are generally laden with tech gear that enable
them to do their job from more locations, and be reachable in more ways,
at any hour of the day. There is less time to call your own, but in
exchange, you're generally paid a good bit more than in academia, where
things are a little more relaxed (not that there aren't emergencies).

I haven't touched on the user
communities, standardization, why Linux has been so big in production in academia but slower to catch on in big business, what "availability of
applications" means in the two environments with respect to Linux, and how ROI and TCO are measured, which answers lots of other questions.
This will all make great material for another article.

Click Here!