July 28, 2004

SysAdmin to SysAdmin: It's the documentation, stupid!

Author: Brian Jones

Dear open source developers: We sysadmins are tired of the crap you've been throwing at us. It seems you've been assuming all these years that we sysadmins are coders, and thus, "the code is the documentation." In reality, we are not coders. We are systems administrators, which is precisely why the word "coder" does not appear on anything which also has our name on it -- and thus, your mantra is no longer cute.

We are but lowly sods who mill about in the background, ensuring things like email and backup are working, and that users can get to the resources they need to do their jobs.
Some of these users are even coders; but not us.

We wish not for fame, but for stability. We dream not of the next killer app, but rather have nightmares about the next killer worm. We don't look forward to going to work every day to mangle bits, but rather to leave without maiming a user.

We are the Men In Black, completely invisible to all but those who seek our help (or who market products to us). We want to be in the background as much as possible. We don't even want to talk to others who aren't sysadmins. We'll go so far as to refrain from joining a single mailing list, so as to avoid
the temptation of asking a question in some public forum, lest we be deemed foolish in some way. Our greatest wish is that those who come into contact with us forget that we even exist only seconds after they leave. We would kill for one of those "flashy things" like Tommy Lee Jones had in that movie.

However, thanks to you open source developers, we instead look like a
conspicuous flock of shiny-headed, drooling lunatics, incapable of deciphering
even the simplest of tasks.

Why systems administrators don't use your stuff

You probably really do have the next killer app. You probably shout about it on
every BBS and every mailing list possible. You've posted your release
announcements to NewsForge. You even
got Slashdotted once. But still, nobody is downloading, and that email link on
your site saying "tell me if you're using it in production" hasn't been
clicked on even a single time. You spend hours coding, coding, coding. You spend
more hours promoting and generally keeping up with all things open source. You
have an infinite number of IRC buddies who think you are 70t@||y 1337.

Whatever. I'm not using it, and I'll tell you why. There is no documentation.

When I say "there is no documentation," I don't necessarily mean that there is
literally not a single file containing English text. What I mean is that there
has not been one ounce of energy expended on the part of the developer to make
clear the ideas behind the creation of your alleged masterpiece. Not an ounce
of consideration has been given to the administrator who must implement,
maintain, upgrade, support, troubleshoot, roll back, and yes, possibly even
hack it into submission using (gasp) actual code. There are no "gotchas" listed
anywhere in the README file, there is nothing but the default INSTALL file, and
the most thoughtfully created file in the entire tarball is the file that gives
thanks to all the people involved. And this is just the source tree I'm talking

GUI apps have a Help menu with nothing listed in there but an "about"
option, which pops up the slickest little thing I've ever seen. It must have
taken hours to create - and yet, there is no documentation worth looking at!
Please get your priorities straight, or delist your freshmeat project.

Sysadmins don't use your stuff because your documentation is disgusting.
Pick yourself up, dust yourself off, and go write some. The bigger problem for
you coders, really, is that there are usually 20 different packages on
freshmeat that all do the same thing. Of those 20, probably one or two have
real-life, usable documentation. With documentation, I can get to know
the software. Then I'll install it on a test box. If it works, great, I'm
tickled pink. If it doesn't quite work, then I'm interested in giving feedback,
because here's someone who will roll it back into the product or the
documentation. This is a useful cycle that benefits millions, not the least of
which is the coder! Documentation ends up resulting in a more mature product!
Wake up!

The code is NOT the documentation

The code might've been the documentation to some degree back when 90% of the
programs were written with 10% of the available programming languages.
Nowadays, this doesn't fly. As an administrator, my job is to keep up with the
latest and greatest in strategies and technologies involved in deploying
services for the benefit of some user base. I sift through information looking
for something relevant to my environment, filter that out, and start looking at
software options I can use to deploy a service in a particular way. Nowhere in
my job description does it say that I have to learn every new
programming language in the event that someone develops a
useful piece of software in it.

I'm not even supposed to be looking at what language it's done in. I shouldn't
have to care. You shouldn't want me to care. Hell, you shouldn't even want me
to know about the different programming languages! Think of all the Java
apps I might've deployed if not for the fact that I know it rather intimately
and avoid it like the plague! But if the code is the documentation, this pretty
much forces the issue of getting into the guts of all of these languages, not
to mention buying way too many O'Reilly books.

No, my job is to read the documentation available for the software. The goal in
this stage of research is to get the coder's philosophy on dealing with the
problem at hand. Hopefully, I can find an explanation that fits my brain. Once
that's done, it's time to read on through the technical side of the
documentation to gain an understanding of the implications of deploying said
software. What does the software need? How does it do its job? How does it
handle concurrent users? How much memory will it take under some example load?
What ports does it open? How do I secure this thing? Does this replace another
tool I'm already using? Why should I use it if I'm already using another app that
seems to do the same thing? What other pieces of software complement this one?
How is that software traditionally configured so that this software can take
advantage of it?

I could go on like this for days. Really. So can most other
administrators, about random packages, or no packages at all. Our job is to be
skeptical, lazy, and a bit suspicious. We ask questions as much to justify our
skepticism as to gain knowledge. Your job, as a coder, is to either answer a
reasonable number of our questions up front (and point to a place where even
more are answered), or get an open source devotee to volunteer to help you
write the documentation for you. Please do so.

Exceptions? There are none. I've seen CLI programs that take only five or six flags,
but there's nothing there to tell me how they actually do their job. Is
it largefile aware? Is it scriptable? Is it comparing values before or after
operation "x"? Sure, I'm an experienced systems guy, so I know how systems
generally work, and I understand a good number of system calls; this doesn't
mean I should have to run strace or tcpdump to get a
grip on what this thing is doing!

I have refrained from naming names here. It would serve no useful purpose, as my sysadmin colleagues can probably
think of exactly the projects I'm talking about (as can the respective coders).
I hope that this open letter gives someone some insight into how systems
administrators approach finding a solution to a problem.

Sincerely yours,
Brian Jones

Click Here!