Setting up server tools for spam- and virus-free mail

It is amazing how much easier it is to get through an inbox uncluttered by unwanted messages.
After a week of email nearly free of spam and viruses, the time and effort it took to configure a Linux mail server with SpamAssassin, MIMEDefang, and sendmail seem well worth the trouble.

This process started when I decided to see if my Linux skills had advanced enough to replace an ailing Mac OS X 10.1 mail server. The Mac was easy to set up and administer, but AppleMail was just not stable, rarely running longer than a week without crashing, and Apple’s suggested fix — upgrade to server 10.2 or 10.3 — carried a $500 price tag. Ouch.

So, instead, I sent off for a set of Red Hat Linux 9 disks for $6.95, and turned a 1 GHz Pentium 4 machine into a new mail server with sendmail, running POP3 and IMAP (the details). After a few minor misadventures, I had a plain vanilla mail server that was stable and reliable.

My experience with the mail server taught me that a good approach is to research a topic thoroughly before beginning. Not only is it wise to line up the HOWTOs, FAQs, and sundry info like relevant directories, config file names, and coding conventions, but it’s also a good idea to read the relevant newsgroups, online forums, and archived mailing lists as well. I can usually follow well-written HOWTOs, but what often stops me is unexpected errors caused by configuration variations and other Linux issues. Newsgroups and mailing lists often flag common problems and their fixes.

Googling turned up lots of ways to use SpamAssassin with mail servers. One strategy involves setting up a separate proxy server that checks mail, flags spam, then relays it to the real mail server. Another way is to configure SpamAssassin to work with the MTA, be it sendmail, Qmail, or Postfix. Finally, you could configure SpamAssassin to work with procmail.

In all three cases, SpamAssassin runs the mail through its Bayesian filters, and optionally checks each mail against Vipul’s Razor or DCC, two spam “clearinghouses” that can help improve spam detection. After processing, mail identified as spam has a new header, X-Spam-Score:, with a score, expressed in integers and asterisks, and an attached text message explaining why SpamAssassin thinks it’s spam.

Searching also turned up MIMEDefang, a mail filter that identifies and removes viruses and can call SpamAssassin as well. A good MIMEDefang/SpamAssassin/sendmail HOWTO written and maintained by Mickey Hill covers the whole process with clear instructions and command-line recipes.

Lastly, I wanted to put Web interfaces on the server so that it would be easy to manage and users could get their mail via a browser. Webmin and SquirrelMail seemed to fill the respective bills.

I downloaded and installed the RPM package for Webmin without a problem. The only glitch came when I tried to log in: I had forgotten to open port 10,000 on the firewall. That solved, I logged in and set up Webmin security to accept connections only from my LAN. I then created new accounts with secure passwords for all the users. During setup I found the book Managing Linux Systems with Webmin very helpful.

I next installed SquirrelMail to provide a Web mail interface. At first I couldn’t get SquirrelMail to accept any of the configuration choices I made, like using my logo and organization name. The culprit turned out to be an alias that the SquirrelMail installer had set in Apache’s httpd.conf file, which is noted but less than clearly explained in the install instructions.

After you’ve installed SquirrelMail in Apache’s document root path (/var/www/html/ in my case) and changed the directory name from SquirrelMail-x-x to, say, “webmail,” you need to reset the path and stop and start Apache. Using Webmin’s Apache Module (Virtual Servers>Default Server>Aliases and Redirects) I found the line that was aliasing /webmail to another directory and changed it to /var/www/html/webmail/. All was fine after I restarted Apache using Webmin’s Apply Changes tab.

It’s a good idea to follow up the SquirrelMail Quick and Dirty install by setting users and permissions as noted at the end of the instructions. Neglecting security issues has a way of biting you later on.

Next up were some Perl module installations that SpamAssassin needs. Mickey Hill’s HOWTO lists them and provides instructions for FTPing the files, unpacking them, and the running make and install from the command line. You can also install the modules using CPAN. I tried a couple of modules both ways and everything appeared to behave the same, errors and all.

On my stock Red Hat 9 machine several of the modules including SpamAssassin generated a segment error and would not install properly. I got stuck on this for a few frustrating hours until an extended Google search turned up a bug fix on the CPAN site under the Time-HiRes module. If you see the segment error, issue the command:

export LC_ALL=C

before issuing the Build or make commands.

Once all the Perl modules and SpamAssassin are installed without errors, follow the instructions for Optional Additional Modules at the bottom of the SpamAssassin Install page. I chose to add the DCC module after reading a number of posts by mail server administrators saying it helped improve the accuracy of SpamAssassin’s filtering.

Now it was time to dive into the MIMEDefang HOWTO. I read through the whole document, trying to anticipate problems. The first task was to recompile sendmail, which made me a little nervous. The HOWTO offers a long list of steps. The only difference between my system and the HOWTO instructions was that the user smmsp already existed on my system, and with a different UID than the HOWTO specifies. This configuration won’t compile in a couple of features present in the default Red Hat 9 version of sendmail. One is Transport Layer Security (TLS) support, which is important if you want traveling users to be able to authenticate to the mail server from other systems and ISPs. To enable it, you need to add a couple of lines to the devtools/Site/site.config.m4 file before you build sendmail.

Also worth noting is the sample sendmail init script. It includes two lines to start and stop MIMEDefang in the correct sequence when starting and stopping sendmail. (MIMEDefang should start before, and stop after, sendmail.)

On its own, MIMEDefang does a good job of catching Windows viruses that come in the form of the usual executable attachments. The MIMEDefang HOWTO has a lot of documentation about customizing MIMEDefang for various applications, but I decided to start out with a near-default setup. The only change I made was not to strip attachments from suspicious email; instead, MIMEDefang inserts a warning at the beginning of the mail, and users can filter based on the warning if they wish.

Since DNS is vital to a correctly running mail server, I set up a caching-only name server on the mail machine using Webmin’s BIND admin module and configuration notes I found on the Web. DNS and BIND, 4th Edition is a good reference if you want to know more about DNS.

At this point, everything was installed and set up. I ran the tests suggested by the MIMEDefang HOWTO, and, after everything checked out, went to bed. The next day, I was pleased to see that almost every piece of the usual overnight deluge of spam was correctly marked. After a week, I had 567 messages in my inbox and 2,668 messages in the spam folder that I set up in my mail client as a precaution against false positives. MIMEDefang had caught about a half-dozen instances of the W32.Mimail.X virus and so marked them

For now I’m watching the server and reading up on the next features I want to add, including authentication for the SMTP server and filters and other modules to extend SquirrelMail. I’d also like to doll up the Webmail interface and include instructions for setting up filters to catch the marked spam in common mail clients. With the time I’m saving by not deleting spam, I may actually get some of these things done.

Chris Gulker, a Silicon Valley-based freelance technology writer, has authored more than 130 articles and columns since 1998. He shares an office with 7 computers that mostly work, an Australian Shepherd, and a small gray cat with an attitude.