August 7, 2006

An open source security triple play

Author: Joe Barr

Want to protect your SOHO machine or LAN from rootkits and malware, but want something a little more real-time than simply running Chkrootkit or another rootkit detector after the fact? Consider
OSSEC-HIDS, an open source host intrusion detection system.

According the OOSEC-HIDS Web site, it's more than a host intrusion detection system (IDS). It's also a security event manager and a security information manager, which makes it the security equivalent of a hat trick in hockey, a triple-play in baseball, or a rare triple-double in basketball.

OSSEC-HIDS runs on both Windows and Linux/Unix. You can download the latest version along with the project's PGP public key, so you can verify the download.

You'll need to have GCC installed on your system in order to install OSSEC-HIDS, as the install script compiles the various tools as it runs. Using the install script is the recommended approach, though you are free to hack your way through the install manually if you prefer.

You can install OSSEC-HIDS in a client-server fashion, or simply on your desktop machine. The script will ask which you're doing, since the chores to be done depend on that. I selected a local installation, meaning it's just set up to run on one machine. If you're going to be running it on several machines, you'll want to install the server first, then the clients. Ubuntu users can find detailed installation instructions by Stephen Bunn on UDSF.

Only advanced users will need or want to choose anything but the default choices during the installation, so just press Enter now and then and watch the lines scroll by in the terminal. At the end of the procedure, the install script tries to set up an init script to start OSSEC-HIDS at boot time. Unfortunately, the script didn't recognize either of the two platforms I've installed it on: Ubuntu or SUSE SLED 10. Bunn's install tips page noted above shows how to rectify that on Ubuntu.

I was more interested in seeing it run than in getting it set up as an init script, at least at first, so I cranked it up by entering this command as root:

/var/ossec/bin/ossec-control start

Assuming you stuck with the default choices, you'll see these messages when it starts up for the first time:

Suse10-1:/home/warthawg # /var/ossec/bin/ossec-control start
Starting OSSEC HIDS v0.9 (by Daniel B. Cid)...
Started ossec-maild...
Started ossec-execd...
Started ossec-analysisd...
Started ossec-logcollector...
Started ossec-syscheckd...

OSSEC-HIDS keeps its configuration in /etc/ossec.conf in the directory it was installed in. If you kept the default directory, the full file spec for it would be /var/ossec/etc/ossec.conf.

Like Snort, the venerable open source network IDS, OSSEC-HIDS is rules-based. The default install puts 26 different rule sets in the rules directory. According to the manual, you can combine and extend the rules to achieve custom-tailored results, but I'm not ready to experiment with that yet.

Active response

In addition to checking your log files, searching for rootkits, and identifying changes to programs or other files in accordance with your configuration, OSSEC-HIDS can also perform active response measures in the event of an intrusion.

This is a somewhat controversial and possibly risky technique. Active response means that when certain thresholds or events occur, the software will execute predetermined commands or responses. A possible unintended side effect of active responses is to tip your hand to the attacker, who may then craft a denial of service attack based on your response. The benefits are that active response can stop certain illicit activities -- port scanning is an example -- and it also allows you to quickly close a possible window of vulnerability.

By default, OSSEC-HIDS blocks an attacking host IP address for 10 minutes using either /etc/host-deny or a firewall drop-IP command whenever a rules violation with a severity level of 6 or above is discovered. The installation process allows you to whitelist IP addresses that you never want to be blocked.

Tweaking the config

When you use all the defaults as recommended, you are going to receive alert messages from OSSEC-HIDS that you probably don't need or want to receive. You'll need to tweak the configuration in order to stop getting them.

One way to do that is by modifying the Syscheck section of the configuration file. By default, it looks like this, after the Windows directives are removed:

  <!-- Frequency that syscheck is executed - default every 2 hours -->

  <!-- Directories to check (perform all possible verifications) -->
  <directories check_all="yes">/etc,/usr/bin,/usr/sbin</directories>
  <directories check_all="yes">/bin,/sbin</directories>

  <!-- Files/directories to ignore -->


In addition to adding or ignoring directories, you can change the checks to be performed on those selected. To tweak the checks performed, set check_all to "no" and then set your choice of check_sum, check_size, check_owner, check_group, or check_perm to "yes".

You can also tweak the settings for rootkit checks and the level for mail alerts.

Is it real or is it Memorex?

If you're paranoid, false positives can scare you. I've installed OSSEC-HIDS on a couple of systems. On one box, I installed a preview release of SUSE SLED 10 three times, and as the first order of business each time, I followed the installation of the OS with an OSSEC-HIDS install. Each time I did so, I received the following alert:

** Alert 1154021500.0: mail
2006 Jul 27 12:31:40 Suse10-1->rootcheck
Rule: 14 (level 8) -> 'Rootkit detection engine message'
Src IP: (none)
User: (none)
Rootkit 'ZK' detected by the presence of file '/etc/sysconfig/console/load.zk'.

I've forwarded all the details to the OSSEC-HIDS community mailing list so that the developers can check it out. Hopefully it will turn out to be a false positive, and they will be able to avoid generating the alert next time. In the meantime, I am eyeing that box with suspicion.

OSSEC-HIDS can become a new standard for security, not just for system administrators but for SOHO users as well. It takes a bit of effort to configure it properly, but if you're concerned about security, as we all should be, it looks to me to be time well spent.


  • Security