May 25, 2016

Linux System Monitoring and More with Auditd


System monitoring
Learn how to use auditd for system monitoring.

One of the keys to protecting a Linux system is to know what’s going on inside it -- what files change, who accesses what and when, and which applications get run. Incrond was used up until some years ago for the former, but, despite rumors to the contrary, development seems to have stopped since about four years ago. Nonetheless, you can still download and use it and try out the examples I talked about in a tutorial published elsewhere.

The newer systemd contains some features that allow for monitoring, but it is a bit clumsy, and the feedback you get is far from detailed enough for a forensic analysis or to run an event-specific application.

These days, your best bet to monitor all your stuff is probably auditd. Auditd is also a good option because, apart from running comprehensive checks, the auditing itself happens at the kernel level, below userspace, which makes it much harder to subvert. This is an advantage over shell-based auditing systems, which will not give accurate information if the system is already compromised before they run.

Audit is actively developed by Red Hat and is available for most, if not all, major distributions. If it is not already installed on your system, you can find it by searching in your system's repositories. In Debian-based systems, the package is simply called audit, while in RPM-based systems, it shows up as auditd. In most Red Hat-related systems, such as Fedora and CentOS, auditd is usually installed by default.

Auditd is made up of several components, but for our purposes today, what you need are: auditd itself, which is the actual daemon that monitors the system, and aureport, a tool that generates reports culled from the auditd's logs.


First things first, though. Install the audit or auditd package using your distribution's software manager and check that it is running. Most modern Linux distributions run auditd as a systemd service, so you can use

> systemctl status auditd.service

to see if it’s active once installed. If it is there, but not running, you can jumpstart it with

> systemctl start auditd.service

or configure it to run at boot with

> systemctl enable auditd.service

Before checking reports and so on, let it run for a while, so it can fill up its logs with events.

Reporting System

Right out of the box, auditd already logs some stuff it deems critical, no extra configuration needed. You can check what it is looking up by running aureport without any arguments -- note that you must be root or have root privileges (i.e., use sudo) to access audit's toolbox:

> aureport
Summary Report 
Range of time in logs: 18/05/16 09:47:34.453 - 22/05/16 11:28:03.168 
Selected time for report: 18/05/16 09:47:34 - 22/05/16 11:28:03.168 
Number of changes in configuration: 195 
Number of changes to accounts, groups, or roles: 30 
Number of logins: 5 
Number of failed logins: 0 
Number of authentications: 136 
Number of failed authentications: 9 
Number of users: 4 
Number of terminals: 12 
Number of host names: 2 
Number of executables: 13 

So, this is already interesting! Take a look at the line that says Number of failed authentications: 9. If you see a large number here, somebody may be trying to access your machine by brute-forcing a user's password.

Let's dig in a little deeper:

> aureport -au
Authentication Report 
# date time acct host term exe success event 
1. 18/05/16 09:47:56 sddm ? ? /usr/lib/sddm/sddm-helper yes 187 
2. 18/05/16 09:48:09 paul ? ? /usr/lib/sddm/sddm-helper yes 199 
3. 18/05/16 09:41:29 root ? pts/1 /usr/bin/su yes 227 
4. 18/05/16 09:57:16 root ? pts/2 /usr/bin/su yes 231 
5. 18/05/16 10:01:57 root ? ? /usr/sbin/groupadd yes 235 

The -au option allows you to see details pertaining to authentication attempts: aureport gives you dates and times, the account being accessed, and whether the authentication was successful.

If you narrow down the output down by filtering it through grep:

> aureport -au | grep no
37. 18/05/16 12:18:24 root ? pts/0 /usr/bin/su no 217 
38. 18/05/16 12:18:38 root ? pts/0 /usr/bin/su no 218 
47. 18/05/16 12:41:15 root ? pts/1 /usr/bin/su no 262 
66. 18/05/16 14:09:55 root ? pts/4 /usr/bin/su no 220 
67. 18/05/16 14:10:05 root ? pts/4 /usr/bin/su no 221 
102. 20/05/16 12:37:07 root ? pts/5 /usr/bin/su no 191 
117. 21/05/16 12:10:39 root ? pts/0 /usr/bin/su no 229 
129. 21/05/16 17:59:08 root ? pts/1 /usr/bin/su no 208 
134. 21/05/16 18:32:05 root ? pts/0 /usr/bin/su no 248

you get all the failed authentication attempts. For our little experiment, remember the last line of the report, line 134, and the time and date, 21/05/16 18:32:05, of the last failed access attempt to access the root account.

Let's now look at the users report:

> aureport -u -i
User ID Report 
# date time auid term host exe event 
1. 18/05/16 09:47:35 unset ? ? /usr/lib/systemd/systemd 136 
2. 18/05/16 09:47:35 unset ? ? /usr/lib/systemd/systemd-update-utmp 137 
3. 18/05/16 09:47:35 unset ? ? /usr/lib/systemd/systemd 138 
4. 18/05/16 09:47:45 unset ? ? /usr/lib/systemd/systemd 139 
5. 18/05/16 09:47:45 unset ? ? /usr/lib/systemd/systemd 140 

The -u option tells aureport to show user activity, and -i tells it to show the user names instead of their ID numbers.

Again, aureport gives us a lot to sift through. What we want to know is who last tried to access root, but failed. If we copy the date and time of the last authentication attempt, that is, 21/05/16 18:32:05 and use that with grep to filter out some of the data, we get:

> aureport -u -i|grep "21/05/16 18:32:05"
2324. 21/05/16 18:32:05 paul pts/3 ? /usr/bin/su 201

Whoops! It was me. Let me clarify that I was not up to anything nefarious. It's just that I am a klutz and often mistype the root password in my own computer.

With a bit of command-line fu, you could do all of the above in one fell swoop:

> accessdate=`aureport -au | grep no | tail -1 | cut -d ' ' -f 2,3`; \
  aureport -u -i | grep "$accessdate"; unset accessdate
2324. 21/05/16 18:32:05 paul pts/3 ? /usr/bin/su 201

Or you could use a temporary intermediate file to show all failed authentication attempts:

> aureport -au | grep no | cut -d ' ' -f 2,3 > accessdates.log; \
  aureport -u -i | grep -f accessdates.log; rm accessdates.log
986. 18/05/16 14:09:55 paul pts/4 ? /usr/bin/su 220 
987. 18/05/16 14:10:05 paul pts/4 ? /usr/bin/su 221 
1577. 20/05/16 12:37:07 paul pts/5 ? /usr/bin/su 191 
1785. 21/05/16 12:10:39 paul pts/0 ? /usr/bin/su 229 
2050. 21/05/16 17:59:08 paul pts/1 ? /usr/bin/su 208 
2090. 21/05/16 18:32:05 paul pts/0 ? /usr/bin/su 248 

As I said, I am a bit of a klutz.

More to Come

There is, of course, a lot more to auditd. I haven't even touched upon what it is most often used for, that is, customized monitoring of files and directories. I'll be looking at how to do that and much more in future installments.

Click Here!