The Eighth Commandment of system administration

A system log is one of the most effective ways to monitor a server’s health and underlying problems. Often before a major hardware or application crash takes place there are indicators of impending disaster within the syslog. As a good and attentive administrator, you should be reviewing your logs on a regular basis, but oftentimes these logs are forgotten due to other duties or important data is lost within pages of white noise telling about normal events.

VIII. Thou shalt not lose system logs when a server dies

When that impending disaster finally hits and the machine falls dead, it can be handy to have a copy of the log files stored safely somewhere else. Then you can immediately review the logs to see what you’re dealing with without having to spend time trying to pull the logs off the server, or worse yet, rebuilding the server without knowing what caused it to crash. The functionality to do this is already built into the Linux kernel, and it is easy to implement.

The syslog daemon has several options you can specify. Among the most useful is the remote logging feature. To enable it on what will be your remote syslog server, go to/etc/rc.d/init.d/syslog and open it with your favorite text editor. Find the line that reads

SYSLOGD_OPTIONS="-m 0"

and modify it so that it reads

SYSLOGD_OPTIONS="-m 0 -r"

Then you must find and modify the same line in/etc/sysconfig/syslog in the same manner. Once you’ve done this, the last thing to do is add the syslog service into/etc/services as 514/udp. When you add it, make sure you add it below isakmp to ensure numerical order within the file. When all this is done, restart syslog by entering the command

#/sbin/service syslog restart

Once syslog restarts, your machine will be listening on port 514 for syslog messages from other hosts.

Now all you need to do is configure those hosts to broadcast their syslogs to the syslog server. To do so, open the/etc/syslog.conf in your favorite text editor and add the following line:

*.* xxx.xxx.xxx.xxx

where xxx.xxx.xxx.xxx is the IP address of the remote syslog server you just configured. Once you’ve done this, restart syslog and check the remote syslog server. There should be an entry at the bottom of its log telling you that syslog has started on the remote machine.

You can configure as many hosts as you want to communicate with the syslog server. The best part is that many devices, such as firewalls, wireless access points, printers, switches, and spam blockers, all allow for remote syslog. This technique lets you keep their log files in one central location.

The commandments so far:
I. Thou shalt make regular and complete backups
II. Thou shalt establish absolute trust in thy servers
III. Thou shalt be the first to know when something goes down
IV. Thou shalt keep server logs on everything
V. Thou shalt document complete and effective policies and procedures
VI. Thou shalt know what cable goes where
VII. Thou shalt use encryption for insecure services
VIII. Thou shalt not lose system logs when a server dies