3 Cool Linux Service Monitors
The Linux world abounds in monitoring apps of all kinds. We're going to look at my three favorite service monitors: Apachetop, Monit, and Supervisor. They're all small and fairly simple to use.
apachetop is a simple real-time Apache monitor. Monit monitors and manages any service, and Supervisor is a nice tool for managing persistent scripts and commands without having to write init scripts for them.
Monit is my favorite, because provides the perfect blend of simplicity and functionality. To quote
monit is a utility for managing and monitoring processes, files, directories and filesystems on a Unix system. Monit conducts automatic maintenance and repair and can execute meaningful causal actions in error situations. E.g. Monit can start a process if it does not run, restart a process if it does not respond and stop a process if it uses too much resources. You may use Monit to monitor files, directories and filesystems for changes, such as timestamps changes, checksum changes or size changes.
Monit is a good choice when you're managing just a few machines, and don't want to hassle with the complexity of something like Nagios or Chef. It works best as a single-host monitor, but it can also monitor remote services, which is useful when local services depend on them, such as database or file servers. The coolest feature is you can monitor any service, and you will see why in the configuration examples.
Let's start with its simplest usage. Uncomment these lines in
set daemon 120 set httpd port 2812 and use address localhost allow localhost allow admin:monit
Start Monit, and then use its command-line status checker:
$ sudo monit $ sudo monit status The Monit daemon 5.16 uptime: 9m System 'studio.alrac.net' status Running monitoring status Monitored load average [0.17] [0.23] [0.14] cpu 0.8%us 0.2%sy 0.5%wa memory usage 835.7 MB [5.3%] swap usage 0 B [0.0%] data collected Mon, 04 Sep 2017 13:04:59
If you see the message "/etc/monit/monitrc:289: Include failed -- Success '/etc/monit/conf.d/*'" that is a bug, and you can safely ignore it.
Monit has a built-in HTTP server. Open a Web browser to http://localhost:2812. The default login is admin, monit, which is configured in
/etc/monit/monitrc. You should see something like Figure 1 (below).
Click on the system name to see more statistics, including memory, CPU, and uptime.
That is fun and easy, and so is adding more services to monitor, like this example for the Apache HTTP server on Ubuntu.
check process apache with pidfile /var/run/apache2/apache2.pid start program = "service apache2 start" with timeout 60 seconds stop program = "service apache2 stop" if cpu > 80% for 5 cycles then restart if totalmem > 200.0 MB for 5 cycles then restart if children > 250 then restart if loadavg(5min) greater than 10 for 8 cycles then stop depends on apache2.conf, apache2 group server
Use the appropriate commands for your Linux distribution. Find your PID file with this command:
echo $(. /etc/apache2/envvars && echo $APACHE_PID_FILE)
The various distros package Apache differently. For example, on Centos 7 use
systemctl start/stop httpd.
After saving your changes, run the syntax checker, and then reload:
$ sudo monit -t Control file syntax OK $ sudo monit reload Reinitializing monit daemon
This example shows how to monitor key files and alert you to changes. The Apache binary should not change, except when you upgrade.
check file apache2 with path /usr/sbin/apache2 if failed checksum then exec "/watch/dog" else if recovered then alert
This example configures email alerting by adding my mailserver:
set mailserver smtp.alrac.net
monitrc includes a default email template, which you can tweak however you like.
man monit is well-written and thorough, and tells you everything you need to know, including command-line operation, reserved keywords, and complete syntax description.
apachetop is a simple live monitor for Apache servers. It reads your Apache logs and displays updates in realtime. I use it as a fast easy debugging tool. You can test different URLs and see the results immediately: files requested, hits, and response times.
$ apachetop last hit: 20:56:39 atop runtime: 0 days, 00:01:00 20:56:56 All: 12 reqs ( 0.5/sec) 22.4K ( 883.2B/sec) 1913.7B/req 2xx: 6 (50.0%) 3xx: 4 (33.3%) 4xx: 2 (16.7%) 5xx: 0 ( 0.0%) R ( 30s): 12 reqs ( 0.4/sec) 22.4K ( 765.5B/sec) 1913.7B/req 2xx: 6 (50.0%) 3xx: 4 (33.3%) 4xx: 2 (16.7%) 5xx: 0 ( 0.0%) REQS REQ/S KB KB/S URL 5 0.19 17.2 0.7*/ 5 0.19 4.2 0.2 /icons/ubuntu-logo.png 2 0.08 1.0 0.0 /favicon.ico
You can specify a particular logfile with the
-f option, or multiple logfiles like this:
apachetop -f logfile1 -f logfile2. Another useful option is
-l, which makes all URLs lowercase. If the same URL appears as both uppercase and lowercase it will be counted as two different URLs.
Supervisor is a slick tool for managing scripts and commands that don't have init scripts. It saves you from having to write your own, and it's much easier to use than systemd.
On Debian/Ubuntu, Supervisor starts automatically after installation. Verify with
$ ps ax|grep supervisord 7306 ? Ss 0:00 /usr/bin/python /usr/bin/supervisord -n -c /etc/supervisor/supervisord.conf
Let's take our Python hello world script from last week to practice with. Set it up in
[program:helloworld.py] command=/bin/helloworld.py autostart=true autorestart=true stderr_logfile=/var/log/hello/err.log stdout_logfile=/var/log/hello/hello.log
Now Supervisor needs to re-read the
conf.d/ directory, and then apply the changes:
$ sudo supervisorctl reread $ sudo supervisorctl update
Check your new logfiles to verify that it's running:
$ sudo supervisorctl reread helloworld.py: available carla@studio:~$ sudo supervisorctl update helloworld.py: added process group carla@studio:~$ tail /var/log/hello/hello.log Hello World! Hello World! Hello World! Hello World!
Visit Supervisor for complete and excellent documentation.