Monit - monitor your processes and services simply
By Jon Archer
Monit is an application I’ve been meaning to setup for a while, I was first made aware of it from a chap I had the pleasure of talking to at OggCamp this year, he seemed to use it to the n’th degree to monitor files and services within docker containers to ensure a development environment was as it should be. This was far more than I really needed, but the monitoring of services definitely caught my attention so I set about installing and configuring. I was pleasantly surprised with the result, and how simple the whole process was.
Scenario: small hosting server with low spec, occasionally gets hit with a large amount of traffic resulting in either apache or mysql dying.
Configuration: In this instance a CentOS 6 server with standard LAMP stack, but i’m sure this will work with other distributions such as Fedora or CentOS 7 just replacing the relevant commands for systemd based commands.
How?
First off lets install monit, this comes from the rpmforge (http://repoforge.org/) repositories so if you haven’t already got them installed do so
yum localinstall http://pkgs.repoforge.org/rpmforge-release/rpmforge-release-0.5.3-1.el6.rf.x86_64.rpm
It would be worth checking the website to ensure that rpm version is correct (http://repoforge.org/use/)
Once thats installed we can install the monit software
yum install monit
lets enable the service to start on boot, and also start it to ensure it works OK before configuring:
chkconfig monit on
service monit start
Note: if using a systemd based distro such as Fedora or CentOS 7 then systemctl commands will need to be used instead of the above (systemctl enable monit and systemctl start monit)
If all is good then we can now tailor the configuration to our needs, monit uses the common approach for config files by having a master config at /etc/monit.conf which also reads in files from /etc/monit.d/. The only directive I changed in the master config file was to uncomment the following line:
set logfile syslog facility log_daemon
Which turns on logging, whether this is needed further down the line is to be decided but for now its great to have during configuration.
Next we can create some config files in /etc/monit.d/ for our services (apache httpd and mysql in this case)
vi /etc/monit.d/mysqld.conf
check process mysqld with pidfile /var/run/mysqld/mysqld.pid start program = "/sbin/service mysqld start" stop program = "/sbin/service mysqld stop" if failed host 127.0.0.1 port 3306 then restart if 5 restarts within 5 cycles then timeout
vi /etc/monit.d/httpd.conf
check process httpd with pidfile /var/run/httpd/httpd.pid start program = "/sbin/service httpd start" stop program = "/sbin/service httpd stop" if failed host 127.0.0.1 port 80 then restart if 5 restarts within 5 cycles then timeout
These two config files will check the pid files for activity outside monit, namely if the process stops without monit stopping it, and take action based on the status. The also monitor the respective tcp ports for the particular applications, 3306 for mysqld and 80 for apache.
Note: these configurations should also work with Debian based distributions but check the location of the pid files, also the service names are slightly different (mysql and apache2 if memory serves correctly).
Lets restart Monit and run some tests, for this I will run a tail on the log file while stopping services and killing processes:
tailf /var/log/messages
service monit restart
[root@web1 monit.d]# service monit restart Stopping monit: Dec 31 12:20:56 web1 monit[5338]: Shutting down monit HTTP server Dec 31 12:20:56 web1 monit[5338]: monit HTTP server stopped Dec 31 12:20:56 web1 monit[5338]: monit daemon with pid [5338] killed Dec 31 12:20:56 web1 monit[5338]: 'web1' Monit stopped [ OK ] Starting monit: Starting monit daemon with http interface at [localhost:2812] [ OK ] Dec 31 12:20:57 web1 monit[6232]: Starting monit daemon with http interface at [localhost:2812] [root@web1 monit.d]# Dec 31 12:20:57 web1 monit[6236]: Starting monit HTTP server at [localhost:2812] Dec 31 12:20:57 web1 monit[6236]: monit HTTP server started Dec 31 12:20:57 web1 monit[6236]: 'web1' Monit started
Lets stop mysqld
service mysqld stop
[root@web1 monit.d]# service mysqld stop Stopping mysqld: [ OK ] [root@web1 monit.d]# service mysqld stop Stopping mysqld: [ OK ]
Within seconds an entry in the log file is presented:
Dec 31 12:22:57 web1 monit[6236]: 'mysqld' process is not running Dec 31 12:22:57 web1 monit[6236]: 'mysqld' trying to restart Dec 31 12:22:57 web1 monit[6236]: 'mysqld' start: /sbin/service [root@web1 monit.d]# service mysqld status mysqld (pid 6526) is running...
OK so that worked nicely, lets try something a little less clean
[root@web1 monit.d]# ps -ef|grep mysqld root 6679 1 0 12:23 ? 00:00:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --socket=/var/lib/mysql/mysql.sock --pid-file=/var/run/mysqld/mysqld.pid --basedir=/usr --user=mysql mysql 6867 6679 1 12:23 ? 00:00:00 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --log-error=/var/log/mysqld.log --pid-file=/var/run/mysqld/mysqld.pid --socket=/var/lib/mysql/mysql.sock [root@web1 monit.d]# kill 6867 [root@web1 monit.d]# service mysqld status mysqld dead but subsys locked
And as if by magic:
Dec 31 12:25:59 web1 monit[6236]: 'mysqld' process is not running Dec 31 12:25:59 web1 monit[6236]: 'mysqld' trying to restart Dec 31 12:25:59 web1 monit[6236]: 'mysqld' start: /sbin/service
Brilliant, it seemed to perform exactly as expected. I wont bore you with the detail, but Apache restarted just the same. And that is it, a really easy to configure monitoring solution. Here, however, I was just scratching the surface of the monitoring capabilities. Take a look at the Monit website and wiki for more details on the vast array of configurables. http://mmonit.com/monit/documentation/ http://mmonit.com/monit/