Ensure systemd services restart on failure

I wrote a post a while ago covering the use of Monit to monitor services running and the use case I covered was to ensure these services restarted on failure. While a useful feature of Monit, it seems to be now a little redundant with SystemD having a built in restart feature.

Same use case where MySQL (or MariaDB in this case) is being killed by Apache’s oom killer.

I first copied the original systemd file associated with Mariadb from /usr/lib/systemd/system/mariadb.service to /etc/systemd/system/mariadb.service

Then under the [Service] section in the file i added the following 2 lines:

Restart=always
RestartSec=3

After saving the file we need to reload the daemon configurations to ensure systemd is aware of the new file

systemctl daemon-reload

Then restart the service to enable the changes.

systemctl restart mariadb

You can test this configuration by firing a kill to the process, e.g.:

 ps -ef|grep maria
mysql    22701 22542  0 06:52 ?        00:00:01 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --log-error=/var/log/mariadb/mariadb.log --pid-file=/var/run/mariadb/mariadb.pid --socket=/var/lib/mysql/mysql.sock
kill 22701
watch "ps -ef|grep maria"

You should see the process restart.

  • Thanks a lot for this. I just got done with a very very long server migration, have php-fpm stopping unexpectedly and it turning back on automatically will let me get some sleep! Overall it’s just wonderful and I had no idea you could do it, I put all my web services on auto-restart and it worked like a charm. Now I can snooze and figure out what’s wrong later!

    You rock.

  • Erik Moeller

    <3 Thanks from me as well, exactly what I was looking for. 🙂

  • unfortunately this fails when the services never came up in the first place. we have a scenario where apache tries to start to early and a nfs folder is not ready just then. so apache fails to start and thus never (re)starts, no matter what you configure. what does work, is a cronjob calling ‘systemctl start httpd’ every few minutes… (which could cause a lot of trouble, but is the only thing that works)

    • (btw not production only in a vagrant vm for local dev)

    • Ben Gosney

      A bit late to the game but you can add “nfs-client” to the after clause. This should mean that systemd doesn’t try to start apache until the nfs client is loaded.