pfletch101
Posts: 423
Joined: Sat Feb 24, 2018 4:09 am

Send email on failure or restart of systemd service

Thu Apr 18, 2019 3:18 pm

I have a number of services, mostly monitoring environmental sensors, running on my workhorse Pi under systemd, most of which are set up to restart automatically on the (generally rare) occasions when they fail. I also have sendmail running as a mail server on the Pi. I would like to be notified by means of an email message if/when one of my services restarts, and/or (particularly) if one tries to restart and definitively fails. I have Googled around, but most of the suggestions I have found are either for a specific service or quite old. I have been bitten before by following obsolete 'recipes'! Could someone either point me to a current 'recipe' for this or give me some guidance? Thanks!

User avatar
DougieLawson
Posts: 35163
Joined: Sun Jun 16, 2013 11:19 pm
Location: Basingstoke, UK
Contact: Website Twitter

Re: Send email on failure or restart of systemd service

Thu Apr 18, 2019 5:48 pm

Easiest is to grab a copy of Nagios. That's been spamming my inbox for months.
Note: Having anything remotely humorous in your signature is completely banned on this forum.

Any DMs sent on Twitter will be answered next month.

This is a hippy & doctor free zone.

pfletch101
Posts: 423
Joined: Sat Feb 24, 2018 4:09 am

Re: Send email on failure or restart of systemd service

Thu Apr 18, 2019 6:19 pm

DougieLawson wrote:
Thu Apr 18, 2019 5:48 pm
Easiest is to grab a copy of Nagios. That's been spamming my inbox for months.
Thanks! I will take a look at it.

[EDIT] Having looked at it, it appears to be an impressive application, but it also seems to be extreme overkill for my relatively simple needs. Also of concern is the following cautionary text in the Linux installation instructions:
"Important: Nagios Enterprises highly recommends and will only support installing Nagios XI on a newly installed, "clean" system (a bare minimal install with nothing else installed or configured).
Attempting to install Nagios XI on a pre-existing system with other applications already installed can cause the Nagios XI installation process to fail, critical system components and settings (e.g. database servers) to be modified in a way that negatively affects other applications, and previously installed applications to be automatically upgraded or removed. While installing XI on a system with other applications is possible, it is not recommended due to the possible interactions and complexity of multiple components that are required for Nagios XI to function. If you choose to ignore these warnings, you do so at your own risk. Other Nagios products (Network Analyzer, Log Server and Fusion) should not co-exist on the same server."

Since I would want to install it on my 'workhorse' Pi, this would seem to rule it out. I have no other system that is running 24/7, and I don't really want to devote a second Pi to this. [/EDIT]

bls
Posts: 72
Joined: Mon Oct 22, 2018 11:25 pm

Re: Send email on failure or restart of systemd service

Fri Apr 19, 2019 10:27 pm

fail2ban will do exactly what you want. Although it's main purpose in life is to block turkeys trying to break into your system, it can easily be made to send mail when it sees something "interesting" (per your definition) in the logs. I have it email me whenever a successful connection is made to my VPN, for instance.

Have a look, and if you're at all interested, I can make the above available.

pfletch101
Posts: 423
Joined: Sat Feb 24, 2018 4:09 am

Re: Send email on failure or restart of systemd service

Sat Apr 20, 2019 1:24 am

After doing some more Googling around, I found a fairly recent 'recipe' (see link) for email notifications from systemd services which included enough detail to allow me to adapt it for my use. My system uses some features of both systemd and shell scripts which may not be well known to those who are not Unix aficionados, and I thought it worth while describing here, for others who may have similar needs.
You first need a shell script to send the email messages. This will expect a first command line parameter, which will be the name of the service on which it is reporting, and an optional second one, specifying a delay before it runs:

Code: Select all

#!/bin/sh
if ! [ -z "$2" ]; then # execute the next line if there is a second parameter on the script's command line
	sleep $2 # pause for the length of time indicated by the second parameter
fi

# I have sendmail installed on my Pi - use the appropriate syntax if you use a different mail handler
# The <<EOF....EOF sequence directs everything from the line below that on which the <<EOF appears to the
# second EOF to sendmail, creating an email message. I have the root address aliased to my regular email.
cat <<EOF | sendmail -t
To: root
From: root
Subject: Notification Email for $1 service

# Include the reporting service's status
$(systemctl status -l -n 50 "$1")
EOF
You then need to create a very simple systemd template 'Unit' file, and copy it to /etc/systemd/system. I called mine '[email protected]'. You can call yours what you like, but the last character before the '.' must be an '@':

Code: Select all

# %i is templated out to whatever is put after the
# @ when this is called from another systemd unit.
[Unit]
Description=Notification email for %i

[Service]
Type=oneshot
ExecStart=/home/pi/path_to_notify_script/notify.sh %i
You should use

Code: Select all

sudo systemctl daemon-reload
so that systemd recognizes the new service, but you don't need to enable it. You've now done most of the necessary work! Modify the systemd 'Unit' files that you want to monitor as per the following:

Code: Select all

[Unit]
Description=PV Solar Monitoring Service
StartLimitIntervalSec=100
StartLimitBurst=5
# the next line uses your shell script to send an email message if the service fails
# the name of the service is passed as the first and only parameter
[email protected]%n.service

[Service]
Type=simple
ExecStart=/usr/bin/python3 /home/pi/path_to_service_script/YourService.py
# the next line uses your shell script to send a message when the service starts (or restarts)
# the delay gives it time to start so that the status message is more useful
ExecStartPost=-/home/pi/path_to_notify_script/notify.sh %n 5s
TimeoutStartSec=60
TimeoutStopSec=5
Restart=on-failure
RestartSec=5

[Install]
WantedBy=default.target
That's it! You will now get an email message if your chosen services fail, and you will also know if they restarted successfully.
{Edited 4/20 to add reference to the source of the 'recipe' I modified}

bls
Posts: 72
Joined: Mon Oct 22, 2018 11:25 pm

Re: Send email on failure or restart of systemd service

Mon Apr 22, 2019 7:52 pm

Pretty Cool!

Return to “Advanced users”