paulv
Posts: 557
Joined: Tue Jan 15, 2013 12:10 pm
Location: Netherlands

Can't get watchdog package to mail or log

Tue Jul 09, 2019 8:43 pm

I have installed the watchdog package and tested various elements. They all work.
The only problem is that I can't get the watchdog package to email or to log.

I'm trying this on a Model B 3+ :
Linux raspi-cam 4.19.42-v7+ #1219 SMP Tue May 14 21:20:58 BST 2019 armv7l GNU/Linux
Here are the relevant details in /etc/watchdog.conf

Code: Select all

# ping the Utility ESP8266
ping                    = 192.168.1.2
# ping the router
ping                    = 192.168.1.1
interface               = wlan0

max-load-1              = 24
#max-load-5             = 18
#max-load-15            = 12

min-memory              = 1

retry-timeout           = 60
repair-maximum          = 1

watchdog-device = /dev/watchdog
# requirement for RPi's is <=15, otherwise default is 60 seconds
watchdog-timeout        = 15

admin                   = [email protected]
#interval               = 14
logtick                 = 4
log-dir                 = /var/log/watchdog

# This greatly decreases the chance that watchdog won't be scheduled before
# your machine is really loaded
realtime                = yes
priority                = 1

In /etc/default/watchdog I added:

Code: Select all

watchdog_options="-v"
to add verbose.

I also configured the ssmtp mailer.
Here are the entries in /etc/ssmtp/ssmtp.conf:

Code: Select all

#
# Config file for sSMTP sendmail
#
# The person who gets all mail for userids < 1000
# Make this empty to disable rewriting.
#root=postmaster
[email protected]

# The place where the mail goes. The actual machine name is required no
# MX records are consulted. Commonly mailhosts are named mail.domain.com
#mailhub=mail
mailhub=smtp.gmail.com:587

# Where will the mail seem to come from?
#rewriteDomain=

# The full hostname
hostname=raspi-cam

# Are users allowed to set their own From: address?
# YES - Allow the user to specify their own From: address
# NO - Use the system generated From: address
FromLineOverride=YES

AuthUser=gmail-account.com
AuthPass=Gmail-account-password
UseSTARTTLS=YES
UseTLS=YES
The mail program works fine when I send a test message:

Code: Select all

echo "test email" | mail -s "[Subject] test mail" [email protected]
When cron finds en error with an execution, it also sends me an email correctly.

Here are the logs for the shell test mail (at 21:53) and the cron test (at 22:00) and then the watchdog error message at 22:03:
Jul 9 21:53:06 localhost sSMTP[1019]: Creating SSL connection to host
Jul 9 21:53:06 localhost sSMTP[1019]: SSL connection using RSA_AES_128_CBC_SHA1
Jul 9 21:53:08 localhost sSMTP[1019]: Sent mail for [email protected] (221 2.0.0 closing connection d1sm11186084wrs.94 - gsmtp) uid=1000 username=pi outbytes=399

Jul 9 22:00:01 localhost sSMTP[1160]: Creating SSL connection to host
Jul 9 22:00:02 localhost sSMTP[1160]: SSL connection using RSA_AES_128_CBC_SHA1
Jul 9 22:00:03 localhost sSMTP[1160]: Sent mail for [email protected] (221 2.0.0 closing connection g12sm5558868wrv.9 - gsmtp) uid=0 username=root outbytes=663

Jul 9 22:03:03 localhost sSMTP[1257]: Unable to locate smtp.gmail.com
Jul 9 22:03:03 localhost sSMTP[1257]: Cannot open smtp.gmail.com:587

Here are the relevant lines in /etc/alternatives:
lrwxrwxrwx 1 root root 23 Jul 8 11:11 mail -> /usr/bin/mail.mailutils*
lrwxrwxrwx 1 root root 39 Jul 8 11:11 mail.1.gz -> /usr/share/man/man1/mail.mailutils.1.gz
lrwxrwxrwx 1 root root 23 Jul 8 11:11 mailx -> /usr/bin/mail.mailutils*
lrwxrwxrwx 1 root root 39 Jul 8 11:11 mailx.1.gz -> /usr/share/man/man1/mail.mailutils.1.gz
lrwxrwxrwx 1 root root 27 Jul 8 11:11 messages -> /usr/bin/messages.mailutils*
lrwxrwxrwx 1 root root 43 Jul 8 11:11 messages.1.gz -> /usr/share/man/man1/messages.mailutils.1.gz
lrwxrwxrwx 1 root root 27 Jul 8 11:11 movemail -> /usr/bin/movemail.mailutils*
lrwxrwxrwx 1 root root 43 Jul 8 11:11 movemail.1.gz -> /usr/share/man/man1/movemail.mailutils.1.gz
I'm at a total loss why the mail program works with the shell test and cron and not with the watchdog package.

The /var/log/watchdog directory stays empty, I can't get any logs recorded either.

I searched high and low on the internet, no help or clues.
Is there something I can do to make the watchdog package work?

Tks!
Last edited by paulv on Thu Jul 11, 2019 7:18 pm, edited 3 times in total.

epoch1970
Posts: 2976
Joined: Thu May 05, 2016 9:33 am
Location: Paris, France

Re: Can't get watchdog package to mail or log

Tue Jul 09, 2019 11:07 pm

From the current example configuration file in the upstream repo (emphasis mine):
# Email address for sending the reboot reason. This needs sendmail to
# be installed
and properly configured. Maybe you should just enable
# syslog forwarding instead?

#admin = root
Do you have ssmtp aliased to sendmail on that machine?

Why is it not logging? No idea. Normally I would use a much larger logtick, verbose and logging to syslog. I never had any issue with this combination, but I’ve never run Buster, and I have to wonder if I’ve run the watchdog program on Stretch.
"S'il n'y a pas de solution, c'est qu'il n'y a pas de problème." Les Shadoks, J. Rouxel

paulv
Posts: 557
Joined: Tue Jan 15, 2013 12:10 pm
Location: Netherlands

Re: Can't get watchdog package to mail or log

Wed Jul 10, 2019 6:29 am

Hi Epoch1970,

Thanks for the hints. I did not find the sendmail reference anywhere I looked, but I already installed sendmail to see if watchdog uses this mail program, but same result, no email. I completely removed/purged that and re-installed ssmtp again.

I have another Rpi Model B 3+ with the watchdog and the same settings and the same kernel version and that one finally works:
Message from watchdog:
The system will be rebooted because of error 101 = 'Network is unreachable'
I need to see if I can find the difference between the two or start with a fresh install.

Still no logging on either one though!

Well, I wanted to give it a new try on a fresh install. Unfortunately, Stretch is no longer available, so I had to use Buster-light. Unfortunately, things got a lot worse. smstp is deprecated on Buster, so you have to use msmtp now.
It took me a while to figure that out, but finally managed to get that package working.
I tried to contribute to the setting-up of msmtp here:https://www.raspberrypi.org/forums/view ... 7#p1496767

Unfortunately, as is so common with a major system change (sigh), cron and watchdog do not know how to use this package.

Here are the relevant entries in /var/log/mail.err :

Code: Select all

Jul 10 13:52:18 raspi-buster sSMTP[811]: Server didn't like our AUTH LOGIN (530 5.7.0 Must issue a STARTTLS command first. g3sm634346edh.24 - gsmtp)
Jul 10 13:53:24 raspi-buster sSMTP[824]: Server didn't like our AUTH LOGIN (530 5.7.0 Must issue a STARTTLS command first. k20sm634261ede.66 - gsmtp)
Jul 10 13:54:34 raspi-buster sSMTP[841]: Server didn't like our AUTH LOGIN (530 5.7.0 Must issue a STARTTLS command first. s47sm655035edd.40 - gsmtp)
Jul 10 13:56:02 raspi-buster sSMTP[854]: Cannot open smtp.gmail.com:587
Seems to me a significant miscommunication has been introduced with the changeover from ssmtp to msmtp.

I am in no position to figure this out. Anybody?
Last edited by paulv on Wed Jul 10, 2019 1:45 pm, edited 1 time in total.

epoch1970
Posts: 2976
Joined: Thu May 05, 2016 9:33 am
Location: Paris, France

Re: Can't get watchdog package to mail or log

Wed Jul 10, 2019 10:45 am

The other day there was a contributor wrangling dealing with linux watchdog, he was using another small footprint MTA and simply had to add a symbolic link to make emails work.
See: https://www.raspberrypi.org/forums/view ... 3#p1493909

Re. /etc/default/watchdog. IIRC, you want "-v" not "v".
"S'il n'y a pas de solution, c'est qu'il n'y a pas de problème." Les Shadoks, J. Rouxel

paulv
Posts: 557
Joined: Tue Jan 15, 2013 12:10 pm
Location: Netherlands

Re: Can't get watchdog package to mail or log

Thu Jul 11, 2019 7:12 pm

The symbolic link is part of the installation, it's already there.
I fixed the "-v" typo, thanks.
Last edited by paulv on Fri Jul 12, 2019 7:16 am, edited 1 time in total.

epoch1970
Posts: 2976
Joined: Thu May 05, 2016 9:33 am
Location: Paris, France

Re: Can't get watchdog package to mail or log

Thu Jul 11, 2019 8:36 pm

I posted a working msmtp config in the other thread you opened for that.
This is a few config bits I have on the same machine for linux watchdog. It does send mails and it logs to syslog.

/etc/default/watchdog:

Code: Select all

$ cat /etc/default/watchdog 
# Start watchdog at boot time? 0 or 1
run_watchdog=1
# Start wd_keepalive after stopping watchdog? 0 or 1
run_wd_keepalive=1
# Load module before starting watchdog
# useless on Raspbian: compiled in kernel
watchdog_module="bcm2835_wdt" 
# Specify additional watchdog options here (see manpage).
# -v: verbose activity message (every logtick)
watchdog_options="-v"
From /etc/watchdog.conf:

Code: Select all

...
# Note that this is the number of pages!
# To get the real size, check how large the pagesize is on your machine.
# getconf PAGE_SIZE -> 4096 - We ask for 200K minimum RAM
min-memory = 50
allocatable-memory = 1
...
# Max temp 85°C (Pi CPU rated ~100°C)
temperature-device = /sys/class/thermal/thermal_zone0/temp
max-temperature = 85
# This is only for watchdog version 6.0
#temperature-sensor = /sys/class/thermal/thermal_zone0/temp
#temperature-poweroff = yes
...
# Email address to send admin mail to. Defaults compiled into the binary (root)
#admin = root
...
# Default is 60, too much for bcm2835_wdt. The wd_keepalive daemon tries
# to set that. Val must be < 16 on Pi
watchdog-timeout = 15
# We run every 10 secs to straddle over the wd timeout.
interval = 10
# Verbose activity message once every 8 hours (28800/10 secs)
logtick = 2880
# Used for stderr/stdout of test and repair binaries
# We instead use syslog.
#log-dir = /var/log/watchdog
...
# This greatly decreases the chance that watchdog won't be scheduled before
# your machine is really loaded
realtime = yes
priority = 1
HTH
"S'il n'y a pas de solution, c'est qu'il n'y a pas de problème." Les Shadoks, J. Rouxel

paulv
Posts: 557
Joined: Tue Jan 15, 2013 12:10 pm
Location: Netherlands

Re: Can't get watchdog package to mail or log

Sun Jul 14, 2019 11:02 am

Well, I'm finally getting somewhere.
The watchdog package is logging to /var/log/syslog, but not to the log file specified in /etc/watchdog.conf.

In /etc/watchdog.conf, I have :

Code: Select all

interval                = 5
logtick                 = 360
log-dir                 = /var/log/watchdog

strangely enough, that directory stays completely empty.

Here is the syslog entry after a new start of watchdog:

Code: Select all

Jul 14 13:08:56 raspi-serv systemd[1]: Started watchdog daemon.
Jul 14 13:08:56 raspi-serv watchdog[24979]: List 'interface' added as 'wlan0'
Jul 14 13:08:56 raspi-serv watchdog[24979]: Integer 'max-load-1' found = 24
Jul 14 13:08:56 raspi-serv watchdog[24979]: Integer 'min-memory' found = 1
Jul 14 13:08:56 raspi-serv watchdog[24979]: Integer 'retry-timeout' found = 120
Jul 14 13:08:56 raspi-serv watchdog[24979]: Integer 'repair-maximum' found = 1
Jul 14 13:08:56 raspi-serv watchdog[24979]: String 'watchdog-device' found as '/dev/watchdog'
Jul 14 13:08:56 raspi-serv watchdog[24979]: Integer 'watchdog-timeout' found = 15
Jul 14 13:08:56 raspi-serv watchdog[24979]: String 'admin' found as '[email protected]'
Jul 14 13:08:56 raspi-serv watchdog[24979]: Integer 'interval' found = 5
Jul 14 13:08:56 raspi-serv watchdog[24979]: Integer 'logtick' found = 360
Jul 14 13:08:56 raspi-serv watchdog[24979]: String 'log-dir' found as '/var/log/watchdog'
Jul 14 13:08:56 raspi-serv watchdog[24979]: Variable 'realtime' found as 'yes' = 1
Jul 14 13:08:56 raspi-serv watchdog[24979]: Integer 'priority' found = 1
Jul 14 13:08:56 raspi-serv watchdog[24981]: starting daemon (5.15):
Jul 14 13:08:56 raspi-serv watchdog[24981]: int=5s realtime=yes sync=no load=24,18,12 soft=no
Jul 14 13:08:56 raspi-serv watchdog[24981]: memory: minimum pages = 1 free, 0 allocatable (4096 byte pages)
Jul 14 13:08:56 raspi-serv watchdog[24981]: ping: 192.168.1.2
Jul 14 13:08:56 raspi-serv watchdog[24981]: ping: 192.168.1.1
Jul 14 13:08:56 raspi-serv watchdog[24981]: file: no file to check
Jul 14 13:08:56 raspi-serv watchdog[24981]: pidfile: no server process to check
Jul 14 13:08:56 raspi-serv watchdog[24981]: interface: wlan0
Jul 14 13:08:56 raspi-serv watchdog[24981]: temperature: no sensors to check
Jul 14 13:08:56 raspi-serv watchdog[24981]: no test binary files
Jul 14 13:08:56 raspi-serv watchdog[24981]: no repair binary files
Jul 14 13:08:56 raspi-serv watchdog[24981]: error retry time-out = 120 seconds
Jul 14 13:08:56 raspi-serv watchdog[24981]: repair attempts = 1
Jul 14 13:08:56 raspi-serv watchdog[24981]: alive=/dev/watchdog heartbeat=[none] [email protected] no_act=no force=no
Jul 14 13:08:56 raspi-serv watchdog[24981]: watchdog now set to 15 seconds
Jul 14 13:08:56 raspi-serv watchdog[24981]: hardware watchdog identity: Broadcom BCM2835 Watchdog timer
Jul 14 13:09:06 raspi-serv watchdog[24981]: device wlan0 did not receive anything since last check
Jul 14 13:09:22 raspi-serv watchdog[24981]: device wlan0 did not receive anything since last check

Note this : String 'log-dir' found as '/var/log/watchdog'

Even after setting the access rights of this directory to 777, no dice...

Still hunting...

epoch1970
Posts: 2976
Joined: Thu May 05, 2016 9:33 am
Location: Paris, France

Re: Can't get watchdog package to mail or log

Sun Jul 14, 2019 12:32 pm

Mhh. Browsing the web reading Debian man pages and a few other sites,
I think now the watchdog daemon only logs to syslog, but “naturally” you can configure syslog to write to a separate file.
The log-dir stanza looks like a directory where the stdout/stderr of test- and repair- binaries only can be stored. Not the output of the daemon itself.

The way I see it, if you’re bent on seeing a specific file in the fs with watchdog messages in it ...
- ... and you’re running systemd-journald without “legacy” syslog daemon: stop thinking like that.
- ... and some variant of syslog is running on your machine: configure /etc/syslog.conf or somesuch to redirect messages issued by watchdog to your location of choice. Capability/syntax will vary according to the syslog implementation you’re using.
"S'il n'y a pas de solution, c'est qu'il n'y a pas de problème." Les Shadoks, J. Rouxel

paulv
Posts: 557
Joined: Tue Jan 15, 2013 12:10 pm
Location: Netherlands

Re: Can't get watchdog package to mail or log

Sun Jul 14, 2019 8:23 pm

Hi epoch1970,

Thanks for the heads-up. I'm beginning to think you're right about the logging.
As a minimum, this would mean that the documentation needs to be updated, especially the man pages for watchdog.conf. And then the default watchdog.conf file itself needs to be updated.

I'm still at a loss why watchdog sends emails on one system (both Stretch) and not on another, with, as far as I can tell, identical configurations. As mentioned above, I can't easily go back to a clean Stretch install because the downloads are gone. Not that it matters much, I can't get it to work on Buster either, but that may very well be due to other causes related to Buster itself.

I will continue the hunt...

Andyroo
Posts: 3255
Joined: Sat Jun 16, 2018 12:49 am
Location: Lincs U.K.

Re: Can't get watchdog package to mail or log

Sun Jul 14, 2019 8:54 pm

paulv wrote:
Sun Jul 14, 2019 8:23 pm
...
I can't easily go back to a clean Stretch install because the downloads are gone.
...
All the old releases are to be found stored by date at http://downloads.raspberrypi.org/raspbian/images/

The last cut of Stretch is http://downloads.raspberrypi.org/raspbi ... 019-04-09/
Need Pi spray - these things are breeding in my house...

Return to “Troubleshooting”