Is this the correct way to install the watchdog package?

[mrpwww]

Hi paulv,

thanks for your research - I also noticed that the hw watchdog became nonfunctional when I switched to jessie but I never found the time to dig any deeper.

Looking at your bug report of May 2016 at
https://bugs.launchpad.net/raspbian/+bug/1582707
I was amazed to find it with status "Unassigned" and importance "Undecided".
Then I thought, maybe we should just turn up the "heat" on this bug report (see the little flame above the milestone column).

But after looking at the kind of bugs posted on
https://bugs.launchpad.net/raspbian/
I began to wonder if that bug tracker is the right place to ask for a fix to this problem. After all, it is associated to Canonical. It seems to deal mainly with bugs related to apps and tools that Raspbian inherits from Debian via Ubuntu.

Therefore, may I suggest that you open a bug at one of these trackers:
https://github.com/raspberrypi/firmware/issues
or
https://github.com/RPi-Distro/repo/issues
I'd assume that over there we might get some response from the developers.

Best regards,
Michael

[paulv]

Thank you for the tips Michael!

I submitted the same report to both suggested github locations.
Maybe we'll get a response this time.

Regards,

Paul

[davefi]

Hi all,

These seems to be one of the most recent and complete threads on the internet for the watchdog on the Pi. But I am still having problems and am still confused.

Rather than rehashing what I have faced, I will simply ask the following:

What, exactly, do I need to do, to enable the watchdog to restart the Pi if it hangs/kernel panics ?

To expand, I have read, multiple times, and tried to follow things, but I am getting confused between the hardware watchdog and the (two?) software watchdogs, and what I need to enable etc.

The watchdog package seems to be the one I want, as it has the options for max-load, low memory etc. It appears from what I can tell, that the hardware watchdog is enabled by default now:

My Jessie Lite pi:

Code: Select all

Linux jessie 4.9.24-v7+ #993 SMP Wed Apr 26 18:01:23 BST 2017 armv7l GNU/Linux

Code: Select all

ls -al /dev/watchdog*
0 crw------- 1 root root  10, 130 Aug 16 00:09 /dev/watchdog
0 crw------- 1 root root 252,   0 Aug 16 00:09 /dev/watchdog0

My Stretch Lite pi:

Code: Select all

Linux stretch 4.9.41+ #1023 Tue Aug 8 15:47:12 BST 2017 armv6l GNU/Linux

Code: Select all

ls -al /dev/watchdog*
0 crw------- 1 root root  10, 130 Aug 18 12:53 /dev/watchdog
0 crw------- 1 root root 252,   0 Aug 18 12:53 /dev/watchdog0

So leaving that as-is, do I simply install watchdog package, edit /etc/watchdog.conf and then it should work ?

ie. I completely ignore /etc/systemd/system.conf ?

Thanks for any pointers!

[epoch1970]

do I simply install watchdog package, edit /etc/watchdog.conf and then it should work ?

ie. I completely ignore /etc/systemd/system.conf ?

This is what I do and as far as I see it works:
- install raspbian lite,
- apt-get install watchdog
- edit /etc/default/watchdog and /etc/watchdog.conf
Edit: - Fiddle with the watchdog systemd unit, otherwise it doesn't start. I think this is still necessary. Last message here: https://bugs.launchpad.net/raspbian/+bug/1582707
- add some scripts if I want a custom test or repair policy. (I always want that, because the watchdog device has a timeout of 15secs on Pi and I don't want to run every test every 10 secs...)

Earlier, loading the relevant Linux module was necessary. Then it came compiled in the kernel, as you have noted. So no need to load anything.
Recently -AFAIK- an overlay was added but my guess is it allows to disable the hardware watchdog and it is enabled by default. I've never used this overlay.
I know systemd has some sort of feature they call watchdog. I suppose this is configured in systemd/system.conf. I've never used that either.

[davefi]

This is what I do and as far as I see it works:
- install raspbian lite,
- apt-get install watchdog
- edit /etc/default/watchdog and /etc/watchdog.conf
Edit: - Fiddle with the watchdog systemd unit, otherwise it doesn't start. I think this is still necessary. Last message here: https://bugs.launchpad.net/raspbian/+bug/1582707
- add some scripts if I want a custom test or repair policy. (I always want that, because the watchdog device has a timeout of 15secs on Pi and I don't want to run every test every 10 secs...)

Earlier, loading the relevant Linux module was necessary. Then it came compiled in the kernel, as you have noted. So no need to load anything.
Recently -AFAIK- an overlay was added but my guess is it allows to disable the hardware watchdog and it is enabled by default. I've never used this overlay.
I know systemd has some sort of feature they call watchdog. I suppose this is configured in systemd/system.conf. I've never used that either.

Thank you epoch1970, after reading another gazillion posts (including some of yours), I think I have it working (on Jessie Lite) now. I'm getting some weirdness on Stretch Lite, but that maybe because I have tried so many things I have messed something else up! I'll reimage and try again over the weekend.

One thing that does seem fixed on Stretch is the /lib/systemd/system/watchdog.service [Install] issue. That is now set to
WantedBy=default.target which is what I used on Jessie Lite, and it worked with no issues.

So, for the interest of clarity, this is exactly what I did, and what I get:

Code: Select all

1	sudo nano /boot/config.txt	
	Add the following to the end of the file:	
	dtparam=watchdog=on	
2	sudo apt-get install watchdog chkconfig	
3	sudo nano /etc/default/watchdog	
	Change to:	
	watchdog_module="bcm2835_wdt"	
4	sudo nano /etc/watchdog.conf	
	Configure as required:	
	max-load-1              = 24
	Enable device:	
	watchdog-device /dev/watchdog
	Set the timeout:
	watchdog-timeout = 14	
5	sudo systemctl start watchdog	
6	sudo systemctl -l status watchdog	
	Check for errors	
7	sudo nano /lib/systemd/system/watchdog.service	
	Change to:	
	[Install]	
	WantedBy=default.target	
9	sudo systemctl daemon-reload	
10	sudo systemctl enable watchdog
11	sudo systemctl -l status watchdog	

And the output I get is:

Code: Select all

● watchdog.service - watchdog daemon
   Loaded: loaded (/lib/systemd/system/watchdog.service; enabled)
   Active: active (running) since Fri 2017-08-18 15:10:05 BST; 6min ago
 Main PID: 9609 (watchdog)
   CGroup: /system.slice/watchdog.service
           └─9609 /usr/sbin/watchdog

Aug 18 15:10:05 hassbian watchdog[9609]: int=1s realtime=yes sync=no soft=no mla=24 mem=0
Aug 18 15:10:05 hassbian watchdog[9609]: ping: no machine to check
Aug 18 15:10:05 hassbian watchdog[9609]: file: no file to check
Aug 18 15:10:05 hassbian watchdog[9609]: pidfile: no server process to check
Aug 18 15:10:05 hassbian watchdog[9609]: interface: no interface to check
Aug 18 15:10:05 hassbian watchdog[9609]: temperature: no sensors to check
Aug 18 15:10:05 hassbian watchdog[9609]: test=none(0) repair=none(0) alive=/dev/watchdog heartbeat=none to=root no_act=no force=no
Aug 18 15:10:05 hassbian watchdog[9609]: watchdog now set to 14 seconds
Aug 18 15:10:05 hassbian watchdog[9609]: hardware watchdog identity: Broadcom BCM2835 Watchdog timer
Aug 18 15:10:05 hassbian systemd[1]: Started watchdog daemon.

[MMOHQ]

Thank you so much for this thread. I have to research how to setup watchdog every time I setup a new Pi, but after sitting down and reading this, not only did it get watchdog working on my new Pi Zero W running, I believe, Jessie Lite...but I also was able to copy it over to a script that should theoretically automate the setup for me in the future.

Figured I'd drop the shell script here for anyone who might want to look it over, improve upon it, or for anyone else who also sets up Pi's a little too often

Oh and I decided to add the max-load-1 part to it, figuring that it's better to be safe than to have a Pi freeze up.. Curious if anyone found the 5 or 15 variant to be a better alternative.

Code: Select all

#!/bin/bash
echo
echo "Setting up watchdog for Raspberry Pi..."
echo

if [ "$EUID" -ne 0 ]
  then echo "Please run this script as root to properly setup watchdog."
  echo
  exit
fi

echo "Checking for watchdog parameter in boot config..."
echo
grep -q -F "dtparam=watchdog=on" /boot/config.txt || sudo sh -c 'echo "\n# activating the hardware watchdog\ndtparam=watchdog=on" >> /boot/config.txt'

echo "Installing watchdog service..."
echo
sudo apt-get install watchdog

echo
echo "Enabling watchdog max-load-1 for protection from system freeze..."
echo
sudo sed -i '/#max-load-1 /c\max-load-1             = 24' /etc/watchdog.conf

echo "Adding watchdog to the boot sequence..."
echo
sudo systemctl enable watchdog

echo
echo "Done! Watchdog should now be enabled."
echo

Nothing too fancy, but it should cut down much of the original post to simply stepping through each part real quick and getting the job done fast. I'd like to say it should work on any newer installation of Raspbian, where it uses the new watchdog drivers (or whatever they changed), assuming they don't go changing watchdog on us again.

[PetrSvetr]

Also because of your sceptical comments here, I started to investigate the possibility to add external watchdog to the Raspberry pi.
Thanks to an excellent support from PhilE I thing we have succeeded.
You can see our findings here: external watchdog on GPIO output

[paulv]

For those of you that still experience a not automatically rebooting RPi after a kernel crash, I happened to find a possible solution that probably came about in a later kernel version.

Try this:
sudo nano /etc/sysctl.conf

And add the following :
Kernel.panic=10

This will set the automatic reboot after 10 seconds of non-activity

Enjoy!

[paulv]

After all this time, I finally got a reply email from the Debian kernel folks. They have now my bug reports.

Here is a link to an updated Watchdog description for Debian: viewtopic.php?t=147501&p=972709
Note that in the text it says to activate the watchdog with 20seconds. This does not work on the RPi, and is related to a hardware timer limitation. The maximum is 10 seconds.

If you're interested in the application level support, you have to move up to one of my earlier posts on this topic, I have an example using Python.

The latest information is that to get the watchdog started on the RPi, you need to do this:

In /boot/config.txt add/change:

Code: Select all

watchdog=on

In /etc/systemd/systemd.conf, change #RuntimeWatchdogSec= to:

Code: Select all

RuntimeWatchdogSec=10s


I also added this information at the top of the first post, to help new users.

[no3rpi]

Thank you for update.

p.s.
Now the first message in the topic it is 99% confusing for the first time reader ( and not only them ) with old useless info still tagged with [EDITED] and [UPDATE] instead of removed from message.
Users will still think they need watchdog package installed and it is not true.

Code: Select all

Nov  2 16:34:10 rpi3 systemd[1]: Starting LSB: Start watchdog keepalive daemon...
Nov  2 16:34:10 rpi3 kernel: [    0.806283] bcm2835-wdt 3f100000.watchdog: Broadcom BCM2835 watchdog timer
Nov  2 16:34:11 rpi3 systemd[1]: Started LSB: Start watchdog keepalive daemon.

[paulv]

You are probably right, but then the whole post has to be re-written.
This will be a big PITA with the poor editing features, so I'm not going to do that.

I'll leave it for somebody else to write a post titled "This is the correct way to use the Watchdog"

Have fun!

Paul

[PlanB]

On my Jessie lite it's not /etc/systemd/systemd.conf but /etc/systemd/system.conf Should the watchdog work if I use 'poweroff' from the cmd line?

[PiGeon]

I have just registered to report my todays achievements with watchdog.

I have a Raspbian Lite installed and updated to following version

Code: Select all

Linux rpi3b 4.9.59-v7+ #1047 SMP Sun Oct 29 12:19:23 GMT 2017 armv7l GNU/Linux

I didn't need to enable the watchdog module nor put the "watchdog=on" to /boot/config.txt

the only thing I had to do was add the watchdog related lines to /etc/systemd/system.conf file

Code: Select all

echo "RuntimeWatchdogSec=10" | sudo tee -a /etc/systemd/system.conf
echo "ShutdownWatchdogSec=10min" | sudo tee -a /etc/systemd/system.conf
cat /etc/systemd/system.conf | grep "Watchdog"

then I could see these lines in system log after reboot (the first line was there since Rasbian setup)

Code: Select all

[email protected]:~ $ journalctl -b | grep dog
Nov 03 18:16:43 rpi3b kernel: bcm2835-wdt 3f100000.watchdog: Broadcom BCM2835 watchdog timer
Nov 03 18:16:43 rpi3b systemd[1]: Hardware watchdog 'Broadcom BCM2835 Watchdog timer', version 0
Nov 03 18:16:43 rpi3b systemd[1]: Set hardware watchdog to 10s.

and after firing the fork bomb scrript the RPI succesfully rebooted after arround 15 seconds

have fun

PiGeon

[sparkie777]

@paulv:
thank you very much sharing your watchdog knowhow with us.

I use the hardware watchdog throughout various raspbian releases in some of my applications. I now did some testing on raspbian/ stretch/ RPi3:

My experiences:

- you need not to inititalize/ load any drivers, dtoverlays etc.
- not any kind of special configuration is required
- don't install the watchdog package
- don't fiddle with systemd watchdog settings
- nonetheless the device "/dev/watchdog" is simply there
- you can write "." to the device "/dev/watchdog" at any time to start the functionality
- this triggers the hardware watchdog which expires in 15s per default (I don't know how to change this interval)
- i.e. the machine reboots unless you write "." again to the device "/dev/watchdog" prior to the 15s expiry period
- to stop the watchdog you simply write "V" to the device "/dev/watchdog"

most of my applications are running some sort of dispatch loop. So the only thing I have to worry about is
to rewrite the "." just in time.

trivial shell script to explain the watchdog behavior

Code: Select all

while :
do
    date
    echo . > /dev/watchdog
    sleep 14
done

if the loop fails to write the '.' just in time (i.e. prior to the 15s expiry period) the machine reboots instantly

[sparkie777]

I don't know how to change this interval

in the meantime I experimented a little how to configure the watchdog timeout (on raspbian/stretch 4.9.59+)

you can configure the timeout on the kernel cmdline like this (example for 10s timeout):

/boot/cmdline.txt:

Code: Select all

... bcm2835_wdt.heartbeat=10

be aware that a val of 15 is max due to hardware/firmware restrictions

[VascoFerraz]

The following solution (thanks Paul) works on the RPi1 model B (BCM2835)?

1)
In /boot/config.txt add/change:
watchdog=on

2.1)
In /etc/systemd/systemd.conf, change #RuntimeWatchdogSec= to:
RuntimeWatchdogSec=10s

2.2)
In /etc/systemd/systemd.conf, change #ShutdownWatchdogSec= to:
ShutdownWatchdogSec=10min

3)
sudo reboot

Thanks.

[sstream]

VascoFerraz » Tue Jan 02, 2018 6:03 am

The following solution (thanks Paul) works on the RPi1 model B (BCM2835)?

You don' t need 1).

Simply,

1.1)
In /etc/systemd/systemd.conf, change #RuntimeWatchdogSec= to:
RuntimeWatchdogSec=10s

1.2)
In /etc/systemd/systemd.conf, change #ShutdownWatchdogSec= to:
ShutdownWatchdogSec=10min

2)
sudo reboot

That's it.

[VascoFerraz]

Thanks. It is working

[Michaeli]

Hey Paul. I just followed your guide in an attempt to get this watch dog timer working. I think that it is working, but I am not really sure that it is until it hangs again. I did try your Fork bomb, and that made it puke and reboot, but that not real world enough. My question is this... My rpi runs as a mqtt server for a bunch of Sonoff wi-fi switches I have throughout my home. sometimes, it just stops responding. Will your setup here reboot the pi if it loses network connection? or just a lock up?

thanks in advance

[paulv]

The systemd version as I have described it will only cover a hung system or unresponsive app. There are a couple more things it can do, but not watch your internet connection, unless you write the code in your app to do that.

The "traditional" watchdog package has the ability to "watch" over many different parameters, the network being one of them.

I suggest you look at the man pages of the watchdog package or do a Google search.

Good hunting!