(Is this) The correct ways to install the watchdog package?

[mrpwww]

Hi paulv,

thanks for your research - I also noticed that the hw watchdog became nonfunctional when I switched to jessie but I never found the time to dig any deeper.

Looking at your bug report of May 2016 at
https://bugs.launchpad.net/raspbian/+bug/1582707
I was amazed to find it with status "Unassigned" and importance "Undecided".
Then I thought, maybe we should just turn up the "heat" on this bug report (see the little flame above the milestone column).

But after looking at the kind of bugs posted on
https://bugs.launchpad.net/raspbian/
I began to wonder if that bug tracker is the right place to ask for a fix to this problem. After all, it is associated to Canonical. It seems to deal mainly with bugs related to apps and tools that Raspbian inherits from Debian via Ubuntu.

Therefore, may I suggest that you open a bug at one of these trackers:
https://github.com/raspberrypi/firmware/issues
or
https://github.com/RPi-Distro/repo/issues
I'd assume that over there we might get some response from the developers.

Best regards,
Michael

[paulv]

Thank you for the tips Michael!

I submitted the same report to both suggested github locations.
Maybe we'll get a response this time.

Regards,

Paul

[davefi]

Hi all,

These seems to be one of the most recent and complete threads on the internet for the watchdog on the Pi. But I am still having problems and am still confused.

Rather than rehashing what I have faced, I will simply ask the following:

What, exactly, do I need to do, to enable the watchdog to restart the Pi if it hangs/kernel panics ?

To expand, I have read, multiple times, and tried to follow things, but I am getting confused between the hardware watchdog and the (two?) software watchdogs, and what I need to enable etc.

The watchdog package seems to be the one I want, as it has the options for max-load, low memory etc. It appears from what I can tell, that the hardware watchdog is enabled by default now:

My Jessie Lite pi:

Code: Select all

Linux jessie 4.9.24-v7+ #993 SMP Wed Apr 26 18:01:23 BST 2017 armv7l GNU/Linux

Code: Select all

ls -al /dev/watchdog*
0 crw------- 1 root root  10, 130 Aug 16 00:09 /dev/watchdog
0 crw------- 1 root root 252,   0 Aug 16 00:09 /dev/watchdog0

My Stretch Lite pi:

Code: Select all

Linux stretch 4.9.41+ #1023 Tue Aug 8 15:47:12 BST 2017 armv6l GNU/Linux

Code: Select all

ls -al /dev/watchdog*
0 crw------- 1 root root  10, 130 Aug 18 12:53 /dev/watchdog
0 crw------- 1 root root 252,   0 Aug 18 12:53 /dev/watchdog0

So leaving that as-is, do I simply install watchdog package, edit /etc/watchdog.conf and then it should work ?

ie. I completely ignore /etc/systemd/system.conf ?

Thanks for any pointers!

[epoch1970]

do I simply install watchdog package, edit /etc/watchdog.conf and then it should work ?

ie. I completely ignore /etc/systemd/system.conf ?

This is what I do and as far as I see it works:
- install raspbian lite,
- apt-get install watchdog
- edit /etc/default/watchdog and /etc/watchdog.conf
Edit: - Fiddle with the watchdog systemd unit, otherwise it doesn't start. I think this is still necessary. Last message here: https://bugs.launchpad.net/raspbian/+bug/1582707
- add some scripts if I want a custom test or repair policy. (I always want that, because the watchdog device has a timeout of 15secs on Pi and I don't want to run every test every 10 secs...)

Earlier, loading the relevant Linux module was necessary. Then it came compiled in the kernel, as you have noted. So no need to load anything.
Recently -AFAIK- an overlay was added but my guess is it allows to disable the hardware watchdog and it is enabled by default. I've never used this overlay.
I know systemd has some sort of feature they call watchdog. I suppose this is configured in systemd/system.conf. I've never used that either.

[davefi]

This is what I do and as far as I see it works:
- install raspbian lite,
- apt-get install watchdog
- edit /etc/default/watchdog and /etc/watchdog.conf
Edit: - Fiddle with the watchdog systemd unit, otherwise it doesn't start. I think this is still necessary. Last message here: https://bugs.launchpad.net/raspbian/+bug/1582707
- add some scripts if I want a custom test or repair policy. (I always want that, because the watchdog device has a timeout of 15secs on Pi and I don't want to run every test every 10 secs...)

Earlier, loading the relevant Linux module was necessary. Then it came compiled in the kernel, as you have noted. So no need to load anything.
Recently -AFAIK- an overlay was added but my guess is it allows to disable the hardware watchdog and it is enabled by default. I've never used this overlay.
I know systemd has some sort of feature they call watchdog. I suppose this is configured in systemd/system.conf. I've never used that either.

Thank you epoch1970, after reading another gazillion posts (including some of yours), I think I have it working (on Jessie Lite) now. I'm getting some weirdness on Stretch Lite, but that maybe because I have tried so many things I have messed something else up! I'll reimage and try again over the weekend.

One thing that does seem fixed on Stretch is the /lib/systemd/system/watchdog.service [Install] issue. That is now set to
WantedBy=default.target which is what I used on Jessie Lite, and it worked with no issues.

So, for the interest of clarity, this is exactly what I did, and what I get:

Code: Select all

1	sudo nano /boot/config.txt	
	Add the following to the end of the file:	
	dtparam=watchdog=on	
2	sudo apt-get install watchdog chkconfig	
3	sudo nano /etc/default/watchdog	
	Change to:	
	watchdog_module="bcm2835_wdt"	
4	sudo nano /etc/watchdog.conf	
	Configure as required:	
	max-load-1              = 24
	Enable device:	
	watchdog-device /dev/watchdog
	Set the timeout:
	watchdog-timeout = 14	
5	sudo systemctl start watchdog	
6	sudo systemctl -l status watchdog	
	Check for errors	
7	sudo nano /lib/systemd/system/watchdog.service	
	Change to:	
	[Install]	
	WantedBy=default.target	
9	sudo systemctl daemon-reload	
10	sudo systemctl enable watchdog
11	sudo systemctl -l status watchdog	

And the output I get is:

Code: Select all

● watchdog.service - watchdog daemon
   Loaded: loaded (/lib/systemd/system/watchdog.service; enabled)
   Active: active (running) since Fri 2017-08-18 15:10:05 BST; 6min ago
 Main PID: 9609 (watchdog)
   CGroup: /system.slice/watchdog.service
           └─9609 /usr/sbin/watchdog

Aug 18 15:10:05 hassbian watchdog[9609]: int=1s realtime=yes sync=no soft=no mla=24 mem=0
Aug 18 15:10:05 hassbian watchdog[9609]: ping: no machine to check
Aug 18 15:10:05 hassbian watchdog[9609]: file: no file to check
Aug 18 15:10:05 hassbian watchdog[9609]: pidfile: no server process to check
Aug 18 15:10:05 hassbian watchdog[9609]: interface: no interface to check
Aug 18 15:10:05 hassbian watchdog[9609]: temperature: no sensors to check
Aug 18 15:10:05 hassbian watchdog[9609]: test=none(0) repair=none(0) alive=/dev/watchdog heartbeat=none to=root no_act=no force=no
Aug 18 15:10:05 hassbian watchdog[9609]: watchdog now set to 14 seconds
Aug 18 15:10:05 hassbian watchdog[9609]: hardware watchdog identity: Broadcom BCM2835 Watchdog timer
Aug 18 15:10:05 hassbian systemd[1]: Started watchdog daemon.

[MMOHQ]

Thank you so much for this thread. I have to research how to setup watchdog every time I setup a new Pi, but after sitting down and reading this, not only did it get watchdog working on my new Pi Zero W running, I believe, Jessie Lite...but I also was able to copy it over to a script that should theoretically automate the setup for me in the future.

Figured I'd drop the shell script here for anyone who might want to look it over, improve upon it, or for anyone else who also sets up Pi's a little too often

Oh and I decided to add the max-load-1 part to it, figuring that it's better to be safe than to have a Pi freeze up.. Curious if anyone found the 5 or 15 variant to be a better alternative.

Code: Select all

#!/bin/bash
echo
echo "Setting up watchdog for Raspberry Pi..."
echo

if [ "$EUID" -ne 0 ]
  then echo "Please run this script as root to properly setup watchdog."
  echo
  exit
fi

echo "Checking for watchdog parameter in boot config..."
echo
grep -q -F "dtparam=watchdog=on" /boot/config.txt || sudo sh -c 'echo "\n# activating the hardware watchdog\ndtparam=watchdog=on" >> /boot/config.txt'

echo "Installing watchdog service..."
echo
sudo apt-get install watchdog

echo
echo "Enabling watchdog max-load-1 for protection from system freeze..."
echo
sudo sed -i '/#max-load-1 /c\max-load-1             = 24' /etc/watchdog.conf

echo "Adding watchdog to the boot sequence..."
echo
sudo systemctl enable watchdog

echo
echo "Done! Watchdog should now be enabled."
echo

Nothing too fancy, but it should cut down much of the original post to simply stepping through each part real quick and getting the job done fast. I'd like to say it should work on any newer installation of Raspbian, where it uses the new watchdog drivers (or whatever they changed), assuming they don't go changing watchdog on us again.

[PetrSvetr]

Also because of your sceptical comments here, I started to investigate the possibility to add external watchdog to the Raspberry pi.
Thanks to an excellent support from PhilE I thing we have succeeded.
You can see our findings here: external watchdog on GPIO output

[paulv]

This post was made irrelevant with updates.

[paulv]

This post was made irrelevant by updates

[no3rpi]

Thank you for update.

p.s.
Now the first message in the topic it is 99% confusing for the first time reader ( and not only them ) with old useless info still tagged with [EDITED] and [UPDATE] instead of removed from message.
Users will still think they need watchdog package installed and it is not true.

Code: Select all

Nov  2 16:34:10 rpi3 systemd[1]: Starting LSB: Start watchdog keepalive daemon...
Nov  2 16:34:10 rpi3 kernel: [    0.806283] bcm2835-wdt 3f100000.watchdog: Broadcom BCM2835 watchdog timer
Nov  2 16:34:11 rpi3 systemd[1]: Started LSB: Start watchdog keepalive daemon.

[paulv]

You are probably right, but then the whole post has to be re-written.
This will be a big PITA with the poor editing features, so I'm not going to do that.

I'll leave it for somebody else to write a post titled "This is the correct way to use the Watchdog"

Have fun!

Paul

[PlanB]

On my Jessie lite it's not /etc/systemd/systemd.conf but /etc/systemd/system.conf Should the watchdog work if I use 'poweroff' from the cmd line?

[PiGeon]

I have just registered to report my todays achievements with watchdog.

I have a Raspbian Lite installed and updated to following version

Code: Select all

Linux rpi3b 4.9.59-v7+ #1047 SMP Sun Oct 29 12:19:23 GMT 2017 armv7l GNU/Linux

I didn't need to enable the watchdog module nor put the "watchdog=on" to /boot/config.txt

the only thing I had to do was add the watchdog related lines to /etc/systemd/system.conf file

Code: Select all

echo "RuntimeWatchdogSec=10" | sudo tee -a /etc/systemd/system.conf
echo "ShutdownWatchdogSec=10min" | sudo tee -a /etc/systemd/system.conf
cat /etc/systemd/system.conf | grep "Watchdog"

then I could see these lines in system log after reboot (the first line was there since Rasbian setup)

Code: Select all

[email protected]:~ $ journalctl -b | grep dog
Nov 03 18:16:43 rpi3b kernel: bcm2835-wdt 3f100000.watchdog: Broadcom BCM2835 watchdog timer
Nov 03 18:16:43 rpi3b systemd[1]: Hardware watchdog 'Broadcom BCM2835 Watchdog timer', version 0
Nov 03 18:16:43 rpi3b systemd[1]: Set hardware watchdog to 10s.

and after firing the fork bomb scrript the RPI succesfully rebooted after arround 15 seconds

have fun

PiGeon

[sparkie777]

@paulv:
thank you very much sharing your watchdog knowhow with us.

I use the hardware watchdog throughout various raspbian releases in some of my applications. I now did some testing on raspbian/ stretch/ RPi3:

My experiences:

- you need not to inititalize/ load any drivers, dtoverlays etc.
- not any kind of special configuration is required
- don't install the watchdog package
- don't fiddle with systemd watchdog settings
- nonetheless the device "/dev/watchdog" is simply there
- you can write "." to the device "/dev/watchdog" at any time to start the functionality
- this triggers the hardware watchdog which expires in 15s per default (I don't know how to change this interval)
- i.e. the machine reboots unless you write "." again to the device "/dev/watchdog" prior to the 15s expiry period
- to stop the watchdog you simply write "V" to the device "/dev/watchdog"

most of my applications are running some sort of dispatch loop. So the only thing I have to worry about is
to rewrite the "." just in time.

trivial shell script to explain the watchdog behavior

Code: Select all

while :
do
    date
    echo . > /dev/watchdog
    sleep 14
done

if the loop fails to write the '.' just in time (i.e. prior to the 15s expiry period) the machine reboots instantly

[sparkie777]

I don't know how to change this interval

in the meantime I experimented a little how to configure the watchdog timeout (on raspbian/stretch 4.9.59+)

you can configure the timeout on the kernel cmdline like this (example for 10s timeout):

/boot/cmdline.txt:

Code: Select all

... bcm2835_wdt.heartbeat=10

be aware that a val of 15 is max due to hardware/firmware restrictions

[VascoFerraz]

The following solution (thanks Paul) works on the RPi1 model B (BCM2835)?

1)
In /boot/config.txt add/change:
watchdog=on

2.1)
In /etc/systemd/systemd.conf, change #RuntimeWatchdogSec= to:
RuntimeWatchdogSec=10s

2.2)
In /etc/systemd/systemd.conf, change #ShutdownWatchdogSec= to:
ShutdownWatchdogSec=10min

3)
sudo reboot

Thanks.

[sstream]

VascoFerraz » Tue Jan 02, 2018 6:03 am

The following solution (thanks Paul) works on the RPi1 model B (BCM2835)?

You don' t need 1).

Simply,

1.1)
In /etc/systemd/systemd.conf, change #RuntimeWatchdogSec= to:
RuntimeWatchdogSec=10s

1.2)
In /etc/systemd/systemd.conf, change #ShutdownWatchdogSec= to:
ShutdownWatchdogSec=10min

2)
sudo reboot

That's it.

[VascoFerraz]

Thanks. It is working

[Michaeli]

Hey Paul. I just followed your guide in an attempt to get this watch dog timer working. I think that it is working, but I am not really sure that it is until it hangs again. I did try your Fork bomb, and that made it puke and reboot, but that not real world enough. My question is this... My rpi runs as a mqtt server for a bunch of Sonoff wi-fi switches I have throughout my home. sometimes, it just stops responding. Will your setup here reboot the pi if it loses network connection? or just a lock up?

thanks in advance

[paulv]

The systemd version as I have described it will only cover a hung system or unresponsive app. There are a couple more things it can do, but not watch your internet connection, unless you write the code in your app to do that.

The "traditional" watchdog package has the ability to "watch" over many different parameters, the network being one of them.

I suggest you look at the man pages of the watchdog package or do a Google search.

Good hunting!

[Faruk]

Hello guys I want say different things. I did what paulv said. watchdog is works good. Additionally I used software watchdog timer in my raspberry pi. Both watchdog timer works good alone. When I use them together, If raspberry catch any problem, watchdog reboot to raspberry 3 or 4 times. At first, I used same name both of them. I thought the problem was here. I Changed same name but raspberry pi go on make reboot. I will try way of brunohpg and I will announce result. Thanks this sharing

[Barabba]

Hi everyone, I've some questions here, I hope someone can answer.

Ok I active watchdog, if nothing is "polling" it before 10 seconds (for example) it wil restart the Raspberry, once is active who does this? Supposing nothing is doing it the Raspberry will kkep rebooting forever. Now.. it's clear something is doing it, and if it is doing how can I interrupt it to reboot?
You may reply me, there are different ways to tell the daemon to reboot the RB, as reported:
May 20 09:33:02 raspi-server watchdog[707]: file: no file to check
May 20 09:33:02 raspi-server watchdog[707]: pidfile: no server process to check
May 20 09:33:02 raspi-server watchdog[707]: interface: no interface to check
May 20 09:33:02 raspi-server watchdog[707]: temperature: no sensors to check
May 20 09:33:02 raspi-server watchdog[707]: test=none(0) repair=none(0) alive=/dev/watchdog heartbeat=none to=root no_act=no force=no
May 20 09:33:02 raspi-server watchdog[707]: watchdog now set to 15 seconds
May 20 09:33:02 raspi-server watchdog[707]: hardware watchdog identity: Broadcom BCM2835 Watchdog timer
May 20 09:33:02 raspi-server systemd[1]: Started watchdog daemon.

How do I deal with them?
1) How can I add a file to check? What is the check consisting? Only if the file exist? Or it should have new time/date then the last check (which happened only 10 seconds before)?
2) what is a pidfile? HOw can I use this?
3) If a interface is down in the exact moment when the daemon reloads the 10 seconds it will not load so the system reboot?
5) test=none(0): if teast is none it meand that when the kernel complety hangs (or the watchdog daemon) it restarts RB, but if I have for example TCP stack hanged but kernel is working it wil do nothing, and the RB will be unreachable, almost useless.
repair=none(0): what it can repair? Reatart a service? How to?
alive=/dev/watchdog
heartbeat=none: why none here? Who is realoading every 10 second watchdog?
to=root no_act=no force=n

It would be also good to try to unmount devices before watchdog resets, of course if daemon is working..