Why is my RaspBian Pi crashing?

[Bosse_B]

I have a Pi1B running RaspBian which I use as a utility Pi for:
- PPTP VPN server
- OpenVPN server
- Printer server using CUPS for my HP Color Laserjet CP1215
- Surveillance camera server

I have used this Pi for more than a year but lately (a few months) it has started to crash at irregular intervals.
This is of course not good since it is the gateway via VPN into my home network when I am away...
Right now it is in this crashed state when I arrived back home after a week and I cannot connect to it via PuTTY so I have to power cycle the Pi.

QUESTIONS:
1) Is there some log or similar where I can find out what brought it down?

2) Can this log be read off of the SD-card if I move it to my Win7 PC?
(This is assuming that any such log gets overwritten on startup or similar, in which case I don't want to start the Pi just yet).

3) Is there some function in RaspBian, which can detect a crash and restart it so it does not go off-line until a manual intervention is done?

I will wait for some time hoping for replies before I power cycle the unit....

[DougieLawson]

1. Yes there lots of logs in /var/log
2. No, the logs are on a ext4 filesystem so you need a linux system to read them (unless you want to install some funky ext4 driver software on Windows)
3. There is a software watchdog function but that doesn't help for a hard crash. Doing some hardware monitoring with an Arduino that triggers when it loses a "heartbeat" signal from the RPi is probably a better option.

One thing that killed one of my RPis was a cruddy power supply that over time (running 24*7) stopped providing the right voltage and current to keep my RPi alive. The other thing that takes my RPis off the network is WiFi dongles that intermittently disconnect so I have a simple cron script that pings my router every six minutes and restarts the interface.

My B+ running my public webserver and OpenVPN is wired to my router.

My IPv6 tunnel is now running on my Technicolor TG582n router not on any of my Linux systems for robustness.

[Bosse_B]

First of all thanks for your reply, I expected the forum to send me a notification about it but I have not received any so that is why I have been late in replying back...

In the end I power cycled the Pi anyway and had a look at the /var/log content. Turns out that there seems to be one logfile lasting a day or so and tghen a number of syslog.X.gz files containing compressed versions of previous syslogs.

I scanned these and found out that on the day I had left home the logs changed structure.

Before that time there was a certain hourly activity as follows:

Code: Select all

Jun  4 12:15:01 raspbian-pi /USR/SBIN/CRON[15203]: (root) CMD ((date; ls /var/www/webcam/*snapshot.jpg | head --lines=-20 | xargs rm -v) >> /tmp/images_deleted 2>&1)
Jun  4 12:17:01 raspbian-pi /USR/SBIN/CRON[15212]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jun  4 12:39:01 raspbian-pi /USR/SBIN/CRON[15225]: (root) CMD (  [ -x /usr/lib/php5/maxlifetime ] && [ -x /usr/lib/php5/sessionclean ] && [ -d /var/lib/php5 ] && /usr/lib/php5/sessionclean /var/lib/php5 $(/usr/lib/php5/maxlifetime))
Jun  4 13:09:01 raspbian-pi /USR/SBIN/CRON[15242]: (root) CMD (  [ -x /usr/lib/php5/maxlifetime ] && [ -x /usr/lib/php5/sessionclean ] && [ -d /var/lib/php5 ] && /usr/lib/php5/sessionclean /var/lib/php5 $(/usr/lib/php5/maxlifetime))

One of these is from my webcam installation and is repeating every hour throughout a normal syslog.

But then something happened:

Code: Select all

Jun  5 00:15:01 raspbian-pi /USR/SBIN/CRON[17086]: (root) CMD ((date; ls /var/www/webcam/*snapshot.jpg | head --lines=-20 | xargs rm -v) >> /tmp/images_deleted 2>&1)
Jun  5 00:17:01 raspbian-pi /USR/SBIN/CRON[17095]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Jun  5 00:39:01 raspbian-pi /USR/SBIN/CRON[17107]: (root) CMD (  [ -x /usr/lib/php5/maxlifetime ] && [ -x /usr/lib/php5/sessionclean ] && [ -d /var/lib/php5 ] && /usr/lib/php5/sessionclean /var/lib/php5 $(/usr/lib/php5/maxlifetime))
Jun  5 00:54:48 raspbian-pi dhclient: DHCPREQUEST on eth0 to 192.168.0.129 port 67
Jun  5 00:54:48 raspbian-pi dhclient: send_packet: Network is unreachable
Jun  5 00:54:48 raspbian-pi dhclient: send_packet: please consult README file regarding broadcast address.
Jun  5 00:54:52 raspbian-pi dhclient: DHCPREQUEST on eth0 to 192.168.0.129 port 67
Jun  5 00:54:52 raspbian-pi dhclient: send_packet: Network is unreachable
Jun  5 00:54:52 raspbian-pi dhclient: send_packet: please consult README file regarding broadcast address.
Jun  5 00:55:03 raspbian-pi dhclient: DHCPREQUEST on eth0 to 192.168.0.129 port 67
Jun  5 00:55:03 raspbian-pi dhclient: send_packet: Network is unreachable

This repeats ad infinum, every 1 hour there is an extra line like this:

Code: Select all

Jun  5 01:14:57 raspbian-pi dhclient: DHCPREQUEST on eth0 to 192.168.0.129 port 67
Jun  5 01:14:57 raspbian-pi dhclient: send_packet: Network is unreachable
Jun  5 01:14:57 raspbian-pi dhclient: send_packet: please consult README file regarding broadcast address.
Jun  5 01:15:01 raspbian-pi /USR/SBIN/CRON[17133]: (root) CMD ((date; ls /var/www/webcam/*snapshot.jpg | head --lines=-20 | xargs rm -v) >> /tmp/images_deleted 2>&1)
Jun  5 01:15:15 raspbian-pi dhclient: DHCPREQUEST on eth0 to 192.168.0.129 port 67
Jun  5 01:15:15 raspbian-pi dhclient: send_packet: Network is unreachable

So the webcam handler continues running but it seems like the network has been lost!
And this brings down all other connectivity as well of course.

Does this shed some extra light on where I should go looking for a reason for the failure????

[mikronauts]

I have experienced a similar issue when logging data to a remote site - the network connection would be lost.

It could be re-acquired by rebooting.

A very inelegant solution is to have a monitoring script check for network connection every minute, and reboot if connection is lost.

[DougieLawson]

A very inelegant solution is to have a monitoring script check for network connection every minute, and reboot if connection is lost.

Why do you need a reboot. An ifdown --force eth0 && ifup --force eth0 should be good enough.