Roaders
Posts: 13
Joined: Sun Mar 20, 2016 8:44 am

How to diagnose Pi becoming unresponsive after several days

Tue Jun 14, 2016 7:39 pm

Hi All

I have a headless Raspberry Pi 3 that I can't connect a monitor or keyboard to. After a few days it becomes unresponsive and I can't SSH into it. The only thing that I can do is turn it off and back on again but when it starts up there is no way of telling what caused it to freeze.

What can I do to get some idea of which process is causing this? Is there a way that I can log the output of top every 10 minutes or so so that when I reboot it I can see what was happening before I had to reboot.

Previously I had a Pi that became unresponsive as it was being bombarded with requests from the internet once the ip address had been found. I have now turned off the port forwarding and that sorted the issue. No ports are forwarded to the Pi that is currently being unresponsive.

I hope you can help,
Thanks

epoch1970
Posts: 5479
Joined: Thu May 05, 2016 9:33 am
Location: Paris, France

Re: How to diagnose Pi becoming unresponsive after several d

Tue Jun 14, 2016 8:26 pm

Can't you read /var/log/syslog on the Pi? (or perhaps the previous log as well: /var/log/syslog.1)
I believe you should be able to spot near the end of the file the moment you trigger reboot, and see if something of interest happens before.
Over the Lan you could also send the system log to another machine running rsyslog.

You could also use the serial console, or serial console over bluetooth (not ideal), if the system is not completely wedged.
"S'il n'y a pas de solution, c'est qu'il n'y a pas de problème." Les Shadoks, J. Rouxel

Roaders
Posts: 13
Joined: Sun Mar 20, 2016 8:44 am

Re: How to diagnose Pi becoming unresponsive after several d

Tue Jun 14, 2016 8:35 pm

Thanks for the suggestion. On looking at syslog.1 I can't see anything untoward. There seems to just be a series of hourly cron jobs logged before the logs when I rebooted it.

epoch1970
Posts: 5479
Joined: Thu May 05, 2016 9:33 am
Location: Paris, France

Re: How to diagnose Pi becoming unresponsive after several d

Tue Jun 14, 2016 8:53 pm

Hmm, only local activity? Network dies maybe?
Console access might be something to consider, then.
Or have an iptable rule log a line when the machine is hit on some port/protocol, and run a loop on another machine knocking at the port every so minutes.
Or do it the other way around, with the Pi actively knocking somewhere.
(every daemon you run may alter the behaviour of the system, the best solution is really the local console.)
"S'il n'y a pas de solution, c'est qu'il n'y a pas de problème." Les Shadoks, J. Rouxel

stderr
Posts: 2178
Joined: Sat Dec 01, 2012 11:29 pm

Re: How to diagnose Pi becoming unresponsive after several d

Tue Jun 14, 2016 9:08 pm

Roaders wrote:Thanks for the suggestion. On looking at syslog.1 I can't see anything untoward. There seems to just be a series of hourly cron jobs logged before the logs when I rebooted it.
You haven't given basic information like if this is wifi or physical ethernet network. You also didn't say whether or not you could still ping the device but just couldn't ssh to it. Also, you didn't say whether you'd set up a script to loop and ping the pi every, say, 10 minutes.

Roaders
Posts: 13
Joined: Sun Mar 20, 2016 8:44 am

Re: How to diagnose Pi becoming unresponsive after several d

Fri Jun 17, 2016 6:31 pm

This is on a WiFi network.

I can't ping it when it becomes unresponsive.

The rest of the network is fine, I can still ping my other Pi (also on WiFi).

I am very new to linux so pretty green when it comes to this technology.

itarian
Posts: 6
Joined: Wed Oct 21, 2015 3:59 pm

Re: How to diagnose Pi becoming unresponsive after several d

Thu Jul 07, 2016 1:22 am

I have the same issue.

I had a B+ running flawlessly for months, until I seemed to develop a hardware issue. No problem, got a Pi2, put all the same stuff on it, had a bit of trouble working out the GPIO stuff, and now it continues to crash within 24 hours to over a week.

I did this post, but got no replies:
viewtopic.php?f=28&t=136854&p=909482#p909482

stderr
Posts: 2178
Joined: Sat Dec 01, 2012 11:29 pm

Re: How to diagnose Pi becoming unresponsive after several d

Thu Jul 07, 2016 2:24 am

itarian wrote:I did this post, but got no replies:
viewtopic.php?f=28&t=136854&p=909482#p909482
Remember that the New Messages feature that people often look at to see if there are new messages doesn't necessarily show your new message for very long and it could easily be missed. It is one of the failings of this message board software.
I had a B+ running flawlessly for months, until I seemed to develop a hardware issue. No problem, got a Pi2, put all the same stuff on it, had a bit of trouble working out the GPIO stuff, and now it continues to crash within 24 hours to over a week.
If the other one developed a hardware issue and you replaced it and you still have problems, maybe the reason for the problem is outside of the pi boards. If this is being used in a project that matters, I mean you've got a budget for parts, I would definitely have a supply of everything for swapping out. I tell people to pretend like they are Noah and get two of everything. A power supply could heat up and go sideways and even trash the pi boards themselves, who knows.

Did you set up a script to ping the pi from outside and keep track of how long it responds? If it is on wifi, the problem is often that something is going to sleep even if you've told everything you know about you don't want that. I would set up another pi and make sure it is solid and then swap it into the production unit's place.

If it still is crashing, does it crash if you aren't running the software you wrote to do your project? You basically need to cut things up to find out where the problem is to move forward.

stderr
Posts: 2178
Joined: Sat Dec 01, 2012 11:29 pm

Re: How to diagnose Pi becoming unresponsive after several d

Thu Jul 07, 2016 2:26 am

Roaders wrote:This is on a WiFi network.
I can't ping it when it becomes unresponsive.
The rest of the network is fine, I can still ping my other Pi (also on WiFi).
This kind of issue is often due to something going to sleep even if you don't want it to. The ping loop idea can keep traffic on the wifi and keep it awake even if it doesn't want to.

Return to “Troubleshooting”