Abe_Stew
Posts: 18
Joined: Mon Mar 16, 2015 9:48 pm

What's killing my Pi2?

Sat Apr 09, 2016 6:35 pm

I have four Raspberry Pi 2s, three of which run constantly with no issues. However, the fourth, that use to run fine for around six months, has developed an issue where it crashes after seemingly random intervals. Over time I have come to notice that one thing that consistently leads to a crash is when Plex or Subsonic do a scan of the media folders on a NAS that are mounted via webDAV. Subsonic's logs though don't really provide much useful information. While Plex has more detailed logs I couldn't really see any warning messages or anything irregular before each crash. I thought some of the audio files with non-Latin characters in the tags might have been a culprit but it doesn't look like it. In terms of the main logs I am not sure which are the best to look through. Any ideas of what logs I should be examing and where they are stored?

Edit: Just noticed while trying to search and read other logs that the media scan (got it running again to recreate a crash) also leads to memory issues. Each time I hit tab to auto-complete I get the following error:
bash: cannot create temp file for here-document: No space left on device
Edit 2: Been through the logs in /var/logs but apart from start-up messages couldn't see any discernable patterns in them.

beta-tester
Posts: 1373
Joined: Fri Jan 04, 2013 1:57 pm
Location: de_DE

Re: What's killing my Pi2?

Sat Apr 09, 2016 7:45 pm

Abe_Stew wrote:Edit: Just noticed while trying to search and read other logs that the media scan (got it running again to recreate a crash) also leads to memory issues. Each time I hit tab to auto-complete I get the following error:
bash: cannot create temp file for here-document: No space left on device
what do you get when you show the available diskspace(s)? e.g. with

Code: Select all

df -h
maybe you have something activated that fills up all the available free diskspace with e.g. log files and therefore you run out of diskspace.

or does the drive you are scanning has symbolic/hard links that are building an endless loop?
so the recursion of the scan process tries do scan deeper and deeper by folowing the endles loop until the recursion eats all available temporary memory...
{ I only give negative feedback }
RPi B (256MB), B (512MB), B+, ZeroW; 2B; 3B, 3B+; 4B (4GB)

Abe_Stew
Posts: 18
Joined: Mon Mar 16, 2015 9:48 pm

Re: What's killing my Pi2?

Sat Apr 09, 2016 8:02 pm

Thanks for the reply. Running df -h I get:

Code: Select all

pi@Phobetor ~ $ df -h
Filesystem              Size  Used Avail Use% Mounted on
/dev/root                28G   25G  1.4G  95% /
devtmpfs                428M     0  428M   0% /dev
tmpfs                    87M  244K   87M   1% /run
tmpfs                   5.0M     0  5.0M   0% /run/lock
tmpfs                   173M  4.0K  173M   1% /run/shm
/dev/mmcblk0p5           60M   20M   40M  33% /boot
tmpfs                   173M   68K  173M   1% /tmp
//192.168.0.3/Audio     7.3T  3.6T  3.7T  50% /home/pi/Audio
//192.168.0.3/Video     7.3T  3.6T  3.7T  50% /home/pi/Video
Currently not getting the bash error at the moment but will run the command again if and when I do.


As far as symbolic links go, as far as I am aware, there isn't any. Kodi scanning the same folders manages it fine. Earlier this week I even cleared out my libraries in Kodi and rescanned everything again to check - with no issues. It's only Subsonic and Plex that I have managed to recreate a crash anywhere from 10 minutes to an hour from starting a scan.

fruit-uk
Posts: 609
Joined: Wed Aug 06, 2014 4:19 pm
Location: Suffolk, UK

Re: What's killing my Pi2?

Sat Apr 09, 2016 8:13 pm

Use% 95% is getting into dangerous territory!

Abe_Stew
Posts: 18
Joined: Mon Mar 16, 2015 9:48 pm

Re: What's killing my Pi2?

Sat Apr 09, 2016 8:17 pm

Aye, I thought that. No idea what's taking up all that space though as apart from Kodi, Plex, etc all the actual files are on the NAS. Apt-get clean got it down to 92%.

Edit: Just ran df -h on another pi that I use for bunch of programs that utilize mySQL databases as well as daily, weekly, and monthly backups of them and it's only sitting at 36% use so something has definitely gone awry somewhere.

fruit-uk
Posts: 609
Joined: Wed Aug 06, 2014 4:19 pm
Location: Suffolk, UK

Re: What's killing my Pi2?

Sat Apr 09, 2016 8:26 pm

No idea on those apps. I have a Pi2 here, no GUI but several hundred maybe thousand logs, /root is 9Gb
Once you start running short of free space things may go wrong - and although 10% free may sound quite a lot, it may all be in very small slots all over the FS.

Abe_Stew
Posts: 18
Joined: Mon Mar 16, 2015 9:48 pm

Re: What's killing my Pi2?

Sat Apr 09, 2016 8:43 pm

Currently running du -h / | tee ~foldersize.txt to try locate the issue.

Edit: It's obviously running too fast to read things in full at the moment but I did briefly spy a tmp directory (didn't have time to read the full directory tree) sitting at 11GB so that'll be getting cleared once du has finished running.

Edit 2: It was the Kodi tmp folder the pi user directory. It was stuff fule of mp3s and video files. Looks like it wasn't clearing out the files it was pulling from the NAS. Spotted one film I know I haven't watched in at least a year. That's 50% showing as free now. Will run the scans again and see if the crash reoccurs.

Edit 3: More pruning has got it down to 40% and dropping - currently clearing out another temp folder with a tone of small files which is taking ages to finish. Got the media scan running again and going to leave it on overnight and will report back in the morning as to whether it has crashed or not.

Abe_Stew
Posts: 18
Joined: Mon Mar 16, 2015 9:48 pm

Re: What's killing my Pi2?

Sun Apr 10, 2016 4:35 am

Unfortunately it crashed again overnight. Bit stumped for what to do now to try and fix it?

Currently, running a forced reinstall of everything incase a package is not working correctly. If it crashes again after that will swap the micro-SD card into another pi to see if the hardware is an issue. Also installing plex on another pi and going to run a scan on the media folders from it.

fruit-uk
Posts: 609
Joined: Wed Aug 06, 2014 4:19 pm
Location: Suffolk, UK

Re: What's killing my Pi2?

Sun Apr 10, 2016 5:27 am

It sounds as though you may have a spare PI to run kodi etc.
Do a fresh install of jessie on the first, then add in the various bits to bring it to the same state as before but do it bit by bit if possible leaving plenty of time between.
May be a while before you see anything fail if at all ;)

Abe_Stew
Posts: 18
Joined: Mon Mar 16, 2015 9:48 pm

Re: What's killing my Pi2?

Sun Apr 10, 2016 1:54 pm

Sadly not a spare, just one that I can re-purpose temporarily.

It's sadly looking like some key files are corrupted somewhere. It's crashed twice again with card in different pi. Other pi scanning the folders with Plex is still working away no problem, and to further test I have a third pi scanning the folders with Subsonic over wifi - again with no problems occuring so far.

What I am doing now is installing everything that was on the faulty one onto another sd card. However, for some reason I can't get plexmediaserver to install on it:
E: Unable to locate package plexmediaserver
After comparing sources.list and the items in sources.list.d folder it seems I installed added a bunch of repositories that required public keys on my other pi's. After a lot of faffing trying to find what I needed to wget and apt-key finally got plexmediaserver installed.

No idea why as it installed fine on the other pi only a few hours ago.

Return to “Troubleshooting”