crozone
Posts: 11
Joined: Mon Jul 30, 2012 11:27 am

Kernel paging error during heavy network and disk load

Mon Jul 30, 2012 11:57 am

Hi, I'm running into some issues while using rtorrent on my raspberrypi.

I have an external, self powered 300gb NTFS drive mounted using ntfs-3g to /mnt/sda1. I am also running rtorrent in a background screen session using the recommended init.d startup script. rtorrent is configured to use ~/rtorrent/watch and ~/rtorrent/session as it's watch and session directories (which are located on the sd card), and /mnt/sda1/rtorrent/downloads as the primary download directory.

After 30 mins or so of downloading a single torrent at ~1MB/s,
the Raspi crashes and I get the message:

Unable to handle kernel paging request at virtual address 8e78b8a2
pgd =cb9e4000
*pgd=00000000
Internal error: Oops: 805 [#1] PREEMPT
Entering kdb (current0xca4ec520, pid 1313 Oops: (null) due to oops @ 0xc024d3d4

along with a whole bunch of other data on registers etc.

I am using a 4gb class 4 SD card with the paging file extended to fill. It does not matter whether I use the 192/64 or 224/32 memory split, it only buys a little more time before the error occurs. CPU usage stays around 80-100% and free memory jumps around 10-50mb the entire time rtorrent is running. ntfs-3g and rtorrent are using all the cpu and memory.

From the appearance of the OK status led, the SD card is getting accessed in short pulses and fairly frequently, but its not getting "hammered" in any way to suggest heavy swap use. Swap Free stays around 92MB, so the RaspberryPi does not appear to be running out of swap.

I'm running the Raspian (Debian wheezy 2012-07-15) image and have my raspberrypi powered using the apple ipad charger, and the voltages seem to stay at 4.9V stable, so I don't think it's a power issue. either. I'm running stock clocks and the cpu and usb chips are warm but not overly hot to the touch.

At this stage the only thing I can think of is that swap read/write requests are somehow timing out (is this even possible?), but other than that I have no idea. The terminal (local, not ssh) does seem very sluggish at times and input does get delayed a bit. I've googled for similar problems but come up with very little. Are there any log files I could check to try and find out more about why this occurred? Any help would be greatly appreciated.

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5097
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: Kernel paging error during heavy network and disk load

Mon Jul 30, 2012 12:01 pm

Do you back a backtrace? (a list of function names where the crash occurred). Can you capture that (a photo of screen is one way).

crozone
Posts: 11
Joined: Mon Jul 30, 2012 11:27 am

Re: Kernel paging error during heavy network and disk load

Mon Jul 30, 2012 2:21 pm

Currently I can only post an image of the screen that is given when rtorrent crashes, (the rest of the error stated above), and the amount of memory/cpu used at the time of the crash (via an ssh top session), although that probably won't be much help. I'm trying to compile rtorrent with debug symbols so that I can try to reproduce the error while using gdb but I'm having some real trouble getting it to build.

I did notice that it was using over 150MB of virtual mem the last time it crashed, although there was 70MB free swap left...not sure if that helps at all. Sometimes I get another error where screen reports that it can't find a process to attach to, and rtorrent exits gracefully.

However, when the kernel paging crash does occur, it's exactly the same every time,

fridgemagnet
Posts: 12
Joined: Fri Jun 01, 2012 3:21 pm

Re: Kernel paging error during heavy network and disk load

Thu Aug 02, 2012 10:00 pm

IMG_20120802_225134.jpg
IMG_20120802_225134.jpg (51.89 KiB) Viewed 6121 times
I may be seeing the same thing. Been playing with the MPD music daemon. If I request it build the database, from files over an NFS (wireless) link, the CPU gets busy for a bit, plenty of network traffic and presumably periodic SD accesses as it writes the database then falls over with a paging fault. See attached for a nice screenshot.
This is with the Raspbian OS build.

javatmn
Posts: 6
Joined: Fri Aug 03, 2012 8:14 am

Re: Kernel paging error during heavy network and disk load

Mon Aug 06, 2012 5:18 am

I may have the same crash issue. I am using ml-donkey to download large files, after downloading about 500M file content, then the PI crashes. It does happened 3 times till now. I suspect the kernel might be not stable enough. Do you have same issue with old debian squeeze version?

Goldy
Posts: 3
Joined: Mon Jun 25, 2012 4:16 pm

Re: Kernel paging error during heavy network and disk load

Thu Aug 09, 2012 1:41 pm

I've got the same problem.

IIn the past, i was fixed this problem by adding vm.min_free_kbytes = 8192 in /etc/sysctl.conf.

The problem was gone, but I don't know why I did that, I updated the kernel with rpi-update, and now it crashes when the load is increased to 5 or more.

Is there any way to backtrace the crash when the raspberry is used without screen?

ubergrog
Posts: 6
Joined: Sun Aug 12, 2012 9:00 pm

Re: Kernel paging error during heavy network and disk load

Sun Aug 12, 2012 9:03 pm

I am experiencing this exact same issue. I'm running transmission-daemon & sharing a drive via samba & have experienced this error several times now. I'm using the 2012-07-15 image which already has the "vm.min_free_kbytes = 8192" in /etc/sysctl.conf.

Any help would be greatly appreciated. TX

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5097
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: Kernel paging error during heavy network and disk load

Sun Aug 12, 2012 10:43 pm

@ubergrog
I'd try doubling the 8192.

ubergrog
Posts: 6
Joined: Sun Aug 12, 2012 9:00 pm

Re: Kernel paging error during heavy network and disk load

Mon Aug 13, 2012 11:09 pm

Running with "vm.min_free_kbytes = 16384" for an uptime of over 13 hours so far, no issues.

Is there anything I can be tracking this? What information might be useful leading up to this issue if it happens again?

Thanks dom.

secretagent
Posts: 36
Joined: Wed Mar 07, 2012 10:09 am

Re: Kernel paging error during heavy network and disk load

Tue Aug 14, 2012 10:04 am

I can post instructions on how to run Kdump on the Pi, which would let users experiencing kernel crashes, such as this, to send complete memory dumps of the crashed kernel for developers to look at. However there is one caveat: this requires a bit of RAM to be reserved for the crashkernel, so that RAM will not be available during normal operation. I can check to see how small can that be made while still working. Also, for this to be useful the original kernel will need to be compiled with debugging symbols and the vmlinux file will need to be made available, so that the memory dumps can actually be used. Dom, any thoughts on this? Or has this been considered already?

fridgemagnet
Posts: 12
Joined: Fri Jun 01, 2012 3:21 pm

Re: Kernel paging error during heavy network and disk load

Tue Aug 14, 2012 7:11 pm

Running with vm.min_free_kbytes = 16384 has made no difference to the problem I'm seeing. I have another SD card with the old Debian Squeeze image on it but I'm still getting things installed (ie. the wifi adaptor) on it to be able to report back on that.

fridgemagnet
Posts: 12
Joined: Fri Jun 01, 2012 3:21 pm

Re: Kernel paging error during heavy network and disk load

Thu Aug 16, 2012 6:23 pm

Quick update - I can confirm a similar thing happens with the Debian/Squeeze build however I'm now fairly sure my problem is related to power. Up till now I have been using a USB WiFi dongle plugged straight into the Pi. Just tried it with the onboard Ethernet and I've not seen the crash, it's been running a good deal longer but I am getting repeated characters from the keyboard which I've read elsewhere is a sign of power issues. Odd because I'm using a Nokia PSU rated at 1.2A but looks like I may need to investigate powered hub and maybe another PSU.

elatllat
Posts: 1338
Joined: Sat Dec 17, 2011 5:05 pm

Re: Kernel paging error during heavy network and disk load

Thu Aug 16, 2012 7:00 pm

fridgemagnet wrote:...
Repeated characters is a software bug not a power issue.

related: http://www.raspberrypi.org/phpBB3/viewt ... 24&t=14394
4k or USB3 : hardkernel.com

FAQ : raspberrypi.stackexchange.com

https://google.com/search?q=site%3Awww.raspberrypi.org+search+first

https://www.raspberrypi.org/forums/search.php?search_id=unanswered

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5097
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: Kernel paging error during heavy network and disk load

Thu Aug 16, 2012 7:36 pm

elatllat wrote: Repeated characters is a software bug not a power issue.
No, it can be either. Repeating characters are particularly common when plugging in >100mA USB devices, which causes the keyboard to get significantly less than 5V.

uberscubajim
Posts: 157
Joined: Thu Jun 21, 2012 12:22 pm
Location: Ipswich, Suffolk. UK.

Re: Kernel paging error during heavy network and disk load

Tue Aug 21, 2012 11:38 am

I too have been running into this problem (see posting at http://www.raspberrypi.org/phpBB3/viewt ... 71&t=15207)

ubergrog
Posts: 6
Joined: Sun Aug 12, 2012 9:00 pm

Re: Kernel paging error during heavy network and disk load

Tue Aug 21, 2012 12:56 pm

FWIW. I have had one freeze during the previous week, along with the running transmission-daemon I had two terminals open, both copying files from one usb drive to another when it happened. I also share the drives via samba but wasn't accessing them that way at the time.

To clarify I am powering the Pi via a 1A usb adapter & the two usb-powered drives I am currently using are on a self-powered usb hub, so I wouldn't think it's a power issue. I did try doubling the vm.min_free_kbytes value again to 32768 with little effect, I suspect it's a case of diminishing return. I've learned to just limit the number of concurrent tasks.

crozone
Posts: 11
Joined: Mon Jul 30, 2012 11:27 am

Re: Kernel paging error during heavy network and disk load

Tue Aug 21, 2012 2:53 pm

This issue appears to be caused by the USB drivers having trouble with the built in hub, there's a major discussion about it over here: http://www.raspberrypi.org/phpBB3/viewt ... p?p=153956

I eventually fixed the issue by switching over to Chris Boot's custom armel kernel, which can be found here http://www.bootc.net/projects/raspberry-pi-kernel/

I've been running transmission-daemon and an apache server without any issues. However, it seems to break omxplayer, probably due to the fact that it's only soft float, rather than the official armhf hard float build.

Goldy
Posts: 3
Joined: Mon Jun 25, 2012 4:16 pm

Re: Kernel paging error during heavy network and disk load

Tue Aug 21, 2012 3:20 pm

I stopped to use the usb hdd and the raspberry is now very stable. The problem seems to be coming from the usb driver.

uberscubajim
Posts: 157
Joined: Thu Jun 21, 2012 12:22 pm
Location: Ipswich, Suffolk. UK.

Re: Kernel paging error during heavy network and disk load

Tue Aug 21, 2012 3:46 pm

There do seem to be some common themes occurring.

Firstly, a lot of us are running bittorrent clients when the machine fails and secondly we seem to be using a Wi-Fi based USB dongle.

There has been a lot of talk about issues with the USB interface. One of the things that occurs when downloading using a bittorrent client via Wi-Fi is that you can receive a lot of data over the USB with no real flow control. Increasing the minfree gives the kernel some extra space to work with but at times it isn't enough.

Or perhaps there's just an issue with the USB drivers that has yet to be fixed (or worked around).....

elatllat
Posts: 1338
Joined: Sat Dec 17, 2011 5:05 pm

Re: Kernel paging error during heavy network and disk load

Tue Aug 21, 2012 6:02 pm

dom wrote:
elatllat wrote: Repeated characters is a software bug not a power issue.
No, it can be either. Repeating characters are particularly common when plugging in >100mA USB devices, which causes the keyboard to get significantly less than 5V.
All I know is I get repeated characters using either of my 1A power supplies and no USB plugged in (keyboard over ethernet) and no other power related problems. The problem is consistent regardless of CUP/GPU/NIC usage.
4k or USB3 : hardkernel.com

FAQ : raspberrypi.stackexchange.com

https://google.com/search?q=site%3Awww.raspberrypi.org+search+first

https://www.raspberrypi.org/forums/search.php?search_id=unanswered

ubergrog
Posts: 6
Joined: Sun Aug 12, 2012 9:00 pm

Re: Kernel paging error during heavy network and disk load

Wed Aug 22, 2012 10:27 am

ubergrog wrote:To clarify I am powering the Pi via a 1A usb adapter & the two usb-powered drives I am currently using are on a self-powered usb hub, so I wouldn't think it's a power issue. I did try doubling the vm.min_free_kbytes value again to 32768 with little effect, I suspect it's a case of diminishing return. I've learned to just limit the number of concurrent tasks.
I've updated to the latest firmware using "apt-get dist-upgrade && rpi-update" so am now running the 3.2.27 kernel & have reverted back to "vm.min_free_kbytes = 8192". So far so good, usb hard drive access is much more stable.

uberscubajim
Posts: 157
Joined: Thu Jun 21, 2012 12:22 pm
Location: Ipswich, Suffolk. UK.

Re: Kernel paging error during heavy network and disk load

Wed Aug 22, 2012 10:28 am

I can confirm that a quick kernel upgrade has indeed solved my problem too.

Looking good guys!

fridgemagnet
Posts: 12
Joined: Fri Jun 01, 2012 3:21 pm

Re: Kernel paging error during heavy network and disk load

Mon Sep 17, 2012 7:11 pm

Just to really add/confirm what is probably already known - swapping over power supplies, hubs etc. did not yield anything positive on the problem I was seeing. However I finally got round last week to upgrading the kernel (via rpi-update) and things are much better now. The original fault I reported has gone as have the other oddities relating to network drop outs and usb stability.

Cheers all.

Return to “Troubleshooting”

Who is online

Users browsing this forum: Bing [Bot], riker1 and 48 guests