Kernel Panic on concurrent network and usb storage [SOLVED]

[abishur]

On the Debian squeeze image (19-04-2012) a kernel panic happens whenever you are trying to use the network to transfer data from or two a usb attached storage device.  It also happens when streaming video from a usb storage device.  The amount of time it takes to happen is random, but for me it usually happened while trying to transfer at 100 MB (it might get through once).

I haven't been able to catch a full dump of the panic, but I do notice that there is an interrupt conflict, which makes a lot of sense.  Ethernet goes through the usb device and at first glance it looks like there's a handling error going on when using the two features of the LAN9512 chip.

Abishur said:

On the Debian squeeze image (19-04-2012) a kernel panic happens whenever you are trying to use the network to transfer data from or two a usb attached storage device.  It also happens when streaming video from a usb storage device.  The amount of time it takes to happen is random, but for me it usually happened while trying to transfer at 100 MB (it might get through once).

I haven't been able to catch a full dump of the panic, but I do notice that there is an interrupt conflict, which makes a lot of sense.  Ethernet goes through the usb device and at first glance it looks like there's a handling error going on when using the two features of the LAN9512 chip.

Interesting, Abishur

I had seen 3 posts now about kernel panics, and 2 of them is about USB and networking being done concurrently fails after some time with a kernel panic.

Somewhere there is an concurrency issue.

Does this only exist on the new Debian image, and not with the previous Debian image?

if it does not occur in the previous Debian image it could be interesting to do a diff of the usb, network, and file system subsystems and the block device driver of the Rpi source code of the 2 Debian images.

[abishur]

It also happened on the old debian image.  That's where I first observed it, but mistakenly identified it as an error caused by minidlna.

[abishur]

Okay here's a screen shot of the dump  I had to break it into 2 parts

Top (ends at backtrace)



bottom (begins at backtrace)

https://github.com/raspberrypi/firmware/issues/9

This is the type of kernel panic I got too - thanks for capturing it!

[abishur]

I recreated the panic and took two more pics (split again at backtrace) to confirm it's the same issue occurring again and again and not unrelated issues being mistaken for the same thing.

(ignore the first two lines, they have nothing to do with the issue )

Top:



Bottom:

Pretty sure this is the same issue I've been having when transferring files via network to usb on both Debian Squeeze and Arch Linux. Interstingly enough I'm not getting kernel panics when it's from one usb device to another on my pi, only through the network and usb.

I too experienced this issue. A workaround mentioned in this forum topic seems to be working for me:

while true ; do echo 3 > /proc/sys/vm/drop_caches ; sleep 1 ; done &

I've streamed a couple of hours of 1080p h264 over the network from an attached USB disk.

I just posted on issues that I don"t get any problems with git clone of a large source.
Also when I scp"d a large 640 Mb file to the pi, no problem

iciclethief said:

I too experienced this issue. A workaround mentioned in this forum topic seems to be working for me:

while true ; do echo 3 > /proc/sys/vm/drop_caches ; sleep 1 ; done &

I've streamed a couple of hours of 1080p h264 over the network from an attached USB disk.

If that solved the problem, then (as I understand it) the problem was a memory buffer cache filling up, and it should be easy enough to confirm if that is the case, by looking at the result of 'free -m' while you attempt to trigger the error (large file transfers, etc.). See also: http://www.linuxatemyram.com/play.html

I compiled a new kernel myself to include IPv6 support and I've seen this issue too.

This happened at speeds around 500 - 700 kb/sec with a usenet binary download, sometimes it took just 10 seconds, other times it took a few minutes.

while true ; do echo 3 > /proc/sys/vm/drop_caches ; sleep 1 ; done &

Will have to try this, my RasPi hangs daily and the main thing it's running is vnc.

n3tw0rk5 said:

while true ; do echo 3 > /proc/sys/vm/drop_caches ; sleep 1 ; done &

Will have to try this, my RasPi hangs daily and the main thing it"s running is vnc.

Unless you're doing (moderately heavy) concurrent network and usb traffic that could be a different issue.

I've eliminated the power/usb cable possibilty by upgrading both, no USB peripherals are attached. only thing I can think of is some sort of memory hog.

The RasPi just freezes allthough the lan light still blinks, but cant connect via console or ssh. Only thing I can do is pull the power.

Nothing of any use in dmesg or syslog/messages

Hey guys,

Same issues here.

I can only have network OR anything USB plugged in, not both, which makes it a little restrictive.

I'm getting the same kernal panic.

I'm using:

debian6-19-04-2012

16G card with file system expanded.

Typing in: "while true ; do echo 3 > /proc/sys/vm/drop_caches ; sleep 1 ; done &" just gives me a while loop outputting text across the screen till I unplug it.

Please show what the output across the screen looks like. There should not be any if the command is working properly.

If anyone else gets stuck in that kind of loop, the way to stop it is issue the "fg" command, which brings the backgrounded process into the foreground, and then you can kill it with CTRL-C.

I do not have my Pi yet, so what I am saying is just based on general Linux knowledge. When I try that line of code on my Embedded Linux PDA, I get the following, and if this is what you get, it either means that the file and possibly directory is not there for "echo" to write to, or that you do not have permission to write to it:

/bin/bash: /proc/sys/vm/drop_caches: No such file or directory

So try this to for diagnostic purposes:

ls -l /proc/sys/vm/drop_caches /proc/sys/vm/

I think it is likely a write permission thing. Might be good to also run "whoami".

[dom]

There has been a bug fix for USB kernel panics. It is now in kernel source and prebuilt in GitHub firmware tree.

aficionado said:

Typing in: "while true ; do echo 3 > /proc/sys/vm/drop_caches ; sleep 1 ; done &" just gives me a while loop outputting text across the screen till I unplug it.

Read Dom's reply about the new kernel first.

but then it's interesting to teach you some Linux things.

You can switch to tty3 by pressing control-alt-F3. Now you have an empty login screen that you can use.

As noted before, you can pull the program to the foreground by typing "fg" and then stopping it with "control-C".

Don't be bothered that what you type is interrupted by output.  What you type still goes to the program that is asking you for the next command.

I just typed:

while true ; do echo hoi; sleep 1 ; done &

and then when I start typing the "hoi" command will come through. Suppose I want to type "firefox" because I want to start my webbrowser, the screen will look like:

hoi
fhoi
ihoi
rehoi
hoi
foxhoi
hoi
hoi

As you can see, besides the stream of "hoi"'s there is just the letters "firefox" intermixed there. Thus when I press enter, the browser will start. If it is getting too messy for you, or you can't remember what you typed, press control-L Usually it will repaint the screen and put everything you typed on one line at the top of the screen....

dom said:

There has been a bug fix for USB kernel panics. It is now in kernel source and prebuilt in GitHub firmware tree.

I can confirm that this works. I rebuilt my IPv6 kernel, and it's been chugging along at usenet downloads for over 7 hours. With the old kernel it would stop within 15 minutes.

Hey everyone, thanks heaps for your help.

With the network and USB plugged in, I am unable to even log in, it just stops at the login prompt.

I did download the new kernel.img and replace the one that was on the SD card.... is that the right way to do it? (can you smell noob? ) but the same thing happened.

I have yet to try a few other things you mentioned, so I'll keep you updated

Here is what comes up after the 'panic'.

OK, got some more done

SDJF, after I ran your code:

ls -l /proc/sys/vm/drop_caches /proc/sys/vm/

This is what I got.



After "whoami" I got "pi"

The stuck loop comes up as:



Possibly a permission thing

Not that I know how to fix it..... chmod ?? drop_caches?

Thanks everyone else for your info too.... that 'fg' thing worked a treat

aficionado said:

With the network and USB plugged in, I am unable to even log in, it just stops at the login prompt.

When you get your kernel to work, note that SSH is disabled by default.

Can someone help aficionado with making sure he has the right kernel installed that has the kernel panic fix in it? Once he does that, all this command line stuff will still be interesting but not essential.

aficionado, yes, I can think of a number of ways to try to make the loop work, but you might want to test them with a simple echo outside of the loop first. Until the "echo 3 > /proc/sys/vm/drop_caches" works without an error, no need to put it into a loop.

I cannot test this stuff without a Pi, but possible solutions without the new kernel would be to give yourself write permission to the drop_caches file with:

sudo chmod +w /proc/sys/vm/drop_caches
ls -l /proc/sys/vm/drop_caches

You would want the output of the ls -l to start with -rw-rw-rw-

If you change the permissions that way, the echo ought to work, but might not because you still would not have write permission to the /proc/sys/vm directories and I am not sure it would be possible or wise to make those writable by all.

The other theoretical way would be to use the sudo command. Again, I do not have a Pi so cannot test this now. Trying on my PDA does not work as I have not been able to get a drop_caches file onto it. Anyhow, to test permissions, just try

sudo echo 3 > /proc/sys/vm/drop_caches

If you do not get an error, then try the loop:

while true ; do sudo echo 3 > /proc/sys/vm/drop_caches ; sleep 1 ; done &

or maybe

sudo while true ; do echo 3 > /proc/sys/vm/drop_caches ; sleep 1 ; done &

And post your results if it still is not working so forum members can help.

But if there is a kernel fix, getting that revised kernel working seems like a better way to go.

Lars, has ssh been disabled on all distributions or just Debian? That is not going to be fun for those of us needing ssh.

sdjf said:

Can someone help aficionado with making sure he has the right kernel installed that has the kernel panic fix in it? Once he does that, all this command line stuff will still be interesting but not essential.

I've got a working kernel at http://positron.dckd.nl/~jeroe.....kernel.img

Download that one, backup kernel.img on your RPi SD image, and place the one you downloaded in its place (so rename it to kernel.img)

....

Lars, has ssh been disabled on all distributions or just Debian? That is not going to be fun for those of us needing ssh.

SSH is just not enabled. There's a file in the boot partition which you can rename to enable SSH at boot.

Hey guys,

Before you ban me from ever posting again, I'd like to thank you all for your help.... it was great.

I seem to have found the problem..... although I thought it was impossible to get one, apparently the mini USB charger I was using was just shy of the 700mA required to power the Raspi, Ethernet and a smidge of USBness.... how does 180mA sound.

Sheesh, I apologize for any inconvenience caused!!

Good news is, so far she seems to be running great, I have OpenELEC on one card the besides some lag when navigating XBMC when playing a file, runs great.

And so far the debian squeeze is going great too.

Thanks again guys