setting to stabilize the official image


28 posts   Page 1 of 2   1, 2
by elatllat » Mon Aug 13, 2012 12:56 am
Please add "smsc95xx.turbo_mode=N" to /boot/cmdline.txt in the official image, it prevents the rpi from crashing under load.
Also prevents the log from filling with this:
Code: Select all
smsc95xx 1-1.1:1.0: eth0: kevent 2 may have been dropped
http://raspberrypi.stackexchange.com/search?q=search%20first
https://google.com/search?q=site%3Awww.raspberrypi.org+search+first
https://www.raspberrypi.org/forums/search.php?search_id=unanswered
Posts: 1336
Joined: Sat Dec 17, 2011 5:05 pm
by El_Presidente » Tue Aug 14, 2012 4:19 pm
Can the logic behind this be explained - i.e. what does the default (which I assume is 'Y') cause the Pi and/or processes to do?
-----------------------------------------------------------------------------------------
Never trust a man, who when left alone with a tea cosey, doesn't try it on.
Posts: 36
Joined: Tue Apr 03, 2012 7:02 am
Location: Bolton, UK
by elatllat » Tue Aug 14, 2012 4:30 pm
El_Presidente wrote:[What does the default cause the rpi to do?]


Maybe you missed the
elatllat wrote:crashing under load

and
elatllat wrote:[log filed with "kevent 2 may have been dropped"]

part of the OP
http://raspberrypi.stackexchange.com/search?q=search%20first
https://google.com/search?q=site%3Awww.raspberrypi.org+search+first
https://www.raspberrypi.org/forums/search.php?search_id=unanswered
Posts: 1336
Joined: Sat Dec 17, 2011 5:05 pm
by El_Presidente » Tue Aug 14, 2012 4:43 pm
I should have explained myself better - I know what the setting aims to cure, but not why the default setting would have been causing the crash under load and error logging.

In other words, what is going on inside the Pi when the turbo mode is on?
-----------------------------------------------------------------------------------------
Never trust a man, who when left alone with a tea cosey, doesn't try it on.
Posts: 36
Joined: Tue Apr 03, 2012 7:02 am
Location: Bolton, UK
by elatllat » Tue Aug 14, 2012 5:51 pm
El_Presidente wrote:I should have explained myself better - I know what the setting aims to cure, but not why the default setting would have been causing the crash under load and error logging.

In other words, what is going on inside the Pi when the turbo mode is on?


Yes that is a good question but they already know there are driver problems, and are working on them.

The point of this thread is to update the default build with a super quick fix that will reduce mass crash-frustration :)
http://raspberrypi.stackexchange.com/search?q=search%20first
https://google.com/search?q=site%3Awww.raspberrypi.org+search+first
https://www.raspberrypi.org/forums/search.php?search_id=unanswered
Posts: 1336
Joined: Sat Dec 17, 2011 5:05 pm
by Burngate » Wed Aug 15, 2012 4:41 pm
elatllat wrote:... they already know there are driver problems, and are working on them.

They know, but we would like to know more.

Magic incantations are ok, but some of the background would be interesting
User avatar
Posts: 4197
Joined: Thu Sep 29, 2011 4:34 pm
Location: Berkshire UK
by elatllat » Wed Aug 15, 2012 5:10 pm
Burngate wrote:Magic incantations are ok, but some of the background would be interesting



I have not read (nor can I because some of it is closed source) every line of code on the rpi so it's mostly "magic incantations" to me, but Google said turbo_mode Enables multiple frames per Rx transaction.
http://lxr.free-electrons.com/source/dr ... smsc95xx.c
So clearly there is a problem with that feature in either the ethernet or usb code.
("SMSC95XX USB 2.0 Ethernet Devices")

Do you want to know anything specific?
http://raspberrypi.stackexchange.com/search?q=search%20first
https://google.com/search?q=site%3Awww.raspberrypi.org+search+first
https://www.raspberrypi.org/forums/search.php?search_id=unanswered
Posts: 1336
Joined: Sat Dec 17, 2011 5:05 pm
by robwriter » Wed Aug 15, 2012 5:10 pm
I don't get random crashes.
Posts: 114
Joined: Sat Feb 25, 2012 6:26 pm
by elatllat » Wed Aug 15, 2012 5:25 pm
robwriter wrote:I don't get random crashes.

It's not "random" it's "under load" as stated in the OP.
A torrent client saving to disk is a good example because it stresses CPU and the USB with IO from eth0 and sda.
Copying lage files over the network and other compound loads are also reported to cause crashing.
http://raspberrypi.stackexchange.com/search?q=search%20first
https://google.com/search?q=site%3Awww.raspberrypi.org+search+first
https://www.raspberrypi.org/forums/search.php?search_id=unanswered
Posts: 1336
Joined: Sat Dec 17, 2011 5:05 pm
by El_Presidente » Thu Aug 16, 2012 5:13 pm
OK - so I've done my own Googling and find that the turbo mode is meant to make the Ethernet more efficient but at the cost of using more Kernel memory. Putting this together with comments from around the Pi forum, it looks like the turbo mode grabs too much resource when heavily using the network, leading to kevent drops and page allocation issues. Others believe that because the Ethernet is effectively a port off the USB 3-port hub built into the Pi, it is flooding the whole USB subsystem, thus being a contributor to overall crap USB experience as well as kernel panic/crash.

Disabling turbo mode will result in lower Ethernet performance, but less stress on kernel memory and lower hit rate on the USB interrupt processing, but I'm not sure if anyone has produced proper before/after stats yet.

That will do for me. Position understood.
-----------------------------------------------------------------------------------------
Never trust a man, who when left alone with a tea cosey, doesn't try it on.
Posts: 36
Joined: Tue Apr 03, 2012 7:02 am
Location: Bolton, UK
by chrisw2 » Thu Aug 16, 2012 5:25 pm
Thanks, El_Presidente.
Posts: 106
Joined: Sat Apr 07, 2012 11:22 am
Location: Manchester, UK
by Burngate » Thu Aug 16, 2012 5:27 pm
El_Presidente wrote:That will do for me. Position understood.

And me. Thanks
User avatar
Posts: 4197
Joined: Thu Sep 29, 2011 4:34 pm
Location: Berkshire UK
by dom » Wed Sep 26, 2012 7:24 pm
Has anyone found anything that is worse with:
smsc95xx.turbo_mode=N

I'd assumed it slows down the network performance, but iperf says it is just as fast.

Any downside to this?
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 4889
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge
by Ignas » Wed Sep 26, 2012 7:32 pm
Been using this setting for a month now. Pi is running rtorrent 24/7. No crashes, dmesg is clear.

Download speed was a disappointment (at around 2MB/s), but now with turbo mode present I can max out my old wrt54gl (~5-6MB/s). Still stable.
Posts: 7
Joined: Wed Jul 18, 2012 2:05 pm
Location: Šiauliai, Lithuania
by dom » Wed Sep 26, 2012 7:45 pm
Ignas wrote:Been using this setting for a month now. Pi is running rtorrent 24/7. No crashes, dmesg is clear.

Download speed was a disappointment (at around 2MB/s), but now with turbo mode present I can max out my old wrt54gl (~5-6MB/s). Still stable.

Any difference in download speed with and without:
smsc95xx.turbo_mode=N
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 4889
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge
by Ignas » Wed Sep 26, 2012 8:05 pm
dom wrote:
Ignas wrote:Been using this setting for a month now. Pi is running rtorrent 24/7. No crashes, dmesg is clear.

Download speed was a disappointment (at around 2MB/s), but now with turbo mode present I can max out my old wrt54gl (~5-6MB/s). Still stable.

Any difference in download speed with and without:
smsc95xx.turbo_mode=N


Well, to be honest I've never really tried to reach max download speed without smsc95xx.turbo_mode=N. And I think it's impossible without crashing the Pi itself. I was keeping download limit at 200-300KB/s to prevent dmesg from flooding with "kevent 2 may have been dropped" messages. I would try now, but my Pi is in the remote location and I don't have the luxury of power cycling if anything goes wrong. I've read that loads of "kevent 2 may have been dropped" will eventually lead to the crash of Pi.
Posts: 7
Joined: Wed Jul 18, 2012 2:05 pm
Location: Šiauliai, Lithuania
by elatllat » Wed Sep 26, 2012 8:12 pm
dom wrote:Has anyone found anything that is worse with:
smsc95xx.turbo_mode=N

I'd assumed it slows down the network performance, but iperf says it is just as fast.

Any downside to this?


No down side that I have seen.
I have not explicitly tested if there is any speed difference because without this setting there is crazy ping times, paket loss, and crashing.
I am able to torrent, decrypt, and stram 2 HD movies off the raspberrypi using this setting.
I get 3MiB/s using scp but ssh eating all the CPU is the limiting factor there.
I get 7 or 8MiB/s using pv over smbfs (external drives: kworker(65%), smbd(28%), etc max out the CPU) sdcard smbd only leaves ~5% idle.

Note that I am stuck using an old kernel:
viewtopic.php?f=28&t=18155&p=181911
but it seems to have only positive effects the ~205 times it's been used:
https://www.google.com/search?hl=en&q=s ... phpBB3/%20"smsc95xx.turbo_mode=N"
http://raspberrypi.stackexchange.com/search?q=search%20first
https://google.com/search?q=site%3Awww.raspberrypi.org+search+first
https://www.raspberrypi.org/forums/search.php?search_id=unanswered
Posts: 1336
Joined: Sat Dec 17, 2011 5:05 pm
by elnoziya » Fri Oct 12, 2012 9:10 am
Hi all:

I have been bearing with transmission-daemon crashes till turning off turbo mode, it seems to be working fine after that. However right now my feeling is that rpi is working much slower. In raspi-config I am using "none" as overclocking option, which one are you using?
Posts: 1
Joined: Fri Oct 12, 2012 8:59 am
by elatllat » Fri Oct 12, 2012 12:39 pm
elnoziya wrote:feeling...slower...overclocking?

ask in another thread how to test and prove your 'feeling'. i'm not overclocking and it is also a different thread, please keep this one clean.
http://raspberrypi.stackexchange.com/search?q=search%20first
https://google.com/search?q=site%3Awww.raspberrypi.org+search+first
https://www.raspberrypi.org/forums/search.php?search_id=unanswered
Posts: 1336
Joined: Sat Dec 17, 2011 5:05 pm
by milhouse » Wed Nov 07, 2012 8:56 pm
I raised issue #153 to report problems with allocation failures while running Transmission with one or two concurrent downloads, and the suggestion to set smsc95xx.turbo_mode=N has fixed the errors (and associated kevent 2 floods) - I've tested with 5 concurrent torrent downloads and there were no problems (512MB board). There is a slight reduction in maximum network throughput, to the order of about 1.5MB/s, but this is acceptable given the alternative.
Posts: 611
Joined: Mon Jan 16, 2012 12:59 pm
by eshatohhin » Sun Dec 30, 2012 6:00 pm
Setting vm.min_free_kbytes to 16384 in /etc/sysctl.conf made my pi more stable under load.

Source: http://www.nerdoncoffee.com/?p=195
http://blogs.linbit.com/p/28/kernel-min_free_kbytes/
User avatar
Posts: 11
Joined: Tue May 15, 2012 12:27 pm
Location: Estonia
by milhouse » Mon Dec 31, 2012 12:44 pm
eshatohhin wrote:Setting vm.min_free_kbytes to 16384 in /etc/sysctl.conf made my pi more stable under load.


See the issue I linked above - even with 32MB (32768) allocated for vm.min_free you'll still get problems. Given the Pi only has 256MB/512MB of RAM, with a chunk of that allocated to the GPU, throwing more RAM at this problem isn't really a solution, even if it succeeded.

While disabling turbo mode does reduce network performance by up 1.5MB/s it does at least ensure 100% network reliability even with only 1536 (1.5MB) allocated to vm.min_free.
Posts: 611
Joined: Mon Jan 16, 2012 12:59 pm
by elatllat » Mon Sep 30, 2013 7:57 pm
Tested the latest stable build and it seems
Code: Select all
smsc95xx.turbo_mode=N

is still required for stability and is still not default.
Code: Select all
cat /etc/issue
Raspbian GNU/Linux 7 \n \l
uname -a
Linux raspbnas 3.6.11+ #538 PREEMPT Fri Aug 30 20:42:08 BST 2013 armv6l GNU/Linux
http://raspberrypi.stackexchange.com/search?q=search%20first
https://google.com/search?q=site%3Awww.raspberrypi.org+search+first
https://www.raspberrypi.org/forums/search.php?search_id=unanswered
Posts: 1336
Joined: Sat Dec 17, 2011 5:05 pm
by jdb » Mon Sep 30, 2013 8:36 pm
Can you post dmesg logs or similar during your "instability"?

Several changes have been made recently that should alleviate the occurrence of these messages.

Note: The "kevent 2" messages are simply as a result of kernel memory allocation failures. The way that usbnet works, it will try initially to allocate a large contiguous section of memory and progressively reduce it in size until the allocation succeeds. There should be no obvious impact apart from userspace program slowdowns.

Edit: Your kernel was built on Aug 30th - please try with latest rpi-update kernel.
Rockets are loud.
https://astro-pi.org
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 1545
Joined: Thu Jul 11, 2013 2:37 pm
by harshalvora » Wed May 07, 2014 2:56 pm
On Kernel 3.14.0, I still get
smsc95xx 1-1.1:1.0: eth0: kevent 2 may have been dropped
Before this I see some kernel Warning crash related to ieee80211 module.

Is it recommended to set
"smsc95xx.turbo_mode=N"
even in the latest kernel?
Posts: 6
Joined: Sat Apr 19, 2014 12:02 pm