jworr
Posts: 10
Joined: Tue Dec 16, 2014 3:05 am

Freezing with RT-patch (Pi 3)

Tue Sep 06, 2016 7:45 pm

I'm using the Pi 3 and need the RT-Preempt patch for a university course I'm TAing. I was able to download kernel version 4.4.9 from https://github.com/raspberrypi/linux/ar ... 6-1.tar.gz and patch it with the RT patch found at https://www.kernel.org/pub/linux/kernel ... 7.patch.gz. I chose this combination because it was the only element in the set of linux distributions available at https://github.com/raspberrypi/linux/releases that had the RT-patch available. However, sometimes when doing fairly computationally-heavy tasks (re-compiling the kernel or transferring fairly large files over sftp) and sometimes even when not doing any computationally-heavy tasks, the Pi will freeze and I will have to unplug its power cord and start the whole process over. This is a fairly common occurrence, and has been re-produced on every Pi we have attempted it on.

The previous version of the course was done on Raspberry Pi 2 and used kernel version 4.1.13 with the RT patch. Course website with specifics: http://www.cse.wustl.edu/~ferryd/course ... linux.html I tried using the exact same version for the Pi 3, but that just bricked it.

I've found other posts describing something similar without any definitive fix. Examples:
http://www.spinics.net/lists/linux-rt-u ... 12789.html talks about freezing with RT patch on Pi 2 running Kernel 3.18, but the "fix" there was a non-issue here. The proposed patch is already in the source code by 4.4.9.

The answer at http://raspberrypi.stackexchange.com/qu ... s-freezing didn't really apply or help anything.

I also found this blog post by Frank Durr that contains a pre-compiled version available for download (coincidentally the same version I initially ran). I downloaded his packaged version and got the same result. http://www.frank-durr.de/?p=203

Has anyone been able to get a steady version of the RT patch to work on the Raspberry Pi 3? If so, could you please tell which version of the kernel (and RT patch) you are using and provide links? Thank you.

James

sasanka
Posts: 3
Joined: Wed Feb 10, 2016 3:17 pm

Re: Freezing with RT-patch (Pi 3)

Wed Sep 14, 2016 2:13 pm

Hello James,

Could you build RT kernel for Pi2 without freeze? I am currently working on building RT-Kernel, I am trying with both Pi2 and Pi3. At high CPU loads, especially when there is heavy traffic on Ethernet port, Pi 2/3 seems to freeze and I see no messages being logged. I have been struggling to fix this problem. I tried from several directions like fixing power supply, disabling FIQ etc. Nothing seems to yield me the result. Kindly share your experience in case you are able to fix the problem.

Kind regards,
Sasanka.

giuliano
Posts: 8
Joined: Thu Sep 15, 2016 11:50 am
Location: Bologna - Italy
Contact: Website

Re: Freezing with RT-patch (Pi 3)

Thu Sep 15, 2016 12:04 pm

Hello,

I was experiencing the same trouble.
After searching around, it would appear that what makes PI 3 freeze in presence of the RT_PREEMP patch is the FIQ (fast interrupt handling) used for USB.
By adding to my cmdline.txt the following:
dwc_otg.fiq_fsm_enable=0 dwc_otg.fiq_enable=0 dwc_otg.nak_holdoff=0
I seem to have fixed the problem. (At least, what was freezing after no more than a few minutes is now running since a couple of hours!)
One word of caution: verify with a dmesg | grep otg that your command line has been properly taken into account. You should see a line telling "dwc_otg: FIQ disabled".

Giuliano

sasanka
Posts: 3
Joined: Wed Feb 10, 2016 3:17 pm

Re: Freezing with RT-patch (Pi 3)

Thu Sep 15, 2016 1:09 pm

Hello Giuliano,

I am currently using

Code: Select all

sdhci_bcm2708.enable_llm=0 smsc95xx.turbo_mode=N dwc_otg.dma_enable=1  dwc_otg.fiq_enable=0 dwc_otg.fiq_fsm_enable=0 dwc_otg.nak_holdoff=0
options in my cmdline.txt. All of those put together from my reading into various forums. As you mentioned, it does make it better, but not freeze free. Did you ever happen to notice any logs when it freezes? Do you have any suggestion towards enabling some kind of logging so as to capture the real reason behind this freeze. Apparently many forums seems to close this topic, however this seems to persist.

Best regards,
Sasanka.

giuliano
Posts: 8
Joined: Thu Sep 15, 2016 11:50 am
Location: Bologna - Italy
Contact: Website

Re: Freezing with RT-patch (Pi 3)

Thu Sep 15, 2016 5:47 pm

Hello Sasanka,

in my current setup I haven't yet seen any freeze, after 7 hours.
I have found a bug in my real-time application, but that's another story. My application segfaults and dies, but the system keeps running.
I'll let it run overnight, and if tomorrow it is still alive, I'll provide you more details, because I had to fight a little with kernel parameters, where the order in which they're put appears to be crucial to make it work.

Giuliano

giuliano
Posts: 8
Joined: Thu Sep 15, 2016 11:50 am
Location: Bologna - Italy
Contact: Website

Re: Freezing with RT-patch (Pi 3)

Fri Sep 16, 2016 9:17 am

Hi Sasanka

I didn't experience any freeze after 22 hours.
11:14:06 up 22:18, 3 users, load average: 0,54, 0,51, 0,49

My setup:
Linux raspberrypi 4.4.13-rt19-v7+ #2 SMP PREEMPT RT Fri Jun 24 17:53:00 CEST 2016 armv7l GNU/Linux

Acually I have used a stock 4.4.13 kernel from the raspberry Git (an rpi 4.4.y) to whom I applied the nearest RT_PREEMP patch (4.4.12-rt19). I need to compile the kernel myself, because I need a patch to support the 9 bit operation of the UART, which is included only in 4.6 kernels.

My full cmdline.txt:
8250.nr_uarts=1 dma.dmachans=0x7f35 bcm2708_fb.fbwidth=1366 bcm2708_fb.fbheight=768 bcm2709.boardrev=0xa02082 bcm2709.serial=0x3adda008 smsc95xx.macaddr=B8:27:EB:DD:A0:08 bcm2708_fb.fbswap=1 bcm2709.uart_clock=48000000 vc_mem.mem_base=0x3dc00000 vc_mem.mem_size=0x3f000000 dwc_otg.lpm_enable=0 console=ttyS0,115200 console=tty1 root=/dev/mmcblk0p7 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait dwc_otg.fiq_fsm_enable=0 dwc_otg.fiq_enable=0 dwc_otg.nak_holdoff=0

It appears that what did the trick has been the order of the fiq disable commands. dwc_otg.fiq_fsm_enable=0 must be set *before* dwc_otg.fiq_enable=0. The other way around it finds fiq_fsm_enable true with fiq_enable false, and it forces fiq_enable true.

This can be checked with
cat /proc/interrups (you should not see a FIQ entry)
or with
dmesg | grep otg
which should show something like:
[ 3.295773] dwc_otg: FIQ disabled
[ 3.295786] dwc_otg: NAK holdoff disabled
[ 3.295797] dwc_otg: FIQ split-transaction FSM disabled

I hope that it helps.

Giuliano

rodizio
Posts: 43
Joined: Sat May 07, 2016 2:40 am

Re: Freezing with RT-patch (Pi 3)

Sun Oct 02, 2016 6:40 am

I have a similar problem with the Raspbian kernel pulled via rpi-source using the same config as stock Raspbian (running kernel config pulled from /proc directory). When using anything higher than 100Hz it freezes randomly.

jpiat
Posts: 10
Joined: Mon Sep 01, 2014 8:07 am

Re: Freezing with RT-patch (Pi 3)

Sun Oct 02, 2016 8:07 am

I experienced the same problem with the Pi Zero. When doing transfer over the USB/ethernet interface, the zero would freeze with RT kernel but would work nice with normal kernel.

giuliano
Posts: 8
Joined: Thu Sep 15, 2016 11:50 am
Location: Bologna - Italy
Contact: Website

Re: Freezing with RT-patch (Pi 3)

Mon Oct 03, 2016 10:01 am

@rodizio
@jpiat
I have submitted the matter to the RT_PREEMPT list, and the developers suggest that there is a bug in the FIQ USB implementation, which is only made more evident by the usage of the RT patch.
You may try and disable the FIQ in the command line. If the freezes disappear also in your cases, then we may confirm the bug and submit the matter to rpi3 kernel peolple.

Giuliano

rodizio
Posts: 43
Joined: Sat May 07, 2016 2:40 am

Re: Freezing with RT-patch (Pi 3)

Mon Oct 03, 2016 12:20 pm

Just for clarity, in my case the issue happens _without_ using any RT patches, the only change I made to the rpi-source kernel was setting the timer frequency to 1000Hz.

Did some kernel compilations while receiving wifi packets (this would make it freeze often), seems stable now with the above mentioned cmdline.txt options.

Not sure how this relates, but I had delayed wifi packets on the USB bus before (with 100Hz timer freq), this is much better now with 1000Hz.

giuliano
Posts: 8
Joined: Thu Sep 15, 2016 11:50 am
Location: Bologna - Italy
Contact: Website

Re: Freezing with RT-patch (Pi 3)

Mon Oct 03, 2016 3:12 pm

rodizio wrote:Just for clarity, in my case the issue happens _without_ using any RT patches, the only change I made to the rpi-source kernel was setting the timer frequency to 1000Hz.

Did some kernel compilations while receiving wifi packets (this would make it freeze often), seems stable now with the above mentioned cmdline.txt options.

Not sure how this relates, but I had delayed wifi packets on the USB bus before (with 100Hz timer freq), this is much better now with 1000Hz.
As there are chances that the freeze is not RT patch specific, but only made more evident by the RT patch, I thought it interesting to verify if disabling the FIQ solves the problem also on a non-patched kernel. This would make more likely that the matter is taken over by Raspbian kernel people.

Giuliano

rodizio
Posts: 43
Joined: Sat May 07, 2016 2:40 am

Re: Freezing with RT-patch (Pi 3)

Sat Oct 08, 2016 11:45 am

So far it works for me, but another user who also tried 1000Hz _without_ RT patches (to get rid of the USB latency which causes delayed wifi packets) reported problems when using wifi cards and an USB joystick at the same time.
I tried my image with the new 1000Hz kernel option, and I can not login anymore, the system seems to be too unstable to be usable it freezes all the time,
The image instability is solved as soon as I changed the timer frequency back to 250hz. The instability with 1000hz frequency is only shown with the usb joystick and the wifi stick (tplink WN722N) connected.
He also uses the above mentioned dwc_otg options in the cmdline.txt

This whole thing is just inherently unstable, especially when USB comes into play ... I still remember how we all laughed when Bill Gates plugged in a USB device at that big presentation on a large screen and Windows would instantly blue-screen. Now about twenty years later, Linux seems to have arrived at the same "stability" :(

User avatar
jens.maus
Posts: 18
Joined: Sun Jan 01, 2017 10:53 am
Location: Dresden, Germany
Contact: Website

Re: Freezing with RT-patch (Pi 3)

Wed Jan 04, 2017 11:30 pm

Hello,

please note that I was suffering from the exact same symptoms as mentioned here. Here, I am using Linux kernel version 4.4.38 with the 4.4.38-rt49 PREEMPT_RT kernel patches under a buildroot 2016.11 driven system and my RaspberryPi3 was freezing after some time until the watchdog automatically resets the Pi3.

After having read through this topic I then changed the kernel command line to contain dwc_otg.fiq_fsm_enable=0 dwc_otg.fiq_enable=0 dwc_otg.nak_holdoff=0 and after having rebooted the system is now stable for more than a half day without further freezes/crashes. So thanks for the comments in here.

I still, however, wonder if there isn't any direct fix in the kernel for this issue so that these kernel command parameters doesn't have to be used anymore which to me seems to be the best long-term solution.

So have there been any progress on that topic and has some kernel developer investigated that issue so far?

giuliano
Posts: 8
Joined: Thu Sep 15, 2016 11:50 am
Location: Bologna - Italy
Contact: Website

Re: Freezing with RT-patch (Pi 3)

Thu Jan 05, 2017 11:42 pm

Yes, finally there is a fix, provided by the linux-rt kernel developers (actually by Ossama Gorbhel). You may find a description and a link to the patch at:
https://www.osadl.org/Single-View.111+M ... c57.0.html
I have not yet tested it, but the test system at OSADL running a real-time cyclictest sports now an uptime of 25 days, which is quite encouraging
https://www.osadl.org/Profile-of-system ... =1#patches

Giuliano

User avatar
jens.maus
Posts: 18
Joined: Sun Jan 01, 2017 10:53 am
Location: Dresden, Germany
Contact: Website

Re: Freezing with RT-patch (Pi 3)

Sat Jan 07, 2017 12:37 pm

giuliano wrote:Yes, finally there is a fix, provided by the linux-rt kernel developers (actually by Ossama Gorbhel). You may find a description and a link to the patch at:
https://www.osadl.org/Single-View.111+M ... c57.0.html
I have not yet tested it, but the test system at OSADL running a real-time cyclictest sports now an uptime of 25 days, which is quite encouraging
https://www.osadl.org/Profile-of-system ... =1#patches
Thanks for your comments and the link to a patch that tries to fix these problems. I was now able to actually integrate this patch set into one of my buildroot-based projects (https://github.com/jens-maus/RaspberryMatic) which was suffering from these kind of freezes where at some point the hardware watchdog was kicking in and resetting my Pi3 system. Now after having integrated the patch set (see https://github.com/jens-maus/RaspberryM ... -270383497) my test system is running stable for more than 24h, thus I was finally able to merge PREEMPT_RT support into my master development branch.

So thanks again for this nice patch, it seems to essentially fix these kind of freezing problems with a RaspberryPi and the PREEMPT_RT patch set. Still I am wondering when these kind of fixes will be accepted by upstream (have they been already submitted to the linux kernel developers?).

User avatar
jens.maus
Posts: 18
Joined: Sun Jan 01, 2017 10:53 am
Location: Dresden, Germany
Contact: Website

Re: Freezing with RT-patch (Pi 3)

Sun Jan 08, 2017 10:26 am

Hi again,

now that the freezing seems to be solved with the patch mentioned above I just recognised that I am receiving the following dmesg notifications from time to time:
[ 96.333042] NOHZ: local_softirq_pending 80
[ 108.953020] NOHZ: local_softirq_pending 80
[ 2251.054859] NOHZ: local_softirq_pending 80
[ 2283.684748] NOHZ: local_softirq_pending 80
[ 3312.780926] NOHZ: local_softirq_pending 80
[ 4984.076761] NOHZ: local_softirq_pending 80
[ 5185.266491] NOHZ: local_softirq_pending 80
[ 6677.041598] NOHZ: local_softirq_pending 80
[ 7125.703680] NOHZ: local_softirq_pending 80
[ 7125.703787] NOHZ: local_softirq_pending 80
And now I wonder why this is not happening with the non-PREEMPT_RT kernel version? Are these messages critical and if so, how can I get rid of them? Anyone knows what these messages are all about (couldn't find any help at google)?

giuliano
Posts: 8
Joined: Thu Sep 15, 2016 11:50 am
Location: Bologna - Italy
Contact: Website

Re: Freezing with RT-patch (Pi 3)

Sun Jan 08, 2017 11:57 am

AFAIK we will have to live wih this situation for some time to come.
The RPI spectific kernel patch has not yet made its way to upstream, and it's maintained by the Raspberry foundation.
The RT_PREEMPT patch has not yet made its way to upstream and it's maintained by the linux-rt group. Mainline upstream integration is a goal, but it's not yet implemented.
To my knowledge, at the moment the easiest way for an end user to get an RPI real-time kernel is to take advantage of the OSADL QA Farm, and get the from there kernel and patches, possibly avoiding the extra measurement patch, which is unnecessary in a "production" kernel.

Giuliano

giuliano
Posts: 8
Joined: Thu Sep 15, 2016 11:50 am
Location: Bologna - Italy
Contact: Website

Re: Freezing with RT-patch (Pi 3)

Sun Jan 08, 2017 12:32 pm

jens.maus wrote:Hi again,

now that the freezing seems to be solved with the patch mentioned above I just recognised that I am receiving the following dmesg notifications from time to time:
[ 96.333042] NOHZ: local_softirq_pending 80
<snip></snip>
And now I wonder why this is not happening with the non-PREEMPT_RT kernel version? Are these messages critical and if so, how can I get rid of them? Anyone knows what these messages are all about (couldn't find any help at google)?
This message tells you that a CPU has gone idle with a soft irq still pending. Meaning that the relate irq service will be delayed until the said CPU will be running again. You don't see it with the non-RT_PREEMP kernel, because the RT_PREEMPT patch transforms a number of hard irq's into soft irq's, to enable preemption. By looking at /proc/interrupts you should be able to tell to which event irq 80 is connected, and determine if it is critical for your application, or irrelevant.
By googling I found out that this is a known issue since a long time (a workaround to reduce the amount of those messages has been provided in 2007!). It would appear that no solution has been found up to now...

You might choose to disable the kernel NOHZ option in kernel configurtion, and leave the system timer always running. You would gain in soft irq response time and get rid of those warnings, but loose in overall efficiency, because of a lot of useless tick interrupts. The choice is application specific.

Giuliano

Return to “Advanced users”