CONFIG_PREEMPT_RT on Raspberry Pi

[tom.k.cook]

Any luck with cyclictest -W or-f?

[aaa801]

Any luck with cyclictest -W or-f?

I didn't build the kernel with tracer support, also doesn't -f need -b?
Il recompile the kernel in a sec

[tom.k.cook]

Any luck with cyclictest -W or-f?

I didn't build the kernel with tracer support, also doesn't -f need -b?
Il recompile the kernel in a sec

Yes it does. I've got 3.6rt kernels to this point but they crash when tracers are enabled .

[aaa801]

Any luck with cyclictest -W or-f?

I didn't build the kernel with tracer support, also doesn't -f need -b?
Il recompile the kernel in a sec

Yes it does. I've got 3.6rt kernels to this point but they crash when tracers are enabled .

what do i need to change in .config to enable tracers

[TrigZu]

Hey Tom,

I was wondering if you had made any progress on the TODOs listed in your original post.

[tom.k.cook]

A quick update: The problem with cyclictest took me a long time to figure out, but its easy when you get there:

Code: Select all

> sudo -s
> mount -t debugfs none /sys/kernel/debug
> echo dwc_otg_hcd_handle_fiq > /sys/kernel/debug/tracing/set_ftrace_notrace
> ./cyclictest -p95 -m -f -b 1000

This no longer crashes. Of course, the numbers aren't great either. In fact, you might call them dreadful, though enabling the tracers makes them around 20x worse. I'm seeing max latency of around 650us at the moment. It's not great, but it could be worse. Now to use tracers to find out what's causing it...

For those wanting details: The problem is that the new dwc_otg driver uses fast interrupts (FIQs) to do some fast processing that makes the USB interface a lot more responsive. When you enable tracers, it instruments the FIQ handler function (dwc_otg_hcd_handle_fiq) along with everything else. The implementation of instrumentation on ARM uses hardware watchpoints, a CPU feature that causes a data abort when a particular memory location is accessed or when an instruction at a particular location is executed. Data aborts in FIQ mode are unhandled in the Linux kernel, resulting in an oops. By excluding dwc_otg_hcd_handle_fiq from the functions traced, we can avoid the oops.

[ranpitime]

For me it is not really clear why with xenomai we see 40-60us jitter and with PREEMPT_RT around 650us,
would prefer to use PREEMPT_RT but in that case this is not really possible...
Any explanations so far?
Thanks!

[tom.k.cook]

They are quite different. Xenomai (so I understand) runs as a hypervisor-style kernel with RT tasks run in parallel with the linux kernel, whereas PREEMPT_RT gives you real-time latencies in the linux userspace. Each has advantages and disadvantages; Xenomai makes it easier to give low latencies, but PREEMPT_RT gives you access to the full linux userspace (all the libraries etc) in a realtime context.

BTW with all the tracing switched off I currently see ~100us maximum latency using 3.8.11-rt13 - it's worse than Xenomai, but not heaps worse.

[aaravmadan]

After downloading the required tools and kernel files from git .. i patched kernel 3.8.13 with rt-11 patch as mentioned above ..

i get 1 hunk failed at line 49 when the patching is at drivers/misc
so it says saving rejects to Makefile.rej

Dont know how to proceed

[AutoStatic]

http://www.spinics.net/lists/linux-rt-u ... 08062.html

With this patch I managed to solve a similar MMC error on a different device (RK3066 Android TV stick), maybe it works for the RPi too.

Edit: nope, doesn't work.

[schorsch76]

As already emailed to tom here is my finding:

I used kernel 3.8.13 from branch rpi-3.8.y and patched with 3.8.13-rt12. Used the config from 3.6.11 and selected full preempt. I got the usual dead lock on booting.

I changed the following:
drivers/mmc/host/sdhci_bcm2708.c: Line 141 enable_llm=0
and activated the following options

Kernel Hacking:
CONFIG_DEBUG_ATOMIC_SLEEP=y
CONFIG_LOCKUP_DETECTOR=y
CONFIG_DEBUG_SPINLOCK=y
CONFIG_DEBUG_MUTEXES=y
CONFIG_DEBUG_LOCK_ALLOC=y
CONFIG_PROVE_LOCKING=y

Now the MMC performance is low, but i can at least boot from the sdcard. I attached my config

Bye
schorsch

[schorsch76]

After running the rpi with the rt kernel and enabled debug options, i found the following in dmesg:

Code: Select all

[   27.320291] mmc0: final write to SD card still running
[   37.351983] mmc0: Timeout waiting for hardware interrupt - cmd12.
[   37.368209] mmcblk0: error -110 sending stop command, original cmd response 0x900, card status 0x900
[   38.177536] mmc0: final write to SD card still running
[   48.211981] mmc0: Timeout waiting for hardware interrupt - cmd12.

that appears again and again. This could hint, that the problem is the recognition of the irq, if it was issued by the hardware. The rt patch changes how the irqs are scheduled, and thus means other timing. Maybe there is a race condition....

There is an other problem too:

Code: Select all

    3.214878] 
[    3.214883] =============================================
[    3.214886] [ INFO: possible recursive locking detected ]
[    3.214900] 3.8.13-rt12+ #6 Not tainted
[    3.214903] ---------------------------------------------
[    3.214914] khubd/17 is trying to acquire lock:
[    3.214973]  (sl){+.+...}, at: [<c02ca878>] DWC_SPINLOCK_IRQSAVE+0x14/0x20
[    3.214976] 
[    3.214976] but task is already holding lock:
[    3.214999]  (sl){+.+...}, at: [<c02ca878>] DWC_SPINLOCK_IRQSAVE+0x14/0x20
[    3.215002] 
[    3.215002] other info that might help us debug this:
[    3.215006]  Possible unsafe locking scenario:
[    3.215006] 
[    3.215009]        CPU0
[    3.215011]        ----
[    3.215016]   lock(sl);
[    3.215022]   lock(sl);
[    3.215024] 
[    3.215024]  *** DEADLOCK ***
[    3.215024] 
[    3.215026]  May be due to missing lock nesting notation
[    3.215026] 
[    3.215034] 3 locks held by khubd/17:
[    3.215067]  #0:  (&__lockdep_no_validate__){......}, at: [<c0297f30>] hub_thread+0x9c/0x11b8
[    3.215092]  #1:  (usb_address0_mutex){+.+.+.}, at: [<c0295ec0>] hub_port_init+0x58/0x998
[    3.215115]  #2:  (sl){+.+...}, at: [<c02ca878>] DWC_SPINLOCK_IRQSAVE+0x14/0x20
[    3.215118] 
[    3.215118] stack backtrace:
[    3.215125] Backtrace: 
[    3.215178] [<c00111ac>] (dump_backtrace+0x0/0x10c) from [<c03d4b30>] (dump_stack+0x18/0x1c)
[    3.215196]  r6:da8ea7b8 r5:c0c63e64 r4:c07e756c r3:da8ea220
[    3.215235] [<c03d4b18>] (dump_stack+0x0/0x1c) from [<c005fca8>] (__lock_acquire+0x1614/0x1bbc)
[    3.215256] [<c005e694>] (__lock_acquire+0x0/0x1bbc) from [<c0060750>] (lock_acquire+0x68/0x7c)
[    3.215277] [<c00606e8>] (lock_acquire+0x0/0x7c) from [<c03da668>] (rt_spin_lock+0x4c/0x5c)
[    3.215294]  r7:c0cb0544 r6:c06e8711 r5:c02ca878 r4:da9e6440
[    3.215319] [<c03da61c>] (rt_spin_lock+0x0/0x5c) from [<c02ca878>] (DWC_SPINLOCK_IRQSAVE+0x14/0x20)
[    3.215329]  r5:da8edc34 r4:da8edc34
[    3.215351] [<c02ca864>] (DWC_SPINLOCK_IRQSAVE+0x0/0x20) from [<c02c205c>] (dwc_otg_hcd_select_transactions+0x168/0x30c)
[    3.215361]  r4:da9e6440 r3:daa91b40
[    3.215379] [<c02c1ef4>] (dwc_otg_hcd_select_transactions+0x0/0x30c) from [<c02c22fc>] (dwc_otg_hcd_urb_enqueue+0xfc/0x198)
[    3.215397] [<c02c2200>] (dwc_otg_hcd_urb_enqueue+0x0/0x198) from [<c02c3060>] (dwc_otg_urb_enqueue+0x228/0x300)
[    3.215415]  r8:daa69400 r7:da928bc0 r6:daa7d920 r5:00000040 r4:00000000
[    3.215440] [<c02c2e38>] (dwc_otg_urb_enqueue+0x0/0x300) from [<c029b2f0>] (usb_hcd_submit_urb+0xb8/0x6e4)
[    3.215465] [<c029b238>] (usb_hcd_submit_urb+0x0/0x6e4) from [<c029ca6c>] (usb_submit_urb+0xfc/0x320)
[    3.215486] [<c029c970>] (usb_submit_urb+0x0/0x320) from [<c029d724>] (usb_start_wait_urb+0x48/0xbc)
[    3.215506] [<c029d6dc>] (usb_start_wait_urb+0x0/0xbc) from [<c029d984>] (usb_control_msg+0xc4/0xf8)
[    3.215525]  r8:80000080 r7:00000100 r6:00000000 r5:00000040 r4:daa90c20
[    3.215544] [<c029d8c0>] (usb_control_msg+0x0/0xf8) from [<c02961c0>] (hub_port_init+0x358/0x998)
[    3.215562] [<c0295e68>] (hub_port_init+0x0/0x998) from [<c0298208>] (hub_thread+0x374/0x11b8)
[    3.215590] [<c0297e94>] (hub_thread+0x0/0x11b8) from [<c003c130>] (kthread+0xa4/0xac)
[    3.215612] [<c003c08c>] (kthread+0x0/0xac) from [<c000e0d0>] (ret_from_fork+0x14/0x24)

It seems to be incorrect locking in the usb controller driver.

Bye
schorsch

[aaravmadan]

hi
i hv been tryin to compile rpi-linux-3.8.13 patched wid rt 11
i seem t compile the bcmrpi_defconfig widout ne errors
bt wen i load the sd card in pi the booting stops giving after this line :

Waiting for root device /dev/mmcblk0p6...

any help wud b appreciated

[AutoStatic]

Here's a diff against linux/drivers/mmc/host/sdhci.c that will allow booting a 3.6.11-rt31 RT kernel with enable_llm=1

Code: Select all

--- linux/drivers/mmc/host/sdhci.c	2013-07-03 11:54:06.306455988 +0200
+++ linux-rt/drivers/mmc/host/sdhci.c	2013-07-03 16:49:33.298601044 +0200
@@ -162,9 +162,17 @@
 			preempt_schedule();
 		}
 		spin_lock_irqsave(&host->lock,*flags);
+#ifdef CONFIG_PREEMPT_RT_FULL
+		disable_irq_nosync(host->irq);
+#else
 		disable_irq(host->irq);
+#endif
 		if(host->second_irq)
+#ifdef CONFIG_PREEMPT_RT_FULL
+			disable_irq_nosync(host->second_irq);
+#else
 			disable_irq(host->second_irq);
+#endif
 		local_irq_enable();
 	}
 	else
@@ -1834,7 +1842,11 @@
 	host = mmc_priv(mmc);
 
 	sdhci_runtime_pm_get(host);
+#ifdef CONFIG_PREEMPT_RT_FULL
+	disable_irq_nosync(host->irq);
+#else
 	disable_irq(host->irq);
+#endif
 	spin_lock(&host->lock);
 
 	ctrl = sdhci_readw(host, SDHCI_HOST_CONTROL2);
@@ -1935,7 +1947,11 @@
 		wait_event_interruptible_timeout(host->buf_ready_int,
 					(host->tuning_done == 1),
 					msecs_to_jiffies(50));
+#ifdef CONFIG_PREEMPT_RT_FULL
+		disable_irq_nosync(host->irq);
+#else
 		disable_irq(host->irq);
+#endif
 		spin_lock(&host->lock);
 
 		if (!host->tuning_done) {

The result:

Code: Select all

pi@jj-rpi:~$ uname -a
Linux jj-rpi 3.6.11-rt31 #1 PREEMPT RT Wed Jul 3 16:56:33 CEST 2013 armv6l GNU/Linux
pi@jj-rpi:~$ cat /sys/module/sdhci_bcm2708/parameters/enable_llm
Y

[aaravmadan]

i have patched kernel 3.8.13 with rt11and cross compiled on ubuntu 12.10
when i copy the kernel .img and modules to rpi and boot it .. it stops after detecting keyboard and mouse and says usb disconnect device no 4 .
if i remove the usb devices and boot kernel it says waiting for root device dev/mmcblk0p6

any help would be appreciable

[AutoStatic]

You need to patch sdhci.c or use the kernel parameter mentioned in the first post of this thread.

[aaravmadan]

Thnx a lot ..
Got my kernel up and running

[emil.s]

Hello!

I have been trying to build an RT-kernel for my Pi for many hours now, but the build always fails...
I have tried with both the 3.6.11 and 3.8.13 kernel...

3.6 fails with:

Code: Select all

include/linux/spinlock_rt.h:106:2: error: implicit declaration of function ‘migrate_disable’ [-Werror=implicit-function-declaration]
In file included from include/linux/time.h:8:0,
                 from include/linux/timex.h:56,
                 from include/linux/sched.h:57,
                 from arch/arm/kernel/asm-offsets.c:13:
include/linux/seqlock.h: In function ‘write_seqcount_begin’:
include/linux/seqlock.h:157:2: error: implicit declaration of function ‘preempt_disable_rt’ [-Werror=implicit-function-declaration]
include/linux/seqlock.h: In function ‘write_seqcount_end’:
include/linux/seqlock.h:170:2: error: implicit declaration of function ‘preempt_enable_rt’ [-Werror=implicit-function-declaration]
include/linux/seqlock.h: In function ‘read_seqbegin’:
include/linux/seqlock.h:234:3: error: implicit declaration of function ‘migrate_enable’ [-Werror=implicit-function-declaration]
cc1: some warnings being treated as errors
/home/emil/src/RPi/linux-rpi-3.6.y/./Kbuild:81: recipe for target 'arch/arm/kernel/asm-offsets.s' failed
make[1]: *** [arch/arm/kernel/asm-offsets.s] Error 1
Makefile:815: recipe for target 'prepare0' failed
make: *** [prepare0] Error 2

And 3.8 with:

Code: Select all

include/linux/sched.h:1671:2: error: implicit declaration of function 'in_atomic' [-Werror=implicit-function-declaration]
cc1: some warnings being treated as errors
make[1]: *** [arch/arm/kernel/asm-offsets.s] Error 1
make: *** [prepare0] Error 2
make: *** Waiting for unfinished jobs....
  HOSTLD  scripts/mod/modpost

Make succeeds without the patch, so I guess there's some problems with the rt-patches?

[sona]

I am booting rPi from sdcard. I applied the rt-patch (3.8..13.13-rt25) patch on the 3.8 kernel(as mentioned above) enabled the PREEMPT_RTB directive. But the kernel is hitting BUG.

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
kenel BUG at ***/fs.jbd2/transaction.c:2065

Internal error: Oops - BUG: 0 [#1] PREEMPT ARM
Entering kdb (current=0xcab83060, pid 1438) Oops: (null)

Pid: 1, comm: swapper
CPU: 0 Not tainted (3.8.13-rt25+ #5)
PC is at do_get_write_access+0x3e4/0x4a0
LR is at do_get_write_access+0x3e4/0x4a0
pc : [<c01914f0>] lr : [<c01914f0>] psr: 20000113
******
******
Flags: nzCv IRQs off FIQs on Mode SVC_32 ISA ARM Segment user
Control: 00c5387d Table: 098b0008 DAC: 00000015
[<c0013a7c>] (unwind_backtrace+0x0/0xf0) from [<c0072a98>] (kdb_dumpregs+0x28/0x 50)
[<c0072a98>] (kdb_dumpregs+0x28/0x50) from [<c0074e1c>] (kdb_main_loop+0x3a8/0x6 fc)
[<c0074e1c>] (kdb_main_loop+0x3a8/0x6fc) from [<c0077500>] (kdb_stub+0x154/0x380 )
[<c0077500>] (kdb_stub+0x154/0x380) from [<c006e634>] (kgdb_handle_exception+0x1 f8/0x668)
more>
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

can't figure out, what is the issue here? can you please suggest, thanks

[tom.k.cook]

You could try reading the rest of the thread. The problem is that the SDHCI driver doesn't work in RT. The solution is to put the root partition on a USB stick or use AutoStatic's patch above: http://www.raspberrypi.org/phpBB3/viewt ... 80#p381480

[sona]

You could try reading the rest of the thread. The problem is that the SDHCI driver doesn't work in RT. The solution is to put the root partition on a USB stick or use AutoStatic's patch above: http://www.raspberrypi.org/phpBB3/viewt ... 80#p381480

thanks tom,
I kept the file system partition in USB (sdcard connected via USB adapter to the usb port of rPi), but still i got same error. Am i missing anything here?

[tom.k.cook]

Presumably you are still trying to build in the SDHCI/MMC drivers into the kernel, so it's still trying to mount the SD card, so it still crashes.

[mung]

Has anyone done any profiling to workout what problems maybe occuring with RT_PREEMPT?

I have patched and tested almost all versions of the rpi kernel from git and they all seem to boot fine from SD using audiostatics patch (some hacking required as they do reject a number of chunks).

But I find that although they run with no noticable problems they can have appaling RT performance.

cyclictest mostly seems okay run from commandline with few other processes running (~60us average), but running X and loading programs like midori can give max spikes of 5000us.

I know nothing about RT profiling, has anyone looked at it?

I am wondering if its the SD card or if the videocore does some weird system management functions (cache memory refresh etc...??).

I am not going to bother looking into it unless some broadcom expert can offer advice, and even then probably wont bother as I don't think I have the expertise.

I have a vague recollection that somewhere (http://www.spinics.net/lists/linux-rt-u ... 10256.html ?????) spinlocks were mentioned but this seems to be the audiostatic patch, I was considering greping the entire kernel tree for occurences of spin_lock_irqsave and analysing the code, but did not as I am guessing this is not the problem?

Also wondering if dma maybe causing problems at some point?

If anyone that actually understands this stuff has time to explain, maybe I may try and spend 20hours hacking things, but I really don't think I am experienced enough to do it, I don't know the kernel source, and it seems way too complex to just start hacking without many hours of research and code greping.

Is there any easy fix to get rid of the high max latency spikes that sometimes occur?

[tom.k.cook]

With what settings did you run cyclictest?

[meki_RPI]

i tried to add kernel=zImage-3.8.13-rt11 to /boot/config.txt (there are no #kernel.... in the file)
but RPi can not boot. Can someone please give me an advise, How can i correctly change/add the row kernel=....?

i also have got these message during booting of new SD-Card:
drivers/rtc/hctosys.c: unabile to open rtc device (rtc0)
Waiting ofr root device /dev/mmcblk0p2...

best regards