CONFIG_PREEMPT_RT on Raspberry Pi


89 posts   Page 1 of 4   1, 2, 3, 4
by tom.k.cook » Tue Apr 09, 2013 9:28 am
Edit: Updated to now boot from the SD card instead of USB.
After months of frustration and distractions (more distractions than frustrations to be honest), here's a quick outline of how to get a working real-time kernel working on the Raspberry Pi:
  • Get the 3.8.13 kernel source from git:
    Code: Select all
    git clone https://github.com/raspberrypi/linux.git
    cd linux
    git checkout rpi-3.8.y
  • Get the toolchain from git:
    Code: Select all
    git clone https://github.com/raspberrypi/tools.git
  • Download the 3.8.13-rt11 patch from https://www.kernel.org/pub/linux/kernel ... ts/rt/3.8/
  • Patch the source:
    Code: Select all
    bzcat patch-3.8.13-rt11.patch.bz2 | patch -p1
  • Start with the stock .config (or get one from a running RasPi). Suppose you've got /proc/config.gz from a running raspi:
    Code: Select all
    zcat config.gz > .config
  • Set up your environment:
    Code: Select all
    export ARCH=arm
    export CROSS_COMPILE=/home/me/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian/bin/arm-linux-gnueabihf-
    export INSTALL_MOD_PATH=/home/me/raspi-image
  • Configure the kernel:
    Code: Select all
    make gconfig
  • Enable PREEMPT_RT and any other RT features you want
  • Save your configuration
  • Build the kernel - add '-j n' where n is the number of cores available + 1 to speed it up:
    Code: Select all
    make bzImage
    make modules
    make modules_install
  • Take the stock Raspbian image and install it on the SD card as per normal.
  • Copy linux/arch/arm/boot/zImage to /boot/zImage-3.8.13-rt11 on the RPI.
  • Copy raspi-image/lib/modules/3.8.13-rt11+ to /lib/modules on the RPI taking care to preserve symlinks - use cp -rp.
  • On the RPI, edit /boot/config.txt. Comment out any line that starts with 'kernel=' and add this line:
    Code: Select all
    kernel=zImage-3.8.13-rt11
  • On the RPI, edit /boot/cmdline.txt. Add this option:
    Code: Select all
    sdhci_bcm2708.enable_llm=0

Done. Your RPi should boot a realtime-capable kernel.

TODOs:
  • Debug the MMC driver so it works in RT with enable_llm=1.

Note that if you want to use ftrace on the RPI, you need to do this:
Code: Select all
echo dwc_otg_hcd_handle_fiq > /sys/kernel/debug/tracing/set_ftrace_filter

before you enable any tracer. Otherwise the USB driver will crash the kernel.

BTW here is the current performance on my RPI:
Code: Select all
root@raspberrypi:/home/pi/rt-tests# ./cyclictest -p95 -n -l 1000000
# /dev/cpu_dma_latency set to 0us
policy: fifo: loadavg: 0.45 0.42 0.32 1/143 4572

T: 0 ( 4523) P:95 I:1000 C: 120110 Min:     20 Act:   44 Avg:   41 Max:  109
Last edited by tom.k.cook on Wed Jun 26, 2013 3:23 pm, edited 1 time in total.
Posts: 43
Joined: Fri Jun 22, 2012 8:51 am
by AutoStatic » Wed Apr 10, 2013 11:11 am
tom.k.cook wrote:After months of frustration and distractions (more distractions than frustrations to be honest), here's a quick outline of how to get a working real-time kernel working on the Raspberry Pi:
Awesome! I've already looked into this too but couldn't get it to boot. I'm now also seeing your mails on linux-rt-users ;)
tom.k.cook wrote:TODOs:
Debug the MMC driver so it works in RT. First step is to compile it as a module and then see what we can see.
That should be fairly trivial, I'm going to compile a RT kernel shortly and I'll see if I can compile the mmc drivers as modules.
tom.k.cook wrote:Get a hardfp toolchain and build using that instead of the softfp toolchain supplied with Ubuntu.
This is a bit unclear, care to elaborate?
RPi and real-time, low-latency audio: http://wiki.linuxaudio.org/wiki/raspberrypi
RPi audio repository: http://rpi.autostatic.com/
RPi and audio blog: https://autostatic.com/tag/raspberrypi/
User avatar
Posts: 326
Joined: Tue Jan 01, 2013 10:27 pm
by tom.k.cook » Wed Apr 10, 2013 11:25 am
AutoStatic wrote:
tom.k.cook wrote:Get a hardfp toolchain and build using that instead of the softfp toolchain supplied with Ubuntu.
This is a bit unclear, care to elaborate?


I'm using the gcc-arm-linux-gnueabi package provided with Ubuntu 12.04, which does not support the hardware floating-point unit present on the RPi. This also means I have to use the softfp firmware, which means that all floating-point operations run on it will be implemented in software instead of using the hardware FPU. To fix it, I just need to download the custom toolchain from git (git://github.com/raspberrypi/tools.git), rebuild using it and swap to the hardfp firmware, but I haven't got around to doing it yet.
Posts: 43
Joined: Fri Jun 22, 2012 8:51 am
by AutoStatic » Wed Apr 10, 2013 11:29 am
tom.k.cook wrote:Patch the source:
Code: Select all
bzcat patch-3.6.11-rt31.patch.bz2 | patch -p1
Should be:
Code: Select all
bzcat patch-3.6.11-rt31.patch.bz2 | patch -p1 -d linux
or
Code: Select all
cd linux
bzcat ../patch-3.6.11-rt31.patch.bz2 | patch -p1
RPi and real-time, low-latency audio: http://wiki.linuxaudio.org/wiki/raspberrypi
RPi audio repository: http://rpi.autostatic.com/
RPi and audio blog: https://autostatic.com/tag/raspberrypi/
User avatar
Posts: 326
Joined: Tue Jan 01, 2013 10:27 pm
by tom.k.cook » Wed Apr 10, 2013 11:33 am
AutoStatic wrote:
tom.k.cook wrote:Debug the MMC driver so it works in RT. First step is to compile it as a module and then see what we can see.
That should be fairly trivial, I'm going to compile a RT kernel shortly and I'll see if I can compile the mmc drivers as modules.


Yes, building as a module should be pretty straightforward, and debugging from there not *too* bad - this should at least mean the SysRq tools will be available. I think the problem is around sdhci.c lines 1282-1284:

Code: Select all
      sdhci_spin_enable_schedule(host);
      mdelay(1);
      sdhci_spin_disable_schedule(host);


But I'm no expert. I was planning to try removing the first and third lines, leaving mdelay(1) with preemption disabled. It'll make performance worse, but hopefully fix the immediate crash.
Posts: 43
Joined: Fri Jun 22, 2012 8:51 am
by jamesh » Wed Apr 10, 2013 11:36 am
tom.k.cook wrote:
AutoStatic wrote:
tom.k.cook wrote:Get a hardfp toolchain and build using that instead of the softfp toolchain supplied with Ubuntu.
This is a bit unclear, care to elaborate?


I'm using the gcc-arm-linux-gnueabi package provided with Ubuntu 12.04, which does not support the hardware floating-point unit present on the RPi. This also means I have to use the softfp firmware, which means that all floating-point operations run on it will be implemented in software instead of using the hardware FPU. To fix it, I just need to download the custom toolchain from git (git://github.com/raspberrypi/tools.git), rebuild using it and swap to the hardfp firmware, but I haven't got around to doing it yet.


Us the linaro toolchain from the github tools - it produces slightly faster code in our experience.
Volunteer at the Raspberry Pi Foundation, helper at Picademy September, October, November 2014.
Forum Moderator
Forum Moderator
Posts: 15801
Joined: Sat Jul 30, 2011 7:41 pm
by tom.k.cook » Wed Apr 10, 2013 11:42 am
jamesh wrote:Us the linaro toolchain from the github tools - it produces slightly faster code in our experience.


Is that the gcc-linaro-arm-linux-gnueabihf-raspbian in git://github.com/raspberrypi/tools/arm-bcm2708?

If so, does the warning that it targets armv7-a by default apply?
Posts: 43
Joined: Fri Jun 22, 2012 8:51 am
by AutoStatic » Wed Apr 10, 2013 11:55 am
tom.k.cook wrote:I'm using the gcc-arm-linux-gnueabi package provided with Ubuntu 12.04, which does not support the hardware floating-point unit present on the RPi. This also means I have to use the softfp firmware, which means that all floating-point operations run on it will be implemented in software instead of using the hardware FPU. To fix it, I just need to download the custom toolchain from git (git://github.com/raspberrypi/tools.git), rebuild using it and swap to the hardfp firmware, but I haven't got around to doing it yet.
Ah ok, thought so already. Thanks for the explanation. I'm already using the custom toolchain so I can just go on and compile.
RPi and real-time, low-latency audio: http://wiki.linuxaudio.org/wiki/raspberrypi
RPi audio repository: http://rpi.autostatic.com/
RPi and audio blog: https://autostatic.com/tag/raspberrypi/
User avatar
Posts: 326
Joined: Tue Jan 01, 2013 10:27 pm
by tom.k.cook » Wed Apr 10, 2013 12:00 pm
AutoStatic wrote:Ah ok, thought so already. Thanks for the explanation. I'm already using the custom toolchain so I can just go on and compile.


Yep, just make sure you get the right firmware.
Posts: 43
Joined: Fri Jun 22, 2012 8:51 am
by AutoStatic » Wed Apr 10, 2013 2:28 pm
Building the mmc drivers as modules failed:
Code: Select all
  Building modules, stage 2.
  MODPOST 1145 modules
ERROR: "enable_llm" [drivers/mmc/host/sdhci.ko] undefined!
ERROR: "sdhci_spin_unlock" [drivers/mmc/host/sdhci-bcm2708.ko] undefined!
ERROR: "sdhci_spin_lock" [drivers/mmc/host/sdhci-bcm2708.ko] undefined!
ERROR: "frc_clock_ticks32" [drivers/mmc/host/sdhci-bcm2708.ko] undefined!
make[1]: *** [__modpost] Error 1
make: *** [modules] Error 2
So I'm going to try to compile without the mmc stack.
RPi and real-time, low-latency audio: http://wiki.linuxaudio.org/wiki/raspberrypi
RPi audio repository: http://rpi.autostatic.com/
RPi and audio blog: https://autostatic.com/tag/raspberrypi/
User avatar
Posts: 326
Joined: Tue Jan 01, 2013 10:27 pm
by tom.k.cook » Wed Apr 10, 2013 2:56 pm
Without is very easy.
Posts: 43
Joined: Fri Jun 22, 2012 8:51 am
by tom.k.cook » Wed Apr 10, 2013 3:42 pm
AutoStatic wrote:Building the mmc drivers as modules failed:
Code: Select all
  Building modules, stage 2.
  MODPOST 1145 modules
ERROR: "enable_llm" [drivers/mmc/host/sdhci.ko] undefined!
ERROR: "sdhci_spin_unlock" [drivers/mmc/host/sdhci-bcm2708.ko] undefined!
ERROR: "sdhci_spin_lock" [drivers/mmc/host/sdhci-bcm2708.ko] undefined!
ERROR: "frc_clock_ticks32" [drivers/mmc/host/sdhci-bcm2708.ko] undefined!
make[1]: *** [__modpost] Error 1
make: *** [modules] Error 2
So I'm going to try to compile without the mmc stack.

Again, I'm not at all expert here, but I think you can fix these by moving the declaration of enable_llm to sdhci.c and then adding
Code: Select all
EXPORT_SYMBOL_GPL(enable_llm);
EXPORT_SYMBOL_GPL(sdhci_spin_lock);
EXPORT_SYMBOL_GPL(sdhci_spin_unlock);
to sdhci.c and
Code: Select all
EXPORT_SYMBOL_GPL(frc_clock_ticks32);
to bcm2708.c.
Posts: 43
Joined: Fri Jun 22, 2012 8:51 am
by AutoStatic » Thu Apr 11, 2013 6:27 pm
Yay, booted!
Code: Select all
pi@jj-rpi ~ $ uname -a
Linux jj-rpi 3.6.11-rt31 #1 PREEMPT RT Thu Apr 11 15:42:34 CEST 2013 armv6l GNU/Linux
And this an extra bonus I wasn't prepared for:
Code: Select all
pi@jj-rpi ~ $ ps -eLo pid,cls,rtprio,pri,nice,cmd | grep -i [i]rq
    3  FF      1  41   - [ksoftirqd/0]
   13  FF     50  90   - [irq/65-ARM Mail]
   33  FF     50  90   - [irq/66-VCHIQ do]
   36  FF     50  90   - [irq/32-dwc_otg]
   37  FF     50  90   - [irq/32-dwc_otg_]
   38  FF     50  90   - [irq/32-dwc_otg_]
 2127  FF     50  90   - [irq/83-uart-pl0]
Threaded IRQs! Thanks tom.k.cook!
RPi and real-time, low-latency audio: http://wiki.linuxaudio.org/wiki/raspberrypi
RPi audio repository: http://rpi.autostatic.com/
RPi and audio blog: https://autostatic.com/tag/raspberrypi/
User avatar
Posts: 326
Joined: Tue Jan 01, 2013 10:27 pm
by tom.k.cook » Thu Apr 11, 2013 6:38 pm
Yeah, I got sdhci building as a module and it loads, but doesn't seem to detect the sd card. And the cyclictest latencies are pretty dire...
Posts: 43
Joined: Fri Jun 22, 2012 8:51 am
by AutoStatic » Thu Apr 11, 2013 7:07 pm
Haven't ran cyclictest yet, just some quick tests with audio. RT on the RPi is not an improvement so far for audio purposes.
RPi and real-time, low-latency audio: http://wiki.linuxaudio.org/wiki/raspberrypi
RPi audio repository: http://rpi.autostatic.com/
RPi and audio blog: https://autostatic.com/tag/raspberrypi/
User avatar
Posts: 326
Joined: Tue Jan 01, 2013 10:27 pm
by tom.k.cook » Thu Apr 11, 2013 7:34 pm
Code: Select all
root@raspberrypi:/home/pi/rt-tests# ./cyclictest
# /dev/cpu_dma_latency set to 0us
policy: other/other: loadavg: 1.49 0.70 0.27 1/140 2048         

T: 0 ( 2048) P: 0 I:1000 C:  31789 Min:     57 Act:  762 Avg:  167 Max:    4963


I'm just building a kernel with some tracers in it to try figure out what causes it. As a first indication, it's a lot better over a serial console (below) than an ssh connection (above):

Code: Select all
# /dev/cpu_dma_latency set to 0us
policy: other/other: loadavg: 0.19 0.40 0.23 2/13

T: 0 ( 2088) P: 0 I:1000 C:   5869 Min:     54 Act:   66 Avg:   86 Max:    1183
Posts: 43
Joined: Fri Jun 22, 2012 8:51 am
by tom.k.cook » Thu Apr 11, 2013 7:55 pm
Though having some clue about how to run cyclictest certainly helps:

Code: Select all
pi@raspberrypi ~/rt-tests $ sudo ./cyclictest -p95 -m
# /dev/cpu_dma_latency set to 0us
policy: fifo: loadavg: 0.08 0.08 0.05 2/141 2148         

T: 0 ( 2148) P:95 I:1000 C:  11460 Min:    108 Act:  161 Avg:  148 Max:     529


Better, but still not brilliant.
Posts: 43
Joined: Fri Jun 22, 2012 8:51 am
by tom.k.cook » Thu Apr 11, 2013 9:01 pm
And, unfortunately, using cyclictest with either the wakeup or function tracers enabled (-W or -f or both) causes a kernel oops. It's too late to look into this tonight.
Posts: 43
Joined: Fri Jun 22, 2012 8:51 am
by elmood » Tue Apr 23, 2013 8:53 pm
Hi folks,

I wonder if anyone has made more progress with this, or might have a kernel with PREEMPT_RT turned on available for others to try?

I've been building my own kernel, but so far it hangs when booting and I'm not sure what's wrong yet. I'm trying again with the MMC stuff turned off as someone mentioned, although I don't understand the significance of this, or what the effect of doing this will be... doesn't it mean that the SD card won't work?

I'd be very interested to know if any progress has been made, and if there would be some step-by-step instructions on making a valid build.


Thanks!
Posts: 3
Joined: Tue Apr 23, 2013 8:48 pm
by tom.k.cook » Tue Apr 23, 2013 9:17 pm
Yes, you are right, the sd card won't work. You need to put the root fs on something else - either usb or nfs. See my separate thread on how to get a booting kernel with PREEMPT_RT_FULL.

Note that it's a booting kernel with the relevant configuration options enabled - the performance is far from what most would consider real-time. If you know anything about debugging kernel latencies, your help would be much appreciated.
Posts: 43
Joined: Fri Jun 22, 2012 8:51 am
by tom.k.cook » Thu Apr 25, 2013 8:53 am
tom.k.cook wrote:Yes, you are right, the sd card won't work. You need to put the root fs on something else - either usb or nfs. See my separate thread on how to get a booting kernel with PREEMPT_RT_FULL.

Note that it's a booting kernel with the relevant configuration options enabled - the performance is far from what most would consider real-time. If you know anything about debugging kernel latencies, your help would be much appreciated.


Please excuse my late-night ramblings posting from my phone - the thread I meant was this one. It will give you a booting RT kernel, though not one with any sort of RT performance.
Posts: 43
Joined: Fri Jun 22, 2012 8:51 am
by schorsch76 » Sun Apr 28, 2013 11:43 am
Hi,
i got the 3.6.11-rt31 Kernel running on my pi version B. The rootfs is on the usb stick and the boot partition is on the sd card.
I got the sdhci-bcm2708 compiled with the upper called patches, but i can not mount the sd card. How could i debug where the error sits? I can modprobe the sdhci-bcm2708 module and the sdhci module. Tried to restart udev, but no events to see /dev/mmc*

Bye
schorsch
User avatar
Posts: 20
Joined: Sun Apr 28, 2013 8:01 am
by tom.k.cook » Mon Apr 29, 2013 8:09 am
schorsch76 wrote:I got the sdhci-bcm2708 compiled with the upper called patches, but i can not mount the sd card. How could i debug where the error sits? I can modprobe the sdhci-bcm2708 module and the sdhci module. Tried to restart udev, but no events to see /dev/mmc*


That's where I'm up to as well. Next step is to rebuild the stock 3.6.11. kernel with sdhci-bcm2708 as a module and see whether that works - to see if the problem is in the generic module code or the RT version of it.

I suspect the latter - I think the module is probably hanging up while probing for an SD card, just as it does when the sdhci driver is built in to the kernel.
Posts: 43
Joined: Fri Jun 22, 2012 8:51 am
by mung » Thu May 02, 2013 6:40 pm
Cool to see people are working on RT_PREEMPT for the 3.6 kernel, I got it booting from SD on the previous 3.2.27 kernel sources then had an SD card crash that destroyed the root filesystem and the patched sources I used (maybe that suggests something was wrong with the kernel :lol: ). I got the kernel off the fat partition and am still using it occasionally for testing, have not got around to trying to make another compilation yet though. Strange thing is the saved kernel has not destroyed another filesystem yet in other tests.

Will be good if someone gets 3.6 kernel to to boot from SD without crashing, I may try out the instruction this weekend see what happens.
Posts: 506
Joined: Fri Nov 18, 2011 10:49 am
by aaa801 » Sun May 19, 2013 1:55 am
Trying a 3.8 kernel with the rt patches atm, my first usb stick died on me while writing the files :@
Will post results

Code: Select all
root@raspberrypi:~/rt-tests# sudo ./cyclictest -p95 -m
# /dev/cpu_dma_latency set to 0us
policy: fifo: loadavg: 0.14 0.19 0.24 1/146 5081

T: 0 ( 5081) P:95 I:1000 C:  20073 Min:     67 Act:  112 Avg:  104 Max:     651
Posts: 428
Joined: Mon Jun 04, 2012 9:06 pm
Location: Berkshire