CONFIG_PREEMPT_RT on Raspberry Pi

[mung]

With what settings did you run cyclictest?

Sorry for delay in reply.

I basically tried the following:
boot, login, startx, open terminal, type 'cyclictest -t1 -p 80 -n -i 10000 -l 10000'
Cyclictest runs giving reasonable 'max' results, I double click the midori icon, after maybe 3 seconds of midori loading the cyclictest max value shoots up to ~4000 or 5000, the average stays about the same at 60, so it must be a single spike?

I really am just doing what it suggests on the rt wiki : https://rt.wiki.kernel.org/index.php/Cyclictest

I assume its the SD card or network with some NMI?

Obviously I am guessing with a very low level of understanding here, I only patched and compiled, I am not any sort of realtime kernel expert.

This seems much the same on 3.10 or 3.12 kernels, not tried on 3.6 or 3.8, but was much better (worst max spike about 700) on 3.2.27 but that did not have some of the firmware drivers compiled so no access to the vc(don't know if that means there is some vc stuff going on behind the scenes when midori loads, but seems unlikely?)

Any hints or tips gratefully received.

[mung]

i tried to add kernel=zImage-3.8.13-rt11 to /boot/config.txt (there are no #kernel.... in the file)
but RPi can not boot. Can someone please give me an advise, How can i correctly change/add the row kernel=....?

i also have got these message during booting of new SD-Card:
drivers/rtc/hctosys.c: unabile to open rtc device (rtc0)
Waiting ofr root device /dev/mmcblk0p2...

best regards

1) I think you need to put the kernel= in /boot/cmdline.txt (I normally use a different method so am not sure)
2) I think your kernel will not work anyway you need to make sure you patch with the autostatic patch or llm=0 otherwise SD will deadlock, and if you have hctosys.c error you need to config your kernel better without that option in menuconfig step of kernel compile

p.s. autostatic patch will fail and you need to edit by hand using the .rej files to workout what needs to be changed in the source code (quite simple and should only take 5 mins once you know what you are doing).

[meki_RPI]

i tried to add kernel=zImage-3.8.13-rt11 to /boot/config.txt (there are no #kernel.... in the file)
but RPi can not boot. Can someone please give me an advise, How can i correctly change/add the row kernel=....?

i also have got these message during booting of new SD-Card:
drivers/rtc/hctosys.c: unabile to open rtc device (rtc0)
Waiting ofr root device /dev/mmcblk0p2...

best regards

1) I think you need to put the kernel= in /boot/cmdline.txt (I normally use a different method so am not sure)
2) I think your kernel will not work anyway you need to make sure you patch with the autostatic patch or llm=0 otherwise SD will deadlock, and if you have hctosys.c error you need to config your kernel better without that option in menuconfig step of kernel compile

p.s. autostatic patch will fail and you need to edit by hand using the .rej files to workout what needs to be changed in the source code (quite simple and should only take 5 mins once you know what you are doing).

Thanks for advise. I am newby in this case and can not really understand the "autostatic patch" and "llm=0 option".
I have only followed this thread step by step and applied the patch with patch-3.12.6-rt9.patch.bz2 for rpi-3.12.y.
Can you please explain me more properly or give me suggestion of information source where i can read about. Thanks a lot in advance for your help!!

[meki_RPI]

Hello,
I got the PREEMT RT Kernel but not really a satified result of latency.

with this test: cyclictest -p95 -n -l 1000000
# /dev/cpu_dma_latency set to 0us
policy: fifo: loadavg: 0.56 0.42 0.31 2/174 2480
T: 0 ( 2480) P:95 I:1000 C: 18375 Min: 14 Act: 54 Avg: 42 Max: 3529

The max latency is much more than i ran RPi without RT-Kernel.
Can someone explain it please.
best regards

[mung]

Hello,
I got the PREEMT RT Kernel but not really a satified result of latency.

with this test: cyclictest -p95 -n -l 1000000
# /dev/cpu_dma_latency set to 0us
policy: fifo: loadavg: 0.56 0.42 0.31 2/174 2480
T: 0 ( 2480) P:95 I:1000 C: 18375 Min: 14 Act: 54 Avg: 42 Max: 3529

The max latency is much more than i ran RPi without RT-Kernel.
Can someone explain it please.
best regards

I am also a PREEMPT_RT newbie so my comments are possibly not correct, but are you sure you enabled 'full preemptive scheduler' for the kernel in menuconfig before compile?

Also there are option to use different types of scheduling alegorythm NOOP, CFQ, DEADLINE etc and I am really not sure about any of this stuff, but it seems the stock rpi kernel has a very good scheduling granularity because there are many IO tasks that are handled by the processor rather than a subsystem with DMA.

It could be possible that giving a realtime task too high a priority stops more important low level processes from creating data that is required by the realtime high priority thread (sort of like deadlock but without the lock)

From what I have read you are running the cyclictest with too high a priority (I think I read -p80 is highest should be used?)

If you have not applied the autostatic patch correctly your kernel will not boot anyways(with full preemt) as the mmc card causes a deadlock (unless the rpi kernel source has been patched already and autostatic is no longer required?), so if you are booting the kernel without the autostatic patch then that means you do not have the full preemptive scheduling anyways.

I am really not sure if what I am doing with cyclictest myself is correct so my advise is possibly not worth much?

As far as I am aware cyclictest should be run while the system is doing the normal type of tasks that would be expected for the realtime application you are going to run on the system, and will show the maximum latency which can be expected. In my case I run cyclictest from an lxterminal promt in X and then run linuxcnc or other programs to see what happens to latency values.

[Markus989]

Hi,
I'm also not a linuxkernel expert, but i have one idea with which the latency problem might be solved.
Without the sd-card-patch we don't have the latencyproblem but we cannot boot from sd-card.
So, is it possible to detect somehow inside our sd driver wether the kernel is booted or not??
If yes, we simply can insert a if-else in the sd-driver which executes the CONFIG_PREEMPT_RT_FULL-Makro only if kernel has not booted yet, and when the kernel is up the sd-card-driver has simply no effect and won't cause those latencies.
Is there a way to figure out wether the kernel is booted???

[kansky]

Seems as if someone claims good preempt_rt latency on RaspberryPi. See here:

http://www.spinics.net/lists/linux-rt-u ... 11200.html

[mung]

Seems as if someone claims good preempt_rt latency on RaspberryPi. See here:

http://www.spinics.net/lists/linux-rt-u ... 11200.html

I may checkout what they are doing, but I tried the most recent rt patch from kernel.org on 3.12.y from the rpi kernel git repo about 3 weeks ago and it was absolutly appalling latency 80000us or more, I assumed it was cause of the new dma stuff for the X server or the fiq stuff for the mmc (only a guess as always I never go further than a simple compile and cyclictest).

Also another thread here says 3.2.52 kernel (I assume mainline from kernel.org) they get max latency ~100us (iirc) I commented on that thread also but no reply: http://www.raspberrypi.org/phpBB3/viewt ... 37&t=67554

Best results I ever got were with an early 3.2.27 kernel that would give ~300us max latency but there were some keyboard problems and possibly mmc corruption (though this may have been due to overclocking @ 900, only happened once after a few hours using X).

Average latency always seems pretty good on most patched kernels ~50us but the max can vary wildly due to all sorts of simple things like swutching terminals, using network, starting X, pluggin or unpluggin usb devices.


I have still not got further into working out how to profile and trace RT latencys, maybe oneday when I am very bored I will read all the documentations.

[mung]

Yeah it looks quite amazing from the graph, I will defintly have a look at this very soon, they have nice graphs and the full kernel .config file.

I hope it actually works for me.

https://www.osadl.org/?id=1663 nice graph and details.
https://www.osadl.org/Profile-of-system ... 7s3.0.html the gritty details with a long list of .config and off tree patches.


cheers for the heads up kansky, I really hope this works with X and networking and vchiq and usb

[mung]

Not sure what I did wrong, but following the instructions exactly as stated on the osdl site does not create a working kernel. All the downloads were done using the quilt script at the bottom of the page linked above.
The kernel .config did not work and threw a load of undefined option that needed to be completed. I pressed enter repeatedly to take the default options then compiled using the gcc-linaro-arm-linux-gnueabihf-raspbian compiler on a ubuntu 12.04 image running in vmware.

This normally produces working kernels for rpi using the rpi git kernels.

I will actually try reading the info and selecting another .config using menuconfig tomorrow, maybe I will get it working then?.

[Zaine.RPi]

Hello,

I'm trying to modify a Raspbian kernel via PREEMPT_RT patch. I use cross compiler on Debian, but when compiling this error occurs:

Code: Select all

make -j1 ARCH=arm CROSS_COMPILE=/home/rpi/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian/bin/arm-linux-gnueabihf- Image

Code: Select all

drivers/usb/host/dwc_otg/dwc_otg_hcd_linux.c: In function 'hcd_init':
drivers/usb/host/dwc_otg/dwc_otg_hcd_linux.c:461: error: 'INTERRUPT_VC_USB' undeclared (first use in this function)
drivers/usb/host/dwc_otg/dwc_otg_hcd_linux.c:461: error: (Each undeclared identifier is reported only once
drivers/usb/host/dwc_otg/dwc_otg_hcd_linux.c:461: error: for each function it appears in.)
make[4]: *** [drivers/usb/host/dwc_otg/dwc_otg_hcd_linux.o] Error 1
make[3]: *** [drivers/usb/host/dwc_otg] Error 2
make[2]: *** [drivers/usb/host] Error 2
make[1]: *** [drivers/usb] Error 2
make: *** [drivers] Error 2

Please help, thank you.

[mung]

Code: Select all

drivers/usb/host/dwc_otg/dwc_otg_hcd_linux.c: In function 'hcd_init':
drivers/usb/host/dwc_otg/dwc_otg_hcd_linux.c:461: error: 'INTERRUPT_VC_USB' undeclared (first use in this function)

Please help, thank you.[/quote]
This sort of question is almost useless, you really need to provide lots more information!
You should give a few details about your experience in compiling kernels (newbie,done it a few times, or expert)
Which kernel you use(where you download it from?), which patch you used, which platform you compile on, what kernel config, which instructions are you following?

I doubt I can help much as I am quite a newbie, but I will tell you what I would do.
First try compiling the kernel without the RT patch, does it work??
If it does not then you need to sort out your .config, if it does work
Try using the config from a running raspbian system using cat /proc/config.gz >> .config

If you get a working vanilla kernel, do a make clean, then tar.gz the source tree, then apply the patch you want to use.
Once the patch is applied check all the .rej files (find ./ -name *.rej -print), workout what you need to fix by hand if necessary.

Then run a make menuconfig and check all the options, there still maybe config problems as it seems some menuconfig options are not added so you may get questions at the command line (probably best to use the defaults?)

My guess is if you have done everything correctly as far as patching then the problem is a missed .config option that the USB system depends on. I have no idea really, you probably best reading the code in the relevant source files and reading the .config and the menu help for all options related to USB when doing a menuconfig?

I should say that as far as I have found the patch never applies clean, and its possible the problems I seem to have could be due to my incorrectly manual patching (I think required addition of spinlock_irqsave, maybe I should not have manually patched?)

The realtime performance sucks if you do get it to compile and boot (according to cyclictest max latency spikes can be huge, don't know why?) average latency does seem fair most of the time though.

As far as I am aware the 3.2.27 still seems to have best lowest 'max latency' there can be problems with keypresses on usb ketboards and other stuff I think I already said in this thread.

Of course there is the possibility that your error is something new due to the code gods of the raspberry kernel trying to fix RT_PREEMPT performance, maybe you just need to get the right config options?

[Zaine.RPi]

Ok, sorry I am a beginner in compiling Linux kernels.

I again studied the information on compiling the kernels and I moved on. I managed to create kernel.img (patched by PREEMPT_RT) and copy it to the SD card. But when I run Raspberry Pi , the Raspbian freezes.

My procedure:

Code: Select all

mkdir rpi
cd rpi
git clone https://github.com/raspberrypi/linux.git
git clone https://github.com/raspberrypi/tools.git
cd linux
cp arch/arm/configs/bcmrpi_cutdown_defconfig .config
make ARCH=arm menuconfig

=> Linux/arm 3.10.32 Kernel Configuration

Code: Select all

make ARCH=arm CROSS_COMPILE=/home/rpi/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian/bin/arm-linux-gnueabihf- Image

=> Kernel: arch/arm/boot/Image is ready (without PREEMPT_RT patch)

Code: Select all

wget https://www.kernel.org/pub/linux/kernel/projects/rt/3.10/patch-3.10.32-rt30.patch.gz
gunzip patch-3.10.32-rt30.patch.gz
patch -p1 < patch-3.10.32-rt30.patch
./ -name *.rej -print

=> ./drivers/misc/Makefile.rej
Fixed (added line "obj-$(CONFIG_HWLAT_DETECTOR) += hwlat_detector.o" to Makefile)

Code: Select all

make ARCH=arm menuconfig

=> Only changed to "Fully Preemptible Kernel (RT)"

Code: Select all

make ARCH=arm CROSS_COMPILE=/home/rpi/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian/bin/arm-linux-gnueabihf- Image

=> Kernel: arch/arm/boot/Image is ready

Code: Select all

make ARCH=arm CROSS_COMPILE=/home/rpi/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian/bin/arm-linux-gnueabihf- modules
make ARCH=arm CROSS_COMPILE=/home/rpi/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian/bin/arm-linux-gnueabihf- modules_install

=> DEPMOD 3.10.32-rt30+

Code: Select all

cd ..
cd tools/mkimage/
python imagetool-uncompressed.py /home/rpi/linux/arch/arm/boot/Image
cp kernel.img /media/notebook/boot/
nano /media/notebook/boot/cmdline.txt

=> added line "sdhci_bcm2708.enable_llm=0"

When I run Raspberry Pi, Raspbian freezes during this time:
http://s30.postimg.org/gg43lnhf5/WP_20140307_008.jpg

[fj.rojas]

Thanks to this post, some months ago I could successfully install the rt-11 with the 3.8.13 kernel patch in RPi. I attach the procedure I followed with the required sdhci.c file. Thank you!

[Zaine.RPi]

Thanks to this post, some months ago I could successfully install the rt-11 with the 3.8.13 kernel patch in RPi. I attach the procedure I followed with the required sdhci.c file. Thank you!

I proceeded exactly according to your procedures, but when I turn on the Raspberry Pi, the screen only displays colored stripes (really exactly according to the attached file). Do you know where is the problem?

http://postimg.org/image/wdp0ipnd7/

[mung]

I proceeded exactly according to your procedures, but when I turn on the Raspberry Pi, the screen only displays colored stripes (really exactly according to the attached file). Do you know where is the problem?
http://postimg.org/image/wdp0ipnd7/

I think that screen means that the blob has stopped and the linux kernel has not initialised, or maybe not?

Are you sure the correct kernel configs is made in the /boot/ partition? what is the kernel name what is the name given in cmdline.txt and what does your config.txt file look like?

[Zaine.RPi]

Ok, SD card contents:

Code: Select all

/media/notebook/boot$ ls -l
total 21936
-rw-r--r-- 1 notebook notebook   17824 Jan  7 18:50 bootcode.bin
-rw-r--r-- 1 notebook notebook     169 Mar  9 17:30 cmdline.txt
-rw-r--r-- 1 notebook notebook     142 Jan  7 20:13 cmdline.txt~
-rw-r--r-- 1 notebook notebook    1206 Mar  9 17:31 config.txt
-rw-r--r-- 1 notebook notebook    1206 Mar  9 17:30 config.txt~
-rw-r--r-- 1 notebook notebook    2068 Jan  7 18:50 fixup_cd.dat
-rw-r--r-- 1 notebook notebook    5783 Jan  7 18:50 fixup.dat
-rw-r--r-- 1 notebook notebook    8829 Jan  7 18:50 fixup_x.dat
-rw-r--r-- 1 notebook notebook     137 Jan  7 22:34 issue.txt
-rw-r--r-- 1 notebook notebook 9789752 Jan  7 18:50 kernel_emergency.img
-rw-r--r-- 1 notebook notebook 3115216 Jan  7 18:50 kernel.img
-rw-r--r-- 1 notebook notebook   18974 Sep 25 21:57 LICENSE.oracle
-rw-r--r-- 1 notebook notebook  480216 Jan  7 18:50 start_cd.elf
-rw-r--r-- 1 notebook notebook 2514136 Jan  7 18:50 start.elf
-rw-r--r-- 1 notebook notebook 3495816 Jan  7 18:50 start_x.elf
-rw-r--r-- 1 notebook notebook 2921616 Mar  9 17:25 zimage-3.8.13-rt11

Files "cmdline.txt" and "config.txt" are here:

[mung]

-rw-r--r-- 1 notebook notebook 2921616 Mar 9 17:25 zimage-3.8.13-rt11

I have no idea really, have you tried login on serial console?

If you cannot login over serial then you have not configured the kernel correctly and it is just not loading, if you can login on serial then there is problem with the framebuffer initialisation.

Do not assume I know what I am talking about, I am just guessing, I have no real understanding of how the kernel works.

[fj.rojas]

I forgot to attach the .config file. My mistake.

Anyway, here is the .zip with all files (.config included).

Later I'll try to follow this guide again. If any modifications arise, I'll be posting them here.

See you!

[Zaine.RPi]

-rw-r--r-- 1 notebook notebook 2921616 Mar 9 17:25 zimage-3.8.13-rt11

I have no idea really, have you tried login on serial console?

If you cannot login over serial then you have not configured the kernel correctly and it is just not loading, if you can login on serial then there is problem with the framebuffer initialisation.

Do not assume I know what I am talking about, I am just guessing, I have no real understanding of how the kernel works.

Success!

I used my procedure (described above). The problem was that I had not copied the folder "raspi-modules" to the SD card (as described in the user fj.rojas). Now everything is OK.

Thank you for your help.

[Zaine.RPi]

Hello,
I had one more question. Why is it necessary to turn off low-latency mode (add line "sdhci_bcm2708.enable_llm=0")?

[minhnt152]

Dear fj.rojas,

I'm new to linux,

So i dont understand much about the procedure.
Is your procedure run on an ubuntu pc then it will create a RT image for RPi?

Thanks

[mung]

Dear fj.rojas,

I'm new to linux,

So i dont understand much about the procedure.
Is your procedure run on an ubuntu pc then it will create a RT image for RPi?

Thanks

If you are a newbie maybe better you say what you are trying to do and why you think you need a real time kernel, it may mean you get more helpful replies.

To try and answer your question, I have not tried the fj.rojas zip file but thought I would have a look at it as I am back checking RT_PREEMPT again. From the looks of things the script uses the cross compiler (export CROSS_COMPILE=~/Desktop/RPi_RT/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian/bin/arm-linux-gnueabihf-
) and downloads the 3.8.y branch of the rpi kernel. That means you would be compiling on an x86 system, ubuntu, debian, or some other recent linux x86 distro should all work, of course I don't know for certain as I have not run the script.

I actually have not really used that branch much (as far as I can remember I may have compiled it, but not found good results in cyclictest or had problems with usb).

the script grabs the kernel source and cross compiler tools with:
git clone https://github.com/raspberrypi/linux.git
git clone https://github.com/raspberrypi/tools.git

I get the feeling a better command would be to get the 3.12 branch only?

git clone -b rpi-3.12.y --depth=1 https://github.com/raspberrypi/linux.git

I think the sdhci.c file is probably okay to copy into the 3.12.y tree as not much seems to change in that file between releases, thought I have not actually looked at it in the zip file.

I normally use the same cross compiler tools from git on an ubuntu system installed from a linuxcnc livecd onto a vmware virtual machine.

I am posting as something spurred me to think there were improvements in the rpi kernel, and I actually grabbed the latest 3.12.y kernel from git last night and it compiled and booted quite well once patched with the RT patch, and seemed to have great cyclictest results and vastly improved usb though still seems to have some problems with keyboard inputs when highly loaded.

I was running X with a few apps and cyclictest command within an lxterminal as suggested in the RT_PREEMPT faq (I think 'sudo cyclictest -p 80 t5 -n') I think max latency was less than 130 with average at a constant 25 and I tried plugging in/unplugging a few usb devices and dragging windows etc, left it running for an hour, but did not try running a browser as I did not have the rpi on the network.

I think current 3.12 branch may give the best results I have seen so far(unfortunatly I have not really documented figures as I have just been trying to get a feel for weather RT_PREEMPT is ready for my purposes), but I will have to try test some networking apps tonight.

I am hoping this could mean I am going to put together another linuxcnc SD card boot image soon(if/when I get enough time).

[twobob]

I have a Linux RASPBERRYPI 3.12.20+ #687 PREEMPT Fri May 30 16:39:11 BST 2014 armv6l GNU/Linux

I see vague reference to kernel-rt-3.12.20-0.rt30.1 on google...

Which is the right set of patches to apply to the kernel at the moment?

Partial noob, done some kernel compiles/x-compiles/patching before on other embedded devices.

Thanks.

Or is there an image out there somewhere that someone is kind enough to share? Thanks again.

[mung]

I have a Linux RASPBERRYPI 3.12.20+ #687 PREEMPT Fri May 30 16:39:11 BST 2014 armv6l GNU/Linux

I see vague reference to kernel-rt-3.12.20-0.rt30.1 on google...

Which is the right set of patches to apply to the kernel at the moment?

Partial noob, done some kernel compiles/x-compiles/patching before on other embedded devices.

Thanks.

Or is there an image out there somewhere that someone is kind enough to share? Thanks again.

I don't know of any images available but maybe google will show something.

I think current rpi kernels should patch with very little hacking but unfortunately I cannot say for certain as I have only patched and hacked based on prior methods I used, which after a little confusion I think may have caused some problems.

I have tried 3.12 and 3.14 by running the standard RT patch then copying across sdhci.c to the source tree but some of the code has changed in more recent version so I think a vanilla patch from kernel.org may work okay?

I don't have time to look any further but will probably make some sort of linuxcnc SD image in the next couple of months as RT_PREEMPT is looking very workable. No load figures show 120us max latency and 30 average, running linuxcnc with a fair amount of loading and realtime threads give max latency about 600us. The usb problems seem to be fixed as far as my system is concerned.

I am not really an expert but I don't know where else you can look for help, you should give a compile a try and post what errors or trouble you get then maybe someone will offer help.