PhilE
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 1761
Joined: Mon Sep 29, 2014 1:07 pm
Location: Cambridge

New RT (Real Time) kernel branch

Thu Mar 01, 2018 12:49 pm

The first topic in the Announcements forum is about the new Real Time Linux branch. Feel free to discuss it and ask questions here.

Heater
Posts: 9361
Joined: Tue Jul 17, 2012 3:02 pm

Re: New RT (Real Time) kernel branch

Thu Mar 01, 2018 4:59 pm

This is really cool.

Recently I have been experimenting with the "isolcpus" Linux command line parameter to keep Linux from scheduling stuff on a core, then running time critical code on the "unused" core.

With that set up, in one experiment, I managed to toggle a GPIO pin at 60Mhz or so from a tight software loop, with only the occasional 5us stutter as some unknown interrupt handler gets run on the core.

Is the RT kernel likely to be of any help in that situation?

PhilE
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 1761
Joined: Mon Sep 29, 2014 1:07 pm
Location: Cambridge

Re: New RT (Real Time) kernel branch

Thu Mar 01, 2018 5:07 pm

I'm not the best person to answer - I'm more of a facilitator for Tiejun - but I wouldn't have thought it would help unless your user thread from being interrupted. However, it could conceivably help the interruptible CPUs cope with the extra interrupt load.

Do let us know your findings if you do build your own RT kernel - the build instructions are here. Just remember to run "git checkout rpi-4.14.y-rt" before the first "make" command.

User avatar
karrika
Posts: 1041
Joined: Mon Oct 19, 2015 6:21 am
Location: Finland

Re: New RT (Real Time) kernel branch

Fri Mar 02, 2018 4:33 pm

Thanks!

This could make bi-directional DMX communication for DiscoHAT reality. Exciting!

Currently reading 512 bytes with no breaks at 250k using standard UART has not worked on the Pi. Sending is ok with the standard kernel. Perhaps this could allow us to build Raspberry Pi based RDM responders for theater light systems.

Heater
Posts: 9361
Joined: Tue Jul 17, 2012 3:02 pm

Re: New RT (Real Time) kernel branch

Sat Mar 03, 2018 7:33 am

karrika,

You might be interested in trying your DMX with the real-time tricks we tried in this thread, with success:

viewtopic.php?f=63&t=200793#p1251184

Basically if you have a multi-core Pi you can kick Linux off of one or more cores with the isolcpus kernel command line parameter "isolcpus". Just add something like "isolcpus=2,3" to your cmdline.txt file in /boot.

Now you can run your DMX thing in one of those free cores:

$ taskset -c 3 ./theDMXthing

You might want to also use "nice" to up it's priority, not sure if that is needed.

User avatar
karrika
Posts: 1041
Joined: Mon Oct 19, 2015 6:21 am
Location: Finland

Re: New RT (Real Time) kernel branch

Sat Mar 03, 2018 7:50 pm

Thanks for the link. I am using SPI for the 2811 LED snakes also but as the snake is updated all the time I have not seen the problems described on the thread.

To allocate a dedicated core to QLC+ sounds like the way to go. My lights run using Qt5 directly from the command line without a desktop. But it will take a while to re-design DiscoHAT before I can test this. For easier cabling I will use a chip to output 4 DMX channels with identical content. One DMX read channel. One 5V buffered SPI LED output. Plus 8 input GPIO buttons. All isolated and protected from surges of course.

zavorra
Posts: 7
Joined: Tue Mar 06, 2018 1:21 pm

Re: New RT (Real Time) kernel branch

Tue Mar 06, 2018 1:24 pm

These are great news!

How am I supposed to install on my pi?

I did apt-get update , apt-get upgrade, rpi-update but it didn't show any -rt kernel

PhilE
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 1761
Joined: Mon Sep 29, 2014 1:07 pm
Location: Cambridge

Re: New RT (Real Time) kernel branch

Tue Mar 06, 2018 2:43 pm

In particular, we have no plans to switch our kernel builds to use PREEMPT_RT or to ship them as alternative kernels.
This means that if you want to try it out you'll have to build it yourself. The general kernel-building instructions are here: https://www.raspberrypi.org/documentati ... uilding.md

You will need to amend the instructions slightly - where it says

Code: Select all

git clone --depth=1 https://github.com/raspberrypi/linux
you will need:

Code: Select all

git clone --depth=1 https://github.com/raspberrypi/linux -b rpi-4.14.y-rt

zavorra
Posts: 7
Joined: Tue Mar 06, 2018 1:21 pm

Re: New RT (Real Time) kernel branch

Tue Mar 06, 2018 8:41 pm

I tried to build and install it following those instructions, but then i got some kind of kernel panic at boot. I don't have the logs here, but I can remember the last message regarded mmc and interrupts.

I will try to reproduce it and post here

PhilE
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 1761
Joined: Mon Sep 29, 2014 1:07 pm
Location: Cambridge

Re: New RT (Real Time) kernel branch

Tue Mar 06, 2018 9:17 pm

Try with "dtoverlay=dwc2" in config.txt - I think there is a problem with the FIQ interrupt.

zavorra
Posts: 7
Joined: Tue Mar 06, 2018 1:21 pm

Re: New RT (Real Time) kernel branch

Wed Mar 07, 2018 8:30 am

thing seemed to get better, but after 6 minutes (347 seconds) I got
[ 347.337186] Disabling IRQ #59

BTW, I am powering the pi with a 2.1A power bank and occasionally get "Under-voltage detected ! (0x00050005)" on the console. Could it be related somehow to IRQ #59?
Last edited by zavorra on Wed Mar 07, 2018 8:34 am, edited 1 time in total.

PhilE
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 1761
Joined: Mon Sep 29, 2014 1:07 pm
Location: Cambridge

Re: New RT (Real Time) kernel branch

Wed Mar 07, 2018 8:33 am

That's not good - where there any other symptoms?

zavorra
Posts: 7
Joined: Tue Mar 06, 2018 1:21 pm

Re: New RT (Real Time) kernel branch

Wed Mar 07, 2018 8:34 am

I am powering the pi with a 2.1A power bank and occasionally get "Under-voltage detected ! (0x00050005)" on the console. Could it be related somehow to IRQ #59?

zavorra
Posts: 7
Joined: Tue Mar 06, 2018 1:21 pm

Re: New RT (Real Time) kernel branch

Wed Mar 07, 2018 8:43 am

This is what i did:
- installed 2017-11-29-raspbian-stretch-lite.zip, addded "enable_uart=1" in config.txt - BOOT OK

- rpi-update - BOOT OK

- install rt kernel - Disabling IRQ #59

- boot kernel7-backup.img - Disabling IRQ#59 (this is weird!)

- added "dtoverlay=dwc2" in config.txt , boot kernel7.img - Disabling IRQ #59

PhilE
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 1761
Joined: Mon Sep 29, 2014 1:07 pm
Location: Cambridge

Re: New RT (Real Time) kernel branch

Wed Mar 07, 2018 10:54 am

IRQ 59 is the AUX IRQ - the AUX peripherals (SPI1, SPI2, UART1) share an interrupt with a separate status register to identify the active sources, and for some reason the -rt kernel is seeing 100000 unhandled AUX interrupts. The Pi 3 serial port (on UART1) also behaves oddly - it appears to start OK, showing the kernel messages and a login prompt, but then at some point after that it stops responding and the interrupt count goes through the roof.

I think that something about the way the downstream AUX interrupt driver is implemented is interacting badly with the -rt kernel. The fact that the restoring the kernel hasn't solved the problem is odd - I can't explain that yet.

zavorra
Posts: 7
Joined: Tue Mar 06, 2018 1:21 pm

Re: New RT (Real Time) kernel branch

Wed Mar 07, 2018 1:02 pm

So if i disable uart and work with a monitor and a keyboard the issue should disappear? I will try it later.

Z

PhilE
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 1761
Joined: Mon Sep 29, 2014 1:07 pm
Location: Cambridge

Re: New RT (Real Time) kernel branch

Wed Mar 07, 2018 1:09 pm

Yes, disabling the UART should fix it, assuming you aren't using SPI1 or SPI2. If you need a UART but don't need Bluetooth then you could use "dtoverlay=pi3-disable-bt".

I think the problem stems from the fact that the -rt kernel is handling the AUX interrupts in a thread context, which requires that they are masked until the thread completes. Because the interrupt line is shared the masking is not working - it really needs to mask out all the AUX interrupts if any are masked (not ideal, but what else can one do). I also saw a warning message about the interrupt handler re-enabling interrupts, so some work is definitely required.

By the way, I'm not blaming the -rt port - it has just shown up a problem that we've previously managed to avoid.

zavorra
Posts: 7
Joined: Tue Mar 06, 2018 1:21 pm

Re: New RT (Real Time) kernel branch

Thu Mar 08, 2018 10:29 am

Ok thanks

I have been able to run for about two hours

I have opened one issue on github here https://github.com/raspberrypi/linux/issues/2424

Tiejun
Posts: 6
Joined: Mon Mar 05, 2018 11:29 am

Re: New RT (Real Time) kernel branch

Thu Mar 08, 2018 11:23 am

We should not thread irq if this irq is shared, so please add this flag, IRQF_NO_THREAD, as I think.

solanum
Posts: 13
Joined: Thu Nov 03, 2011 9:54 pm

Re: New RT (Real Time) kernel branch

Sat Mar 31, 2018 5:34 pm

On the RPI B, it initially hung on usb otg line. I got it to boot by adding to my cmdline.txt the following:
dwc_otg.fiq_fsm_enable=0 dwc_otg.fiq_enable=0 dwc_otg.nak_holdoff=0

I should add that is with kernel 4.14.27-rt21+ and the default config.

guysoft
Posts: 19
Joined: Fri Jun 14, 2013 12:55 pm

Re: New RT (Real Time) kernel branch

Mon Apr 23, 2018 1:11 pm

Hey, creator of RealtimePi here (https://github.com/guysoft/RealtimePi). I've been maintaining a distro with ships with a realtime kernel for the past 6 months. RealtimePi is based on CustomPiOS so its easy to build other distros on top of it (FOSS IOT FTW).

Also, we are not getting it to be part of the Zynthian project ( http://zynthian.org/ ).

Are you guys planning to maintain this?
Should I start building from that branch and stop maintaining my patachset?
I have had some users playing around and reporting quality, if you want we might be able to share results.

Thanks,

timwood0
Posts: 1
Joined: Wed May 09, 2018 12:33 am

Re: New RT (Real Time) kernel branch

Wed May 09, 2018 1:30 am

New poster here. It's great to see progress on an ASMP real-time Linux design. This approach occurred to me in 2015, but I never found the time for it! :|

I pulled and built the 4.14.y-rt branch (v4.14.34-rt27-rebase tag). The concise instructions for installing a kernel made it easy to stage the modified kernel directly onto the flash drive mounted on the PC I used to cross-compile. For safety, I first did:

Code: Select all

cd /media/timwood/boot  # Ubuntu auto-mounts the flash drive partitions
mkdir -p dtb.orig
mv *dtb dtb.orig
mv kernel7.img kernel7-orig.img
mv kernel.img kernel-orig.img
mv overlays overlays.orig
mkdir overlays
so I can restore the last working kernel on the flash drive from the build PC.

After booting the 4.14.y-rt kernel, /var/log/syslog showed:

Code: Select all

May  8 13:46:18 sturdley kernel: [   20.911984] irq 59: nobody cared (try booting with the "irqpoll" option)
May  8 13:46:18 sturdley kernel: [   20.911998] CPU: 0 PID: 86 Comm: irq/59-bcm2835- Tainted: G        WC      4.14.34-rt27-v7+ #3
May  8 13:46:18 sturdley kernel: [   20.912002] Hardware name: BCM2835
May  8 13:46:18 sturdley kernel: [   20.912035] [<8010fa44>] (unwind_backtrace) from [<8010bce4>] (show_stack+0x20/0x24)
...
May  8 13:46:18 sturdley kernel: [   20.912382] Disabling IRQ #59
May  8 13:46:18 sturdley kernel: [   20.923883] raspberrypi-firmware soc:firmware: Request 0x00030046 returned status 0x80000001
I see other's have found IRQ 59 going off unrequested as well. The 2nd message above issues every 2 seconds from that point.

This kernel does not seem to support the Python SPIdev library, even when rebuilt and reinstalled. My Python scripts that monitor a photocell or a pot show no output, but work fine under the stock kernel. I verified I can talk to the driver with spidev_test. From the discussion, sounds like this tangles with the IRQ 59 issue.

Sysprogs has a pretty nice product stack that uses a J-Link and a Visual Studio plugin to debug the kernel on the RPi. Has anyone here invested in a different tool collection?

Gene1934
Posts: 25
Joined: Tue May 02, 2017 12:47 pm

Re: New RT (Real Time) kernel branch

Mon Jun 25, 2018 12:43 am

I have git pulled the zip of the kernel being announced here, but there is not a sufficiently new versions of things such as sphinx available to a jessie install, so I can't build the pddfdocs to serve as a quide while building it on jessie installed on a pi3b.

Where can I find a copy of sphinx-1.3 or higher, built for jessie.

Instructions or suggestions to upgrade to stretch will be followed when stretch and amanda have signed a truce. Setting up an amanda-client on stretch is a crash-o-matic when amanda comes calling to do the backup. Not to mention that except for local keyboard/mouse event throwaways, its running a 1500 lb lathe flawlessly

Can anyone suggest a backports repo that does have the newer sphinx? The 1.2.3 installed does not satisfy the make pdfdocs makefile section supplied in this zip.

Thanks folks.

Cheers, Gene83

Gene1934
Posts: 25
Joined: Tue May 02, 2017 12:47 pm

Re: New RT (Real Time) kernel branch

Tue Jun 26, 2018 12:02 am

Ping!

Cheers Gene1934;

Return to “Advanced users”

Who is online

Users browsing this forum: 180lifer and 11 guests