charliearmorycom
Posts: 5
Joined: Fri Jun 15, 2018 12:30 pm

Trouble with real-time code

Fri Jun 15, 2018 12:56 pm

I've spent days on this I really hope someone can help me.

I need to do some real-time processing on the PI Zero W. I have a simple test program which toggles a GPIO pin on and off as fast as possible. It goes plenty fast enough, but exactly once per second my program is interrupted and it pauses for 50mS. Ultimately I'd like to get this down to 1mS (or at least down to a few mS).

I haven't read about other people having this same effect. Most people seem to be getting much shorter latency.

Thanks for any help.


Here are the things I've tried:

2 different OS installs, one off-the-shelf from Adafruit, the other is a Preempt RT patched image I downloaded.

I always use sudo when I launch my application.

Running at SCHED_FIFO priority. The system slows to a crawl, but it doesn't change the 50mS delay every second.

mlockall(MCL_CURRENT | MCL_FUTURE);

Disabling all interrupts. This actually does work, but if I leave them disabled for more than a few seconds it seems to cause major problems with the system. I don't know specifically which interrupt is causing the problem.


Here's a code snippet:

while(bcm2835_gpio_lev(BUTTON)) // Exit when button is pressed
{
bcm2835_gpio_write(LED, HIGH);
bcm2835_gpio_write(LED, LOW);
}

User avatar
The Traveler
Posts: 361
Joined: Sat Oct 21, 2017 3:48 pm

Re: Trouble with real-time code

Fri Jun 15, 2018 2:55 pm

Back in the old days, we compiled the linux kernel for real time. This usually meant activating the PREEMPT_RT patch code during a kernel configuration session, recompiling the kernel and installing it. You can find some general aspects here: https://www.linuxfoundation.org/blog/in ... evelopers/

I don't know if the ARM kernel can be compiled this way.

Cheers.
Retired IT professional, C programmer and "beardie weirdie".
RPi interests: Developing an Infinite Improbability Drive
“Thinking outside of the box allows you to get rewards outside of your reach.” Matshona Dhliwayo

User avatar
joan
Posts: 13578
Joined: Thu Jul 05, 2012 5:09 pm
Location: UK

Re: Trouble with real-time code

Fri Jun 15, 2018 3:06 pm

What are you trying to do?

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 20737
Joined: Sat Jul 30, 2011 7:41 pm

Re: Trouble with real-time code

Fri Jun 15, 2018 4:02 pm

The Traveler wrote:
Fri Jun 15, 2018 2:55 pm
Back in the old days, we compiled the linux kernel for real time. This usually meant activating the PREEMPT_RT patch code during a kernel configuration session, recompiling the kernel and installing it. You can find some general aspects here: https://www.linuxfoundation.org/blog/in ... evelopers/

I don't know if the ARM kernel can be compiled this way.

Cheers.
Yes, it can. ISTR there is a branch in our kernel repo specifically for the RT patches.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Please direct all questions to the forum, I do not do support via PM.

charliearmorycom
Posts: 5
Joined: Fri Jun 15, 2018 12:30 pm

Re: Trouble with real-time code

Sat Jun 16, 2018 12:09 am

I'm trying to track a stepper motor. I'm measuring quadrature, it's similar to reading a rotary encoder.

The PI runs plenty fast enough except for once a second when my application just stalls for 50 milliseconds.

It's not approximately once a second, I measured it on my scope. It looks like it's exactly once a second.

What does the PI do exactly once a second? Maybe if I knew what it was doing I could turn it off.

Thanks.
Last edited by charliearmorycom on Sat Jun 16, 2018 4:22 am, edited 1 time in total.

User avatar
The Traveler
Posts: 361
Joined: Sat Oct 21, 2017 3:48 pm

Re: Trouble with real-time code

Sat Jun 16, 2018 12:26 am

Yes, it can. ISTR there is a branch in our kernel repo specifically for the RT patches.
Ah, good to know, thanks.

Cheers.
Retired IT professional, C programmer and "beardie weirdie".
RPi interests: Developing an Infinite Improbability Drive
“Thinking outside of the box allows you to get rewards outside of your reach.” Matshona Dhliwayo

User avatar
joan
Posts: 13578
Joined: Thu Jul 05, 2012 5:09 pm
Location: UK

Re: Trouble with real-time code

Sat Jun 16, 2018 4:35 am

Does http://abyz.me.uk/rpi/pigpio/examples.h ... monitor_py also show a 50ms delay? Perhaps you mean 50µs?

ghellquist
Posts: 68
Joined: Thu Aug 02, 2012 8:47 am
Location: Stockholm Sweden

Re: Trouble with real-time code

Sat Jun 16, 2018 5:59 am

Is there a difference if you use isolcpus?

charliearmorycom
Posts: 5
Joined: Fri Jun 15, 2018 12:30 pm

Re: Trouble with real-time code

Sat Jun 16, 2018 9:53 am

Regarding monitor.py, please correct me if I'm wrong, but isn't that for input? While it is true that I ultimately want to do input and I was having difficulty with missed steps, due to that I wrote an extremely short output routine that just flashes an LED at maximum speed. I can see on my scope that it stops flashing once a second for 50 milliseconds (not microseconds). A twentieth of a second is quite a long time and is clearly visible on the LED.

It seems weird that the PI has such long pauses - even on high priority threads and nobody noticed before. That makes me think I'm missing something important. But since I've simplified the test as much as possible and tried it on two completely unrelated OS builds, I am at a loss.

Was your suggestion about monitor.py because you thought the delay might be in the bcm2835 library itself? If so, I suppose I could try writing to the bcm2835 chip myself without using the library.

Regarding the question about isolcpus, I have never used it but it looks to me like it's only for multi-core processors. The Raspberry PI Zero W only has a single core. I would try testing on a multi-core device, but I don't have one and I need to resolve this one way or another quickly (preferably this weekend).

Thanks again for your assistance.

User avatar
joan
Posts: 13578
Joined: Thu Jul 05, 2012 5:09 pm
Location: UK

Re: Trouble with real-time code

Sat Jun 16, 2018 10:17 am

monitor.py shows the GPIO levels. It doesn't know or care what mode the GPIO are in (INPUT, OUTPUT, ALT0, ..., ALT5).

What are you trying to do?

charliearmorycom
Posts: 5
Joined: Fri Jun 15, 2018 12:30 pm

Re: Trouble with real-time code

Sat Jun 16, 2018 11:05 am

Okay, I just learned something.

If I don't run my program at SCHED_FIFO, but instead I run at normal priority there are significant changes to the behavior:

First, the delay isn't exactly 50mS exactly every second. It's instead random lengths at random times.

Running on the stock OS the maximum delay I measured is 30mS. It's still a long time, but it is a slight improvement.

On the Preempt RT patched OS the maximum delay I measured was under 6mS. That's almost what I need (I probably need 2-3).

I guess that confirms that Preempt RT is doing something, but it's interesting that it seems to have no effect when the task is running at high priority.

It looks to me like running at high priority causes the OS to queue up everything it needs to do until it can't wait any longer (once per second), then tries to execute everything at once. This giant spurt of activity is actually worse than running at lower priority.

If I could break this activity into smaller pieces that occur when I can spare the time I might be able to get this to work. I'm thinking either turning off interrupts while I'm tracking the motor, (for 100 or so mS) then switching the interrupts back on for a few milliseconds and back off again. Or perhaps the same thing, but raising and lowering the priority. Though I don't know how the OS will react to having the priority change a dozen times a second (or if it can even change it in less than a millisecond).

Any thoughts?

Thanks.

User avatar
Gavinmc42
Posts: 2130
Joined: Wed Aug 28, 2013 3:31 am

Re: Trouble with real-time code

Sat Jun 16, 2018 11:46 am

I got jitter on stepper motors when software looping on Linux.
I never really tried Joan's library because I had other issues with the i2c.

Ultibo is my preferred way to make apps these days, never really noticed much jitter from OS interrupts.
But I have not tested it properly, just made a two stepper walking bot, that seemed fine.
I do use it for realtime control but is only temperature controlling, 100ms here or there does not matter for that.

Ultibo does allow better control of realtime stuff but even then guys have found 400ns of jitter that has not been tracked down yet.
For serious RT stuff I always use a microcontroller controlled via the i2c.
Simple stepper code on a microcontroller is easier than trying to figure out the insides of the Pi Vc4 RTOS/ARM OS/hardware interactions.

I still think at least two steppers could be controlled via the serial PCM/PWM using DMA methods.
Just a theory at my current skill level, but if two channels of audio work then two steppers should too.
PCM has FiFo and DMA so as long as the FiFo is filled within a certain time the waveform will not have jitter.

I do not know of anyone who has done this yet.
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

charliearmorycom
Posts: 5
Joined: Fri Jun 15, 2018 12:30 pm

Re: Trouble with real-time code

Sun Jun 17, 2018 6:10 am

> Ultibo is my preferred way to make apps these days

Thanks. I wasn't aware of it until now. It could be helpful for future projects. Unfortunately it's not helpful right now, since for this I need to be able to talk to a printer and Ultibo doesn't support printer drivers.

> For serious RT stuff I always use a microcontroller controlled via the i2c.

Yes, I think I'm going to have to add an inline microcontroller that can track the motor and then spoon-feed the Raspberry PI at a rate that it can handle. After the motor position information is processed down to a count and packed into a couple of bytes, I doubt the data rate will be more than about 300 baud - quite a contrast to the millisecond timing I require now.

Rather than I2C, I plan to use a single asynchronous serial line, that way I can reduce pin count on the µC.

It's unfortunate that the RPI can't do this on its own. Everything I read suggested it was possible to get raspbian down to 500µS latency. Maybe if I'd spec'd a multi-core PI for this project rather than the PI Zero I wouldn't be in this nightmare.

Despite the headaches, I think through this project I've developed a fondness for the PI. I'll have to order a few of them to have around for future projects (especially a multi-core version!).

> Simple stepper code on a microcontroller is easier than trying to figure out the insides of the
> Pi Vc4 RTOS/ARM OS/hardware interactions.

Definitely!

> I still think at least two steppers could be controlled via the serial PCM/PWM using DMA
> methods. Just a theory at my current skill level, but if two channels of audio work then two
> steppers should too. PCM has FiFo and DMA so as long as the FiFo is filled within a certain
> time the waveform will not have jitter.

I think you are correct, provided that you use a stepper driver which takes step/direction inputs rather than raw waveforms (with raw waveforms you'd need both channels just to provide simple quadrature for a single motor).

> I do not know of anyone who has done this yet.

I've seen audio I/O used for some strange things (e.g. 3D printing), but I don't recall seeing anyone do what you propose. Not yet anyway. :)

User avatar
Gavinmc42
Posts: 2130
Joined: Wed Aug 28, 2013 3:31 am

Re: Trouble with real-time code

Sun Jun 17, 2018 8:40 am

I think you are correct, provided that you use a stepper driver which takes step/direction inputs rather than raw waveforms (with raw waveforms you'd need both channels just to provide simple quadrature for a single motor).
For real Stepper motor gadgets I use the Hybrid stepper motors, 17, 23 or 34 size with proper step/direction drivers 24VDC+.
Most CNC/3Dprinting is 2 axis with step incrementing Z axis, the audio hardware should be able to do this?

Joan's software can also do arbitrary waveform generation that should drive them too.

I used the cheap reduction steppers that come with 2003 driver PCBs, with waveform generated by Ultibo.
https://tronixlabs.com.au/robotics/moto ... australia/
It was a test of Ultibo and these unipolar steppers, conclusion - rewire for bipolar, change driver and try again ;)
Perhaps even replace the ferrite magnet with a super magnet ;)

I was hoping they might be strong enough to making a hexapod walking bot or robot arm as they are cheap :(
Need many more IO, perhaps use a Zero as IO controller, hard to get any other cpu with 512MB, 1GHz for $5 ;)
I have fun booting Zero's from USB, no SD card, USB comms not sorted yet.
Got a recent clue, after USB booting the Zero may still be in OTG mode :?

I got a Cluster Hat to experiment with Zero's for Pi3B+ IO controllers, vision/audio processing etc.
A Pi3B+ with 4 Zero's should make a kick-arse robot system core.
On hold until baremetal OTG USB comms figured out ;)

8 x Arm cores - 4 Aarch64 with NEON, 4 Aarch32 with SIMD , 3GB RAM, 10 VPU's, 240 ALU's in 60 QPU's.
Need to get MUCH better at coding to use that lot to the max :lol:

Use the 2 x VPU's as RT controllers? Not needed if video not used?
RPT guys have told me the QPU have no access to GPIO, there are 48 cpu's in them :(
There is a compiler/assembler for the VPU's now, but that's pretty bleeding edge.
If the two VPU's could be used for RT cpu's it would be similar to the Beaglebone's RT cpu's.

One of the examples for Ultiobo is multicore coding,
One arm core for motors, one for display, one for G-code, one for comms/control?
Thought about it, on the list of things to try - big list :lol:
Plenty of stuff in Pi's to keep me busy for a few more years yet.
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

tjrob
Posts: 19
Joined: Tue Feb 19, 2013 5:31 pm

Re: Trouble with real-time code

Sun Oct 07, 2018 6:43 pm

I assume that your program sets its scheduler to SCHED_FIFO and high priority. By default the kernel does not permit such real-time processes to starve other, non-realtime processes, so once per second it pauses the realtime process for 50 ms. As you observed.

Here is my code to avoid this:

Code: Select all

#include <sched.h>
	
        int prio = sched_get_priority_max(SCHED_FIFO);
        struct sched_param param;
        param.sched_priority = prio;
        sched_setscheduler(0,SCHED_FIFO,&param);
        // This permits realtime processes to use 100% of a CPU, but on a
        // RPi that starves the kernel. Without this there are latencies
        // up to 50 MILLISECONDS.
        system("echo -1 >/proc/sys/kernel/sched_rt_runtime_us");
The downside is that this will starve other processes, including essential kernel threads. If you just do a tight loop toggling GPIO pins that will be a problem, even on a multi-core RPi 3B+. But if your loop does a blocking read or write of an SPI or I2C device, all is well. Calling usleep() also works, but it can have latencies up to 50 microseconds.

Return to “Advanced users”