Solution: Dedicating one core to a real-time process

[tjrob]

Linux is most definitely not a real-time operating system, and if you just use the default configuration you'll find you occasionally have latencies as large as 50 milliseconds. That's pretty hopeless for anything that needs real-time response. You can certainly install some real-time OS, or write a kernel module -- those require real expertise and are a major effort. If the requirements are such that this approach can meet them, it will be much easier. Note that the results of using some RTOS, or writing a kernel module or "bare metal" code may be less than expected, as the RPi hardware is not capable of disabling interrupts on a single core (tests show that the Linux Kernel routine local_irq_disable() disables interrupts on all cores).

I describe how to achieve latencies less than 3 microseconds 99+% of the time, with a worst-case latency of 41 microseconds. This is on a Raspberry Pi 3B+ running 2018-06-27-raspbian-stretch-lite, without any kernel drivers or other exotica. Note that on single-core models the results will be much worse. While this is not "hard" real time, it is sufficient for many applications. My applications are for a scientific instrument and involve sampling hardware at 1,000 Hz and writing each result to a socket.

By "latency" I mean the time difference between the time that "something" happens, and the time the program knows that it happened. Here "something" could be an edge on a GPIO, a time-delay ends, a specific time of day, etc. As best I can tell, large latencies are due to an interrupt causing the kernel to schedule some other task. Because of that, worst-case latencies do not add, and during one loop the code can call several routines that have latencies, but it essentially never happens that more than one of them gets a large latency. The RPi has a hardware counter that increments at 1 MHz, so measuring latency with 1 microsecond (us) resolution is easy.

The idea is to dedicate one core on a RPi 3B+ to the real-time process, and then write it as a simple loop doing whatever it needs to do:

Code: Select all

initialize
for(;;) {
    wait for connection to the socket
    for(;;) {
        wait for GPIO edge indicating ADC data are ready
        read ADC channels
        fprintf(socket, ...)
    }
    close socket
}

Yes, there is time to use fprintf(). But using stdout to an ssh connection is barely possible at 1 kHz, while using a bare TCP connection leaves a lot more headroom.

How to do it

1. add this argument to /boot/cmdline.txt:
   isolcpus=3
   This prevents Linux from scheduling processes on core 3. But interrupts still happen on it, and there is an essential kernel task running on it. Still, it is a good start, but not at all sufficient.
2. Set the process's CPU affinity to 3. See attached code.
3. Disable turbo mode. Otherwise the OS will change the CPU clock frequency, and that screws up the SPI clock and the overall timing. See attached code.
4. Set the process to real-time FIFO scheduling, with maximum priority. See attached code.
5. Permit real-time processes to use 100% of the CPU. Without this you'll get 50 millisecond delays once per second. See attached code.

Attached are Realtime.h and Realtime.cc that implement the code needed in your process for #2-5. Just call Realtime::setup() during initialization.

Also attached is latency.cc to measure the latencies. Note that to measure the GPIO latency it needs GPIO22 connected to GPIO23. Here is a screenshot of an 18-hour run on a Raspberry Pi 3B+:

The attachment Screen Shot 2018-12-10 at 5.39.52 PM.png is no longer available

The line
0us: 0 76961522 31486427 9058 ...
Means that no sample had a 0-microsecond latency, 76961522 samples had a 1-microsecond latency, 31486427 samples had a 2-microsecond latency, etc. See the comments in the code for an explanation of what is measured.

[tjrob]

The code attachment got lost. Here it is.

I also forgot to show how to build the latency program:

Code: Select all

g++ -o latency latency.cc Realtime.cc -lpigpio -lpthread

[Joel_Mckay]

Guaranteed latency OS like LinuxRT essentially install a program as a kernel module.

Our clubs rc1 variant of Stretch with this kernel will be out Dec. 28, 2018 (includes many updates):
https://sourceforge.net/projects/microm ... pberry-pi/

Cheers,
J

[tjrob]

That latency program doesn't actually do anything in the real-time process. In particular, it does not do the fprintf(socket, ...). Actually using it, I learned what should have been obvious -- network flow control can back up and block the fprintf(), causing it to miss samples.

Fortunately there is a simple fix: split the program into two threads, the real-time thread that takes samples, and an IO thread that writes to the socket. Connect them via a thread-safe queue. Have the real-time thread set its CPU affinity to CPU 3, and the IO thread to CPUs 0,1,2; both need real-time priority.

With the improved version, an 18-hour run using 1 kHz sampling missed no samples. For 99.999% of the samples the queue had fewer than 10 entries, but there were two episodes where it exceeded 1,000 entries (and then rapidly decreased to 0) -- that's an I/O delay of more than a second! The measured time between samples never differed from 1,000 us by more than 50 us. So this meets my requirements.

My code is specific to the hardware being sampled, but if anyone wants me to post it, just ask. My first program implements a 200 MHz frequency counter in a CPLD, sampled at 1 kHz; I will be doing an ADS1256 4-channel ADC sampled at 1 kHz, and also an SR04 ultrasonic ranger at 20 Hz (with accuracy of 2 mm RMS -- much better than most because it times the echo using the SPI clock).

[nixiebunny]

Hi. I I'm an engineer at the University of Arizona who works on radio telescopes. I'm working on building a real-time controller for a telescope secondary mirror, to replace an aging FPGA system. This thing moves really fast - 20 milliseconds from start to finish.
https://public.nrao.edu/wp-content/uplo ... 85x300.jpg

I am interested in your method of locking one core to do the real-time task. In my case, it will be a 200 microsecond PID loop, receiving position feedback over serial port running at 2 Mbit/sec.

I expect to use shared memory to implement the communication between the host interface thread and the PID loop thread.

Would you be willing to post your code for your loop? I'd like to see how you solved it, to save effort on my end.

Thanks for posting.

[Joel_Mckay]

Would you be willing to post your code for your loop? I'd like to see how you solved it, to save effort on my end.

FPGA solve a different set of problems, and FIR/CIC are used in DSP given the uniformity of timing latency.

I am not familiar with the equipment in the photo, but would recommend looking at a NIOS soft-cpu like Altera offers.
This runs a specialized Linux kernel that allows mapping shared memory into the dedicated FPGA sections.
Or put another way, the real-time is handled by the FPGA hardware, and the DMA buffers are polled by the soft-cpu multitasking OS.

There is a high probability things have changed over the years.. but repeatability is usually very important to the credibility of a scientist's works.

[nixiebunny]

I am not familiar with the equipment in the photo, but would recommend looking at a NIOS soft-cpu like Altera offers.
This runs a specialized Linux kernel that allows mapping shared memory into the dedicated FPGA sections.
Or put another way, the real-time is handled by the FPGA hardware, and the DMA buffers are polled by the soft-cpu multitasking OS.

Joel,

Thanks for your concern. I know about doing DSP in FPGAs - I programmed one to be a 10 GSPS FFT spectrometer.

This application needs to calculate two motor drive numbers from ten parameters every 200 microseconds. It's easily doable by an Arduino, except for the I/O. The previous designers used an FPGA because they weren't being clever.

I just need to ensure that the maximum latency is under a hundred microseconds.

[Joel_Mckay]

I just need to ensure that the maximum latency is under a hundred microseconds.

This may ultimately depend on how tolerant your device is to timing jitter, and whether your 10 variables are concurrently sampled.
https://en.wikipedia.org/wiki/Segal%27s_law

One could compile linuxRT to get 5kHz task resolution, lock the CPU/GPU clock, and inhibit the USB IRQ subsystems with a busy-loop (costs about 30% of standby CPU load on the Pi3B+). I have not personally tried higher RT task resolutions on the pi.... YMMV with your custom kernel setup.

Your group may want to have a look at this publication at some point too....
"Optimal signals of Golomb ruler class for spectral measurements at EKB SuperDARN radar: Theory and experiment" ( O. I. Berngardt, A.L.Voronov, and K. V. Grkovich )

Indeed, one may get better results with a bare-metal atmega system, as Arduino SDK has a timer0 IRQ service you would need to explicitly cripple to get repeatable timing.

Best of luck,
~J~

[Paleloshow]

Thanks for this Real time solution. I was struggling to make Xenomai work on my Pi. I managed to run a neural network using your Realtime.h library and I got an execution time of 14 ms. I measured this using the empty() section of the latency.cc code (I call the neural network between begin and end Realtime::micros). Although it's a good time, I would like to lower this time even more. Do you know any means to do that? The neural network I am using it's a basic one: Iris data classification.

[PurpleMark]

I had a go at benchmarking this approach on the Raspberry Pi 4.

Firstly, new peripheral base address: 0xFE000000.

I ran the 'short delay' and 'long delay' tests only, at 600MHz, 1GHz and 1.5GHz for a duration of 4 hours.
(arm_freq=600/1000/1500 & force_turbo=1 in /boot/config.txt)
Note: I get the impression from reading other posts that fixing the clock at 1.5GHz has a risk of overheating under a heavy load so I kept cores 1/2/3 at near idle (resulting in a fairly stable 63 degC in 23 degC room). Incoming updated firmware (that shows a >3 degC improvement) and CPU cooler would be beneficial here.

Latency.png (38.79 KiB) Viewed 5139 times

As expected, the latency tail improves at higher clock speeds (bear in mind it's a logarithmic probability scale so the effect is very tiny), but the maximum latency doesn't change by much. It would be interesting to find find out what the hold-up here is.

[u0078867]

tjrob, great post!

Is it possible to have the version of the code with the I/O extra thread?

[u0078867]

What do you think of viewtopic.php?f=29&t=216051&p=1599557&s ... 5#p1599557 ?

[akashkumar13]

Thanks for the great post. One question, if we anyway bind the process to a particular CPU in the realtime.cc program, do we still need to isolate a processor for the entire duration (as mentioned in the first step in the first post)? I essentially want to run a task twice a day, and only during that time (takes about a minute or so) I would like to reserve the core. Reserving a core for the entire time would then be useless.

[HermannSW]

+1 on "Thanks for the great post."

I played with the bash demo application in Realtime.cc -- really cool.

First I verified with cpuburn that isolcpus works as expected:
https://github.com/ssvb/cpuburn-arm/blo ... burn-a53.S

Code: Select all

pi@raspberrypi4B2:~ $ tail --bytes 12 /boot/cmdline.txt 
 isolcpus=3
pi@raspberrypi4B2:~ $ ./cpuburn-a53 &
[1] 5577
pi@raspberrypi4B2:~ $

"top" confirms that only cores 0-2 get utilization from the 4 cpuburn threads:

Code: Select all

pi@raspberrypi4B2:~ $ top -bn 1 -1 | grep "^%Cpu"
%Cpu0  :100,0 us,  0,0 sy,  0,0 ni,  0,0 id,  0,0 wa,  0,0 hi,  0,0 si,  0,0 st
%Cpu1  :100,0 us,  0,0 sy,  0,0 ni,  0,0 id,  0,0 wa,  0,0 hi,  0,0 si,  0,0 st
%Cpu2  : 80,8 us, 19,2 sy,  0,0 ni,  0,0 id,  0,0 wa,  0,0 hi,  0,0 si,  0,0 st
%Cpu3  :  0,0 us,  0,0 sy,  0,0 ni,100,0 id,  0,0 wa,  0,0 hi,  0,0 si,  0,0 st
pi@raspberrypi4B2:~ $ 

"taskset" confirms for each cpuburn pid:

Code: Select all

pi@raspberrypi4B2:~ $ for p in ` ps -ef | grep cpuburn-a53$ | cut -b10-14`; do taskset -c -p $p; done
pid 5577's current affinity list: 0-2
pid 5578's current affinity list: 0-2
pid 5579's current affinity list: 0-2
pid 5580's current affinity list: 0-2
pi@raspberrypi4B2:~ $

Finally remove unnecessary load:

Code: Select all

pi@raspberrypi4B2:~ $ killall cpuburn-a53
pi@raspberrypi4B2:~ $ 

Next build Realtime.cc bash demo
"sudo" is needed because of mmap, final "su - pi" runs bash as pi user:

Code: Select all

pi@raspberrypi4B2:~ $ cd Realtime; g++ -DTEST_REALTIME Realtime.cc -o Realtime
pi@raspberrypi4B2:~/Realtime $ sudo ./Realtime
Realtime mode setup, here's a shell on CPU3
root@raspberrypi4B2:/home/pi/Realtime# su - pi
pi@raspberrypi4B2:~ $ 

Then start chromium browser on core 3:

Code: Select all

pi@raspberrypi4B2:~ $ DISPLAY=:0.0 chromium-browser nema17/x.anim.gif 2>/dev/null >/dev/null &
[1] 4467
pi@raspberrypi4B2:~ $ 

All chromium browser processes started are now runnning on core3:

Code: Select all

pi@raspberrypi4B2:~ $ for p in `ps -ef | grep chromium-browser | cut -b10-14`; do taskset -c -p $p; done
pid 4467's current affinity list: 3
pid 4488's current affinity list: 3
pid 4490's current affinity list: 3
pid 4517's current affinity list: 3
pid 4519's current affinity list: 3
pid 4660's current affinity list: 3
pid 4664's current affinity list: 3
pid 4687's current affinity list: 3
pid 4694's current affinity list: 3
pid 4827's current affinity list: 3
pi@raspberrypi4B2:~ $ 

[cleverca22]

and if OP is still trying to deal with an ADC, there are new solutions that dont need realtime

the SMI peripheral can deal with sampling 18 gpio pins at once at a defined clock rate (possibly with an external trigger/strobe?)

that can then give you a high enough sample rate to see the colorburst in NTSC for example:

[HermannSW]

Next build Realtime.cc bash demo
"sudo" is needed because of mmap, final "su - pi" runs bash as pi user:

Code: Select all

pi@raspberrypi4B2:~ $ cd Realtime; g++ -DTEST_REALTIME Realtime.cc -o Realtime
pi@raspberrypi4B2:~/Realtime $ sudo ./Realtime
Realtime mode setup, here's a shell on CPU3
root@raspberrypi4B2:/home/pi/Realtime# su - pi
pi@raspberrypi4B2:~ $ 

All processes started in that bash inherit the current affinity list of that bash and run on core 3.

Useful for verifying that for processes started in background is below taskset command:

Code: Select all

pi@raspberrypi4B2:~ $ raspivid -md 4 -w 1012  -h 760 -fps 80 -o tst.h264 -pts tst.pts &
[1] 16198
pi@raspberrypi4B2:~ $ taskset -c -p $!
pid 16198's current affinity list: 3
pi@raspberrypi4B2:~ $