User avatar
joan
Posts: 14351
Joined: Thu Jul 05, 2012 5:09 pm
Location: UK

Re: Only 99% reliability with ws281* : sometimes it produces mess

Thu Dec 28, 2017 10:21 am

The WS281* are self-clocking. They need accurate sub-microsecond pulses to work reliably. You can not generate these pulses reliably solely by software under Linux. That's why the (reliable) Pi WS281* solutions are based on programming Pi hardware. Originally the PWM peripheral, latterly the SPI peripheral.

Heater
Posts: 13601
Joined: Tue Jul 17, 2012 3:02 pm

Re: Only 99% reliability with ws281* : sometimes it produces mess

Thu Dec 28, 2017 10:24 am

The APA102 looks like a great solution. Offloads the critical timing to SPI hardware. Looks like it could be bit banged without error as we keep things in sync with the clock signal we are driving.

As long as that extra wire is not a pain and as long as the actual LEDs are as nice.

As you say, OutoftheBOTS, the fix we have here for the WS281 timing requirements is somewhat resource heavy.

I was wondering if there was a more gentle approach....one that does not require dedicating a whole core to the problem.

We only need to prevent the kernel from interrupting the WS281 writer during the time it is actually writing new data out to the LED string. In between writing each "frame" it does not really matter if there is a bit of jitter in the process. Which means the core it is running on can also be running other processes as normal during that time.

I'd like to think that using tasksel to fix the process into a core and setting it's priority high would do the trick.

Perhaps though, we need to disable the kernel from scheduling that core during the frame writing time.

I don't know if this is possible.
Last edited by Heater on Thu Dec 28, 2017 10:51 am, edited 1 time in total.
Memory in C++ is a leaky abstraction .

Heater
Posts: 13601
Joined: Tue Jul 17, 2012 3:02 pm

Re: Only 99% reliability with ws281* : sometimes it produces mess

Thu Dec 28, 2017 10:50 am

joan,
The WS281* are self-clocking. They need accurate sub-microsecond pulses to work reliably.
That is indeed the problem.
You can not generate these pulses reliably solely by software under Linux.
I take that as a challenge!

The WS281 only needs a 400KHz bit rate. That should not be a problem for a 1GHz core of a Raspberry Pi 3.

Heck, even 16MHz AVR can do it:

http://bleaklow.com/2012/12/02/driving_ ... z_avr.html

From that article:

"The output timings are as follows:

logical 0 high 338 nsec
logical 0 low 912 nsec
logical 1 high 680 nsec
logical 1 low 570 nsec

That leads me to suspect that the most important thing when driving the WS2811 is not the exact intra-cell timings for low and high bits, it is getting the bit rate as close to the specified 800 KHz as possible and in avoiding jitter between each block of 24 bits."

Should be a piece of cake.

Looks like the solution we have come up with here does it. All be it at a heavy cost.
That's why the (reliable) Pi WS281* solutions are based on programming Pi hardware. Originally the PWM peripheral, latterly the SPI peripheral.
That sounds neat.

Which solutions are your referring to exactly?

How do they deal with preventing the kernel interjecting between blocks of 24 bits? Do they use DMA to feed the SPI hardware? If not surely the same jitter problems exist there.
Memory in C++ is a leaky abstraction .

Heater
Posts: 13601
Joined: Tue Jul 17, 2012 3:02 pm

Re: Only 99% reliability with ws281* : sometimes it produces mess

Thu Dec 28, 2017 11:59 am

Looks like there is a better way to get hold of a core, keeping the kernel off it, and run our real-time code.

Using cgroups this can be done dynamically at run time rather than the once and for all isolcpus kernel command line parameter:

http://linuxrealtime.org/index.php/Impr ... Properties

Assuming the kernel on raspbian supports all that...
Memory in C++ is a leaky abstraction .

doublehp
Posts: 77
Joined: Wed May 02, 2012 1:11 am

Re: Only 99% reliability with ws281* : sometimes it produces mess

Thu Dec 28, 2017 12:12 pm

Heater I have read various tutos about the CPU topic; there are at least 4 variables involved; in my use case, this one works, and does the job. maxcpus will keep some cores reserved, but without saying which ones; isolcpus is specific and states directly the one you want to keep. There are various uses cases where isolcpus may not work; especially with very large scale servers with very complex CPU topologies, or hotplug CPUs ... where the CPU # can not be predicted ...

Also, I guess it could be possible to change the number of max CPU after boot; imagine you start with maxcpus=8, on a machine with 4 CPUs; then, you add 12 CPUs; at some point, you may want to upgrade maxcpus to 12 or 16 without rebooting. But, such options may be turned off by default in the default conf of Linux. That's why I am pretty sure there exists an option to isolate a core after boot. But, it's easier to do it via a reboot because I am not on a critical server. But when you run Linux on a 1024+ CPU server, there must be a way to do it without reboot.

Of course bash could share it's core with system; but, why not use a dedicated one ? it's an rPi2 with quad core, let's use it :)

I have a detail I want to mention to those who will read this, to understand what's involved behind dedicating a core to the WS2811 task: why it may not be enough, and why the reliability is only 99.9999999%, and not 100.000000000000%. Python is playing with a PWM subsystem of the CPU. And this subsystem is accessed via DMA. Dedicating a core to the task helps a lot because it avoids having the core running python to interrupt the python process to run a higher priority process; ie, if python is launched on core #0, we have a very high chance to get interrupted. So, /usr/bin/taskset -c X nice -n -20, with X!=0 would already improove the issue (by avoiding the kernel interruptions) (in this case, nice is important). So, dedicating a core to python API prevents any interruption in the execution of the process.

But this may not be enough: python is doing DMA access; so, at some point, you are talking to the DMA manager; depending on the CPU design, it may, or not, involve the kernel; on ARM9, it's seems like it works without calling thekernel (don't try this on 386: if you could ever find a dual core 386, a DMA call would still need the kernel to b einvolved). So, if an other process from an other core was trying to agressively talk to the DMA manager, and/or, access the same DMA, it could be possible to mess the LED frames. This case would probably mean that an other user would intentionally make attemps to access the PWM module via the same DMA as Python does; it would probably be a specific attack, and this is absolutely very unlikely on a dedicated unconnected rPi. But, on the paper, a hacker could login my rPi, make calls to access the PWM, and this would probably mess my LEDs.

But, considering the context, this risk is acceptable.

So, isolcpus+taskset is not the "perfect" solution; but it's "good enough for me". Of course, on the paper, an Arduino, or using APA102 would be better technical coices.

I had heard of those features 10 years ago, but I had forgotten about them, and would not have thought using them in here; so thanks for reminding me.

mahjongg I don't have better transistor in stock *now*. Any dx.com/Amazon link welcome.

Heater, no, you are wrong about pigpio: it's a software bitbang library; it allows to perform SPI/I2C on any pin, even when the hardware was only designed for GPIO. But it does it with the help of a kernel driver in order to have very high reliability, like LIRC. Pig does not always use SPI peripheral. I sware. I use it to perform I2C on the second port of an rPi1: GPIO28-31. And rPi1 is single core. Hmmm ... maybe this explains why my i2c bus goes wrong once a month ? Maybe I should move that project to a multicore pi ...

... I need to check if Orange Pi Zero H2 is multicore ... if it does ... I am going to have lots of ideas ... (it does have MISO ... it's an open door to so many things ... FOR ONLY 6€ !!! POE included )
the fix we have here for the WS281 timing requirements is somewhat resource heavy
yes, of course, it's not optimal; but, that rPi2 has nothing else to do in it's life !!! so who cares ?
I was wondering if there was a more gentle approach....one that does not require dedicating a whole core to the problem.
Yes: removing the Linux kernel, and use a monotask system ... what in fact, is even worst :)
We only need to prevent the kernel from interrupting
This can be done if you write a kernel driver for the task; but it still would not be 100% reliable. Also depend which kernel you are using. Up to 2.2, a driver is allowed to freese the system completely. Since 2.4, a driver can not do this anymore: a driver can be interrupted if it takes too much time (this feature also require compatible CPU: not available on 286).

Heater, any solution you may think of may be usable on some CPU, and not other, depending on the CPU design. Since my very first message, I stated I am using an rPi2 ... and never expected my solution to be usable on any other system. Especially, it will NOT, it CAN NOT work on rPi1 !!! The WS2811 is DEFINITIVELY hopeless on rPi1 !!!
Looks like the solution we have come up with here does it. All be it at a heavy cost.
An rPi2 was 32€; but WS2811 are much cheaper than APA102; so, for a very long LED stream, the difference of cost of WS2811 may be in favour of dedicating a core from an rPi2/3, compared to the lower cost of an Arduino (like the pro micro: 4€), and the cost of the same length of an APA102 stream ... So, after how many LEDs is WS2811+rPi2 cheaper than Arduino+APA102 ?

And if you allow me to go offtopic, an orangepi0 (including ethernet and SPI) is only 6€; using an opi0, the WS2811 is cheaper since the first meter !!! below 30 LEDs.

Yes most Python API for WS2811 use DMA. For example, this one does:
https://github.com/jgarff/rpi_ws281x

Heater
Posts: 13601
Joined: Tue Jul 17, 2012 3:02 pm

Re: Only 99% reliability with ws281* : sometimes it produces mess

Thu Dec 28, 2017 2:59 pm

doublehp,

I think that is a great experiment you have done there with excellent results. It admirably demonstrates our points about the non-real-time nature of Linux and how to overcome that. By kicking Linux off of a core or two. This certainly spurs me on to continue this line of investigation with other techniques.

Yes, it seems we can dynamically claim CPUs at run time for real-time purposes. See my post above. Assuming the Raspbian kernel supports cgroups and all.

Thinking about it, dedicating a core to something like LED driving seems wasteful but is it really? The ARM cores on the Raspi SoC are only 1% of the silicon, as far as I understand, the rest is video core. We are only "wasting" a quarter of that 1%. And then the SoC is a small part of the whole board. The "waste" is less than adding an Arduino or other hardware to do this!

We could dump Linux altogether and use some RTOS. When you are software bit banging at 800KHz I don't think that would help much. Unless you dedicate a core... No, it's asking too much to give up the luxury of Linux and all that rides on it.

Certainly I was not expecting this to work on a single core system. Don't care. I don't have any single core Linux systems anymore. Well, except some very small ones.

Great work, duoblehp.
Memory in C++ is a leaky abstraction .

Heater
Posts: 13601
Joined: Tue Jul 17, 2012 3:02 pm

Re: Only 99% reliability with ws281* : sometimes it produces mess

Thu Dec 28, 2017 4:10 pm

doublehp,

Are you up for a simple test of dynamic CPU isolation? Should only take half an hour. Honest.

There is a tool called partrt that can isolate cores from the kernel at run time. I just installed and gave it a quick spin to see if it even builds and runs on a Pi. But I don't have any nice real-time task to test it with just yet.

This is the partrt experiment:

Grab the partrt tool source code:

Code: Select all

$ git clone https://github.com/OpenEneaLinux/rt-tools.git
Make a build directory for it:

Code: Select all

$ mkdir build
$ cd build
Build partrt:

Code: Select all

$ cmake /home/pi/rt-tools
$ make
$ sudo make install
Now we can make some real-time cores, by default 2 real-time, 2 non-real-time:

Code: Select all

$ sudo partrt create 0xc
List the CPU partitions:

Code: Select all

$ partrt list
Now run our program on a real-time, isolated, core:

Code: Select all

$ sudo partrt run -c 0x8 rt myProgram
I'd love to hear how well your LED strip runs under partrt.

More info on partrt here:

http://linuxrealtime.org/index.php/Impr ... Properties See: "The CPU Partitioning Tool - partrt"
Last edited by Heater on Fri Dec 29, 2017 7:40 am, edited 1 time in total.
Memory in C++ is a leaky abstraction .

User avatar
OutoftheBOTS
Posts: 711
Joined: Tue Aug 01, 2017 10:06 am

Re: Only 99% reliability with ws281* : sometimes it produces mess

Thu Dec 28, 2017 9:06 pm

as long as the actual LEDs are as nice.
My side by side comparison is that the colours of the APA102 r much deeper colors than the pale colours of the WS281also the PWM freq for controlling colour brightness of the ws281 is very low compared to the APA102 which make the ws281 have a flicker. You can also update the APA102 at a much higher rate too. The all round specs of the APA102 is much better than the ws281 but as mentioned above so is the price.

Do be ware if buying APA102 make sure that you get the real McCoy as many sellers advertise APA102 but when you get then they r the SK9822 which r inferior

doublehp
Posts: 77
Joined: Wed May 02, 2012 1:11 am

Re: Only 99% reliability with ws281* : sometimes it produces mess

Sat Dec 30, 2017 6:26 pm

Heater wrote:
Thu Dec 28, 2017 4:10 pm
Are you up for a simple test of dynamic CPU isolation? Should only take half an hour. Honest.
I have spent one hour it, and it failed.

rt-tools failed on make:

Code: Select all

[ 50%] Building C object bitcalc/test/CMakeFiles/test_bitcalc.dir/__/src/bitcalc.c.o
cc1: error: unrecognized command line option â-fsanitize=addressâ
cc1: error: unrecognized command line option "-Wno-aggressive-loop-optimization" [-Werror]
cc1: all warnings being treated as errors
bitcalc/test/CMakeFiles/test_bitcalc.dir/build.make:54: recipe for target 'bitcalc/test/CMakeFiles/test_bitcalc.dir/__/src/bitcalc.c.o' failed
make[2]: *** [bitcalc/test/CMakeFiles/test_bitcalc.dir/__/src/bitcalc.c.o] Error 1
CMakeFiles/Makefile2:148: recipe for target 'bitcalc/test/CMakeFiles/test_bitcalc.dir/all' failed
make[1]: *** [bitcalc/test/CMakeFiles/test_bitcalc.dir/all] Error 2
Makefile:123: recipe for target 'all' failed
make: *** [all] Error 2

Heater
Posts: 13601
Joined: Tue Jul 17, 2012 3:02 pm

Re: Only 99% reliability with ws281* : sometimes it produces mess

Sat Dec 30, 2017 7:54 pm

doublehp,

Oh, sorry, I seem to have wasted an hour for you.

Last night I managed to get rt-tools built and running. With some nice results. Only thing is I am running a 64 bit Debian here rather than Raspbian: https://github.com/bamarni/pi64

With rt-tools and a modified version of Gert Van Loo's direct register access GPIO example C code I can bit bang a nice clean square wave of 1MHz:
https://elinux.org/RPi_GPIO_Code_Sample ... ter_access

Code: Select all

    void delay (int n)
    {
        volatile int count;
        count = n;
        while (count > 0)
        {
            count--;
        }
    }
...
...
    g = 7;
    while (1)
    {
        GPIO_SET = 1 << g;
        delay(80);;
        GPIO_CLR = 1 << g;
        delay(80);;
    }
...
...
See scope shots below.

Shrinking those delays I can get a signal up to 50MHz but it does not look so nice on my crude scope setup.

When run normally, with a few other processes loading the Pi, we see that signal drop out for 10ms and more at a time. As expected. Under partrt there are no such drop outs.

But, there is some annoying interrupt firing off every 1ms that cases a little glitch every 1ms. As we see in the scope shot below.

I don't know if this is better than the isolcpus solution we have already. It's a bit more complex to set up. I'll be playing with it some more later.
Attachments
NewFile3.png
NewFile3.png (45.39 KiB) Viewed 3672 times
NewFile2.png
NewFile2.png (34.07 KiB) Viewed 3672 times
NewFile1.png
NewFile1.png (59.87 KiB) Viewed 3672 times
Memory in C++ is a leaky abstraction .

doublehp
Posts: 77
Joined: Wed May 02, 2012 1:11 am

Re: Only 99% reliability with ws281* : sometimes it produces mess

Sun Dec 31, 2017 3:05 pm

What is your scheduler frequency ?

Code: Select all

zcat /proc/config.gz | grep -i CONFIG_HZ
Even if you dedicate a core to a task, GPIO_SET probably calls the BCM2835 kernel driver is probably called at some point; and this will always remains subject to kernel scheduling issues. Much less than any other cases. But still.

Even in my current case with a core dedicated to python calls, I still have a glitch every 5-10mn (with 4-6 calls per second; that's an error rate of one frame through 1000~2000). Dedicating a core to python API fixes only the python code; I don't think it's possible to run a specific driver on a core different than the one running the kernel (not with modern Linux - I wonder if the Hurd could run the BCM driver on a dedicated core).

Still, you could make some room for the kernel by letting the system boot with at least two cores (including core #0), and after boot, ask to move as many process as possible ... away from #0 (it may require process migration - recent Android has it by default), or try to remove affinity to core #0 as soon as possible (in initrd).

User avatar
rpdom
Posts: 15362
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: Only 99% reliability with ws281* : sometimes it produces mess

Sun Dec 31, 2017 3:10 pm

doublehp wrote:
Sun Dec 31, 2017 3:05 pm
GPIO_SET probably calls the BCM2835 kernel driver
No, it writes directly to the GPIO control registers on the hardware. It is a macro that uses a memmap'd area of memory that directly corresponds.

Heater
Posts: 13601
Joined: Tue Jul 17, 2012 3:02 pm

Re: Only 99% reliability with ws281* : sometimes it produces mess

Sun Dec 31, 2017 3:43 pm

doublehp,

Not quite sure what my HZ is but if I run my test normally and load the Pi up with a dozen other time consuming processes I can see my output being stalled for 5 or 10ms at a time very often. As the kernel reschedules things.

When running my test on a core reserved by partrt all that goes away. Except there is some interrupt getting handled by that same core still. It causes 5us glitches every few milliseconds.

As noted above, my test code does not use any kernel driver. It is a modified version of Kert Van Loo's example here: https://elinux.org/RPi_GPIO_Code_Sample ... ter_access. Basically it maps the ARMs I/O registers into user memory space and thereafter writes directly to those registers. This way I can toggle a GPIO pin at almost 60MHz!

If I could figure out how to migrate whatever interrupt that is off the core then we would have a glitch free output.
Memory in C++ is a leaky abstraction .

doublehp
Posts: 77
Joined: Wed May 02, 2012 1:11 am

Re: Only 99% reliability with ws281* : sometimes it produces mess

Wed Jan 10, 2018 9:33 am

Burngate wrote:
Wed Dec 27, 2017 5:36 pm
How about Image
That circuit is known to not work with WS2811. I did not fully understand why, but the result is that this circuit does not respect timings, and kills high frequencies signal.

Solutions have been given in the Orange link I gave just a few days ago put below. Hardware part is explained at the end of my tutorial.
Last edited by doublehp on Wed Jan 10, 2018 11:10 am, edited 1 time in total.

Heater
Posts: 13601
Joined: Tue Jul 17, 2012 3:02 pm

Re: Only 99% reliability with ws281* : sometimes it produces mess

Wed Jan 10, 2018 9:53 am

Could you post that link again? I can't for the life of me spot it here. Thanks.
Memory in C++ is a leaky abstraction .

doublehp
Posts: 77
Joined: Wed May 02, 2012 1:11 am

Re: Only 99% reliability with ws281* : sometimes it produces mess

Wed Jan 10, 2018 11:09 am

Oh, I had not paste/crosslink it here :

Getting OrangePi Zero work with WS2811
http://www.orangepi.org/orangepibbsen// ... 318&extra=

Heater
Posts: 13601
Joined: Tue Jul 17, 2012 3:02 pm

Re: Only 99% reliability with ws281* : sometimes it produces mess

Wed Jan 10, 2018 12:38 pm

The circuit you recommend there is give my a link to www.falstad.com not to any actual circuit.
Memory in C++ is a leaky abstraction .

doublehp
Posts: 77
Joined: Wed May 02, 2012 1:11 am

Re: Only 99% reliability with ws281* : sometimes it produces mess

Wed Jan 10, 2018 3:06 pm

There are many bugs in the forum API. I have fixed the links.

Heater
Posts: 13601
Joined: Tue Jul 17, 2012 3:02 pm

Re: Only 99% reliability with ws281* : sometimes it produces mess

Wed Jan 10, 2018 3:36 pm

OK. Thanks.
Memory in C++ is a leaky abstraction .

doublehp
Posts: 77
Joined: Wed May 02, 2012 1:11 am

Re: Only 99% reliability with ws281* : sometimes it produces mess

Sun Feb 11, 2018 6:28 pm

I may have found a clue about ORangePi; I am not sure yet if it will have impact, but I have good suspiscions. OrangePi0+Armbian include cpufreq; and by default, iddle system run at 240MHz; when heavu tasks start, freq grows up to 1.2GHz. I suspect this speed change to affect the topic of this thread.

So, from now on, all time-critical opis will have this at various places:

Code: Select all

/usr/bin/cpufreq-set -f 1200MHz ; /usr/bin/cpufreq-set -g performance
This detail may also affect rpi2/3. When you want real-time execution of code, and interact with external harware, make sure all your internal frequencies are stable (CPU, bus, memory, video, other ... ). Glitches may be realted to freq changes.

I have discovered this while playing with LIRC.

danjperron
Posts: 3403
Joined: Thu Dec 27, 2012 4:05 am
Location: Québec, Canada

Re: Only 99% reliability with ws281* : sometimes it produces mess

Sun Feb 11, 2018 10:05 pm

Yes I did have that problem! The leds suddenly was disturb . The 3.3V wasn't the problem but it was the OS. I used the SPI instead to send data and the problem went away.

But because I was just using the raspberry Pi to toggle led on my light post I decided to switch all the system with an ESP8266. The Raspberry Pi is way more powerfull to do something so simple and also I had to wait 60 seconds to get the light on. (Boot time!)
The signal is still on 3.3V because the ESP8266 is 3.3V. This demonstrates that the 3.3V is not the issue!

B.T.W. the WS2818 leds are daisy chain so only the first led got the 3.3V data all the other will have the 5V data!

viewtopic.php?f=44&t=182755&hilit=led
https://dl.dropboxusercontent.com/s/zsd ... mppost.mp4

doublehp
Posts: 77
Joined: Wed May 02, 2012 1:11 am

Re: Only 99% reliability with ws281* : sometimes it produces mess

Wed Feb 14, 2018 3:53 pm

danjperron modern distributions are faster to boot than old ones; not because of CPU power, but they have redesigned the init procedure. An rpi2 with a recent raspbian is much faster than 60s. Maybe 30 or 20s, if you put your service in rc.local.

You may also gain more time at power cycle by playing with the default state of pins; a clever design of the electronic part could switch the light ON when a given pin is in input state; for example, set two 1K resistors in serie, between 0V and +3.3, and put the middle point to the GPIO; with 2 or 3 transistors you can determine if that point is floating (input state = 1.6V), or 0 or 3.3 (output state).

When you input 3.3 in a WS2812:
- the rest of the chain does not get 5.0V; the LED chip add only 0.8V per LED; so the output of the first LED is only 4.1V.
- some models do not accept 3.3V data; depends on manufacturer
- there are tons of ways to use WS2812 LEDs with 3.3V data signals; the easiest one is to use two different supplies, and shift their respective 0V levels with a 1N4004; this 0.6V shift is suffisant to bring the data line from out-of-spec to 100%-in-spec.

danjperron
Posts: 3403
Joined: Thu Dec 27, 2012 4:05 am
Location: Québec, Canada

Re: Only 99% reliability with ws281* : sometimes it produces mess

Wed Feb 14, 2018 5:18 pm

danjperron modern distributions are faster to boot than old ones; not because of CPU power, but they have redesigned the init procedure. An rpi2 with a recent raspbian is much faster than 60s. Maybe 30 or 20s, if you put your service in rc.local.
Well it is around ~10 sec with my esp8266 and it does link the wifi inside that period. Also the esp8266 fit inside one of my post outside.
https://www.thingiverse.com/thing:2502685 .
If you scroll the picture you will see the content of the first post box. Only one cpu for up to 30 posts.

My original plan was a Raspberry Pi but I switch to an esp8266. I did have problem time to time with some flickers using the standard library. This is why I started using the SPI.

Now the raspberry pi send pattern to the posts via ethernet packet and the raspberry Pi does something else like my home automation.(nrf24L01 server, MQTT,faumaux,etc)..

What I mean is that the raspberry Pi is sometime to powerful for its own good.

You are correct about the VIH (0.7VDD = 0.7 * 5 = 3.5V) . It is out of speck. Your diode method with a pullup resistor is the best choice. (0.3VDD = 1.5V)

About the DO pin output I can't find anything related to the voltage of DI Pin. This will need some verification. I will check your assumption about 3.3V DI to 4.1V(+0.8V) D0 . But in reality 4.1V is inside the specification!

Return to “General discussion”