Only 99% reliability with ws281* : sometimes it produces mess

[doublehp]

Hello.

Hardware: Raspberry Pi v2, with good PSU.

On pin 18: one LED strip of 12V groups of WS2811 (12 blocks); then, the data line is sent to a serie of individual WS2812 LEDs (5V). For a total of 12+48+1=61 pixels.

Took me some time to understand how to send my very first WS2811 frame, and I had a few hardware bugs (I was supplying the 12V stripe only from one end; when putting full power light, the end of stripe was having lower voltage, and could not transmit the signal correctly to the 5V part).

The problem I have now is that, "once in a while", the LEDs show an unexpected pattern. The bug is completely random; it does not correlate with any particular pattern, ambiant parasite, or crontab execution. Bug can happen only once in a minute, or twice in the same second.

To have bug tracking more consistent, I am now always sending the same frame (the same pattern). My pattern is using the sequence RED-GREEN-BLUE-WHITE-BLACK at 10% intensity.

I have tried 3 different APIs to communicate with the chips:

https://github.com/joosteto/ws2812-spi.git
which depends on https://github.com/doceme/py-spidev.git
and this code includes two methods:

- brute force, where the bug produces the good pattern, but shifted, a bit like is the first bytes were dropped.

- numpy optimisation, where the bug produces a completely random pattern, with some LEDs at full intensity on random colors (like pink or purple, which is not part my pattern)

https://www.fabriqueurs.com/raspberry-p ... pollution/
which uses https://github.com/jgarff/rpi_ws281x.git and neopixel library
... and this produces the same bug as the numpy methode.

Since the behaviour depends on the API used, I have concluded there is nothing wrong with my stripes, and the problem is the API itself.

Whatever goes wrong, the pattern is fine on the next frame.

Is there any chance to get SW2811 100%reliable, or is it a lost cause with rPi2 ?

If it's a lost cause, can you indicate me a better chip that works 100.000000% ? sk9822, APA102, WS2813, WS2818 ?

[mahjongg]

The LED chip needs High Level voltages (ViH) that are 70% (0.7 times) of the power voltage or 0.7 x 5V = 3.5V, or more, to work reliable, the PI only delivers high voltages of 3.3V, so you need a level shifter to deliver reliable "highs" to the LED strip. see the chips datasheet.

[doublehp]

It's true I am using the raw rPi pin, and I have a voltage converter I can use to fix this detail, but, I believe it will not fix the bug I have.

[doublehp]

https://www.amazon.fr/gp/product/B0148B ... UTF8&psc=1
makes things worst. It completely blocks the signal; I don't have any other model; I may try with a pair of transistors.

After reboot, things work ... without level adapter.

[NotRequired]

What type of LED strips are you using, and how is everything wired up (scheme if possible)? I've tried to get SK6812's to work with Pi3B, but there was a timing issue on the data signal that forced me into using an Arduino controlled by the Pi instead.

[Heater]

doublehp,

The problem I have now is that, "once in a while", the LEDs show an unexpected pattern.

The obvious first thought is that this is a consequence of Linux not being a real-time operating system. I would expect that occasionally the kernel fiddles around with it's scheduling, your LED driver process gets stalled a little bit and poof your data transmission is corrupted.

One thing you could try is running your LED driver at a higher priority than everything else. Use the "nice" command to run it like so:

Code: Select all

$ sudo nice -20 myLedProcess

Note you need to run with root privs to set a higher priority.

See here for more info:
https://www.tecmint.com/set-linux-proce ... -commands/

That may not totally fix the issue but if makes a noticeable difference then we know where the problem is.

If you happen to be on a multi-core machine there is a neat trick I discovered recently to get exclusive use to a core and never be bothered by the scheduler.

Basically you can specify the number of cores Linux will use on the boot command line. Say 3 on a four core system.

But, amazingly you can then start a processes and pin it to run on core 4. Which means it has the whole core to itself!

Sorry I don't have the details to hand. I suspect they are a bit more than you want to deal with anyway. I can dig them out if you are interested.

[doublehp]

After reboot (and removing some daemons to free more CPU time), bug occurs much less.

My question was: many forums mention that WS2811 hardly work on rPi; my error rate is now between 1:400 and 1:1000. Are there better libraries than the ones I have mentionned, or do other references provide better reliability ? sk9822, APA102, WS2813, WS2818

I never had the intention to use

My issue is definitely software. I can reproduce the problem by launching a fork bomb.

I can switch from WS2811 to dual line model; but I have to keep rPi2, and can not use Arduino.

I have chained something looking like this (3 LEDs per address, 12V)
https://fr.aliexpress.com/item/2M-5M-WS ... eLevelAB=5
and something like this (one RGB per address, 5V)
https://fr.aliexpress.com/item/50-Pcs-s ... eLevelAB=5

GPIO10 (pin 19) goes directly in the first stripe, then, data goes to the second string. Supplies are a mess to explain. LEDs don't work in the other order.

Anyway, I am now certain the issue is software.

SK6812 is just the same as WS2811.

[Heater]

So did you try the "nice" command I suggested?

[doublehp]

Heater Of course the kernel is messing around; I said I am using an rPi2, so, it's a quad core CPU.

Nice could help, but, if I apply it to my bash script, what will happen to child process ? my generic code is in bash; but the LED API is in python; so my bash script is calling python each time I need to send a frame. Wuld child inherit the nice ? or, do I need to make bash script nice python ? since python is interpreted ... are all steps (compile and run) done under the same child ? if it, could it mess nice ?

Same questions for your core dedication; I knew that years ago, but I had forgotten about it; it's a very good idea. I can restrict boot to 2 cores for system, then, the 3rd core for my bash part, and then, bash would always run python on the 4th core. Do you have tutorials for this ?

Due to rPi specific details, I fear a generic tuto may not work for rPi; because most tutos will probably expect grub ... which we don't have in here.

So yes, I am very interessed by both solutions methods.

It will be hard to provide feedback, because bugs now occur only once every 2 to 10mn. So rare events are hard to catch. I am not going to stare at my LEDs for 20mn just to catch one or two 0.2s event ...

[Heater]

doublehp,

I'm pretty sure child processes inherit the niceness. You can check that out using the tools described in the link I gave.

But is seems to me you only need the nice on the part that deals swith the LEDs. So you could put the nice into your BASH script commands at appropriate places.

I'll try and find the info I had on restricting core usage.

GRUB or not is no consequence. The kernel boot command is. On the Pi that is specified in /boot/cmdline.txt

I can see that checking the operation of all this is a bit of a problem. Ideally you would put an oscilloscope on it. On that has waveform test functionality and can watch the signal for you and trigger on errors. Otherwise you might need to build some hardware to watch the signal. It could probably be done with an Arduino and some programming.

For now, you might have to try the nice command and do some staring !

[joan]

It is much simpler to use a chip such as the APA102 which has separate clock and data lines. They have none of the tight timing problems inherent to the WS281* chips and which are hard to meet using the Raspberry Pi.

[mahjongg]

Sigh, NO! its not timing problems ... you do not listen, is a well know problem... Its the reason why it does work with a 5V logic using arduino.
Its just that the logic level rules are not adhered to, you need to lift the logic 1 level from (less than) 3.3 Volt to over 3.5 Volt, if you would have some knowledge of logic circuits you would know that if a logic input needs 3.5V to reliable see a logic 1, it would not work reliably with less.
Just a simple level converter would make this circuit work 100% reliable.

[Heater]

Certainly, looking at the data sheet, meeting the logic high level from a Pi output is problematic. That would be the first to fix.

However or OPs claims that the "corruption" gets worse when certain software activity is going on on the Pi don't totally rule out timing issues as well.

Anyway, first things first.

[NotRequired]

The problems I had with SK6812 was due to timing issues, no doubt! I had wired up a transistor (not 100% sure, but I think it was a P2N2222A with a VBE(sat) of 2) to lift 3.3V to 5V and it still would not work right. I tried all the libraries I could find, but still no cigar! Never tried the dedicated core or Nice, though..

[mahjongg]

Certainly, looking at the data sheet, meeting the logic high level from a Pi output is problematic. That would be the first to fix.

However or OPs claims that the "corruption" gets worse when certain software activity is going on on the Pi don't totally rule out timing issues as well.

Anyway, first things first.

more activity could put a larger load on the 3v3 regulator which might affect the logic high GPIO output voltage, (which is *very* critical at this point) so no the fact that corruption gets worse with CPU activity isn't automatically an indicator that it isn't a logic level problem, the more I read the more it points to logic level problems, as at least the main reason, it behaves exactly as I would expect.

and yes, fix the level problem first, if afterwards you still have problems you can look for other problems.

[mahjongg]

The problems I had with SK6812 was due to timing issues, no doubt! I had wired up a transistor (not 100% sure, but I think it was a P2N2222A with a VBE(sat) of 2) to lift 3.3V to 5V and it still would not work right. I tried all the libraries I could find, but still no cigar! Never tried the dedicated core or Nice, though..

with a single simple transistor I think you would have also inverted the signal, not just level shifted it, for that you would need two transistors, so perhaps that was the issue.

timing issues as such would not influence the order in which GPIO signals would change, but instead could insert unexpected delays in the execution of your code, (so perhaps a delay between changing of the data, and change in the rising or falling of the clock signal) but that would not normally corrupt the clocking in of data.
at least I would not expect the order of signal changes on the GPIO's to occur, just by the fact you are running on a multitasking OS.

I think that writing a bitbang I/O routines on Raspbian should work fine, if the device you are talking to works asynchronous, and would even work for UART code (which is more or less timing dependant) as long as the intervals are short enough (fractions of a millisecond) compared to the bitrate so that variations introduced by the task switching nature of a non real time OS are less than say 3%.

[NotRequired]

with a single simple transistor I think you would have inverted the signal, not just level shifted it, for that you would need two transistors.

Yes, I believe I used this method, but there still was some weird "artifacts" every now and then. I ended up using an arduino nano powered by a USB-port through wich I then could send commands from the Pi to the nano with a python script.

EDIT: I needed to correct a wrong link with the right one in order for this post to make any sense. The wrong link was something I intended to try, but went with the arduino sollution instead. Oh, and I did not use 2N2222's but BC337 instead, sorry for the mess!

[Burngate]

How about

[rpdom]

How about

Well, you'd only need half that circuit, as they aren't i2c devices. Just a single data line.

[OutoftheBOTS]

In the end if you switch to the APA102 LEDs you will find much better reliability with the RPi and easier to use with the clock line. You can juts run them off the SPI bus.

Here is a simple driver that wrote for them https://github.com/OutOfTheBots/APA102

[doublehp]

I did not say adding a transistor can't help. I said that adding the voltage converter I have make things worst, and that the issue can be solved by stopping X, and made worst by running fork bombs.

Fact is, I am used to work with many chips; and honestly, a chip that does not take 1.8V as a logical '1' is just outdated. So if 3.3 is not enough for a logical '1', really, this chip sux.

Anyway; after stopping all CPU consuming services, the Pi have been running more than 4mn without a single bad frame. Using raw 3.3V ping. After 240s, I got bored, and went away. So, if dedicating a core to this task can do it, I take this solution.

I am perfectly aware it's not the best solution; but, untill now, nobody certified me that APA102 would bork better. You say it's easier to use; you didn't say it would provide 100.000000% reliability.

P2N2222A is a very bad idead; it's a BJT, and this, it's very slow; the base capacity would either limit the max work frequency, or put the output pin of the rPi under very heavy stress due to flowing current. These LEDs work at very high frequency; and you REALLY want to use MOS to drive the data lines. A BJT may seem to work at very low frequencies for synchronous bus with clock line, but it's stupid to introduce such a limiting factor in the design.

mahjongg i don't need to design bitbang; http://abyz.me.uk/rpi/pigpio/ does a very good job for I2C and SPI.

NotRequired with the current state of the project, it would be easier to move LEDs to APA102, rather than controller to Arduino. See first message.

Burngate I am not going to assume that WS2811 are pulldown devices. I already gave a link for a product that is sold to be an I2C voltage converter, and that makes things worst. If it's based on your schematic, then, your schematic is not going to help.

After a simple BS170, the signal is completely distorted.

Solution:

Heater ... I have done everything you said, and it works just perfectly now:
- add this at kernel boot prompt, to reserve cores 2 and 3 (first core is #0): isolcpus=2,3
- start my bash script on core 2: /usr/bin/taskset -c 2 /root/ws2812-spi/ws2812.sh
- in this script, call the python API (the one that really performs the SPI calls) on core 3: taskset -c 3 nice -n -20 /root/ws2812-spi/ws2812.05.py
since I am now using taskset, nice is probably useless. And then, reboot.

Use 'top', and press the hotkey '1' to check if things are working as expected.

Thanks all for help (and yes, next time, I will buy APA102)

[Heater]

doublehp,

Heater ... I have done everything you said, and it works just perfectly now:

Wow, if my suggestion is working perfectly I think you have just made my day.

You see, I was not totally sure about it and was about to try some experiments myself.

- add this at kernel boot prompt, to reserve cores 2 and 3 (first core is #0): isolcpus=2,3

I'd never heard of isolcpus. I was going to use maxcpus.

- start my bash script on core 2: /usr/bin/taskset -c 2 /root/ws2812-spi/ws2812.sh

Again, I had never heard of taskset. I was about to start writing some C code and set CPU affinity on some thread....

I suspect you don't need nice if you do this. Also I think you should only need to reserve one core for the python thing. Surely it does not matter if the BASH shell script above that gets interrupted a bit?

So thanks for all that. I'll have to try this out for myself.

I would not have such a downer on the WS2812. It does not have a clock line which saves wiring I think. I suspect it demands a higher level on it's input because potentially it could be driven, or driving, a long length of wire between the driver and the LED and then LED to LED.

I'd be checking prices and the quality of the light before switching to soemthing else.

[mahjongg]

mahjongg i don't need to design bitbang; http://abyz.me.uk/rpi/pigpio/ does a very good job for I2C and SPI.

exactly my point, bit-banging I/O works under Raspbian.

and yes, you will need a fast transistor for the inverter, and also a low value pullup, and 10K is too much, using 10K will bork up the signal good. Its better to use a logic buffer chip that accepts 3V3 input levels, but outputs 5V level, especially with very high clock rates.

for the same reason an I2C level shifter is not such a good choice, no bidirectonal functionality is necessary, and it only works with slow clock speeds (300KBps), especially with weak 10K pullups.

In any case signal integrity is one of the first things to improve when you see reliability issues.

[Heater]

mahjongg,

...exactly my point, bit-banging I/O works under Raspbian.

I think that conclusion does not follow from what doublehp said.

pigpio uses the I2C and SPI peripherals on the Raspi SoC to do, well, I2C and SPI. That is not bit banging. That is delegating the high speed, time critical, port twiddling to dedicated hardware. Same as we have been doing with good old fashioned UARTS for decades.

Actual software bit banging cannot work on a Linux OS unless you can stop the kernel grabbing your time whenever it feels like it. A real-time kernel goes someway to help with that but not when you are wanting to wiggle a port at high speed, the scheduler timing granularity is too coarse.

Software bit banging can work if you are also banging out the clock signal that is used by whatever you are talking to, to clock data in and out. Then little hiccups in timing don't matter because you have control of the clock.

It won't work for asynchronous work like this LED driver or UARTs etc.

I think doublehp and I have demonstrated that such bit banging can be made to work if you keep the kernel off a core and dedicate that core to your bit banging code and nothing else. Which can be done with isolcpus and tasksel.

I say "I think" because I'm going to have to experiment with this to prove it to myself.

I do agree about signal integrity. But hey, if it works it works

[OutoftheBOTS]

I am perfectly aware it's not the best solution; but, untill now, nobody certified me that APA102 would bork better. You say it's easier to use; you didn't say it would provide 100.000000% reliability.

I can't say if APA102 r totally reliable but so far I haven't had trouble with them.

It seems that the problem with the ws281 was a timing issue that you fixed by throwing enough resources at it till the timing could be maintained. Comparing to the APA102 which don't have any timing issues because of the clock line means you won't have this problem even with very minimum resources used. So certainly for timing the APA102 r much more relaible