Using threads in C for reducing picamera dropped frames

[catubc]

Hi

I've got a very specific application that I'm trying to implement: a 60Hz precise strobing application for imaging in neuroscience applications. The 3 key things are: (i) 60Hz acquisition 256x256 or 128x128; (ii) strobing two different lights alternating at 60Hz; and (iii) saving up to 20min recording or longer (to USB hard disk (non-SSD)) with little or no dropped/skipped frames.

It seems that acquiring continuously with the picam and saving to disk leads to periodic (random?) freezes in acquisition for some reason. We've done many tests and recordings as short as 30 seconds (but certainly for a few minutes) at >30fps can have up to a few hundred milliseconds of missing data (this has also been documented by others). We've even seen freezes of up to 7seconds during a 10minute recording. However, saving to memory (e.g. using a python list and saving the frames to disk after recording is complete) seems to reduce all (or almost all) frame drops. But that is limited by the raspberry pi 3 memory, so ~45seconds (for recording a 256 x 256 window @ 60Hz) and about 3.5mins for 128x128.

So I wrote 2 C-based modules to call using python threading: (1) to do light strobing (i.e. flash a light at the beginning of every frame) and (2) implement a parallel save function that essentially accumulates 100-500 frames and then saves them to disk "without" interupting the encoders.py acqusition loop in _callback_write().

(1) Strobbing code is working surprisingly well. So using a shared variable (using python ctypes) between C and python, my C-based strobing code (above (1)) is mostly on time and lighting every single frame usually within 15usec or less of a new frame.

(2) C-based saving does not work so well. It seems that despite saving being passed to a 2nd thread, there continue to be periodic hangups. The errors are less than before - and the hangups are usually much shorter. If anyone is interested in this code, let me know.

I would like to better understand why saving to disk - even using a "2nd" core - continues to interfere with the picam grabbing frames from the GPU. My best guess is that the GPU->CPU pipeline somehow overlaps with the CPU->USB pipeline, or something along those lines.

Thanks!

[KnarfB]

Hi, that`s an interesting problem. I don't have a solution, but here are some thoughts: You may, just for testing, decompose your software: grabbing only (without saving) and start a completely independent process writing arbitrary data to the disk. If this works, you proved your best-guess wrong.
Next I would try to use a named pipe (FIFO) for decoupling the producer and consumer processes. This is built-in in Linux. The last alternative would be using a thread-safe queue in C to decouple producer and consumer. hth Frank

[jamesh]

I think you have taken the right approach. You need to find out though what is causing the delay. I'd suspect the writing to disk (via USB) is taking all the processor time, or causing some uninterruptible delay in the kernel. You need to use some performance analysis tool to find out where things are going pear shaped. Google might find some relevant tools for Linux.

[YCN-]

I'll advice you to try PREMPT RT and see if it's better.
You can also set a higher priority to your application and threads (withor without PREEMPT RT).

What distribution are you using? You can go for a distribution done with buildroot that will only handle your application. However this could be a long process and if the problem comes from your application it will still be the same result...

I do think the glitchs are comming from the kernel, because the kernel might sometimes pause your application to do some kernel stuff and then your application will struggle untill it gets back the processor and hardware, and that can cause this kind of glitches with CPU intensive application.

You can also try to sleep your program sometime between data acquisition to let the kernel breath when you do not need to acquire data. But that depends on how you implement your program. If it's running as fast as the machina can, ie with a while(1) loop that never sleeps and heavy processing in it, with open write or driver calls, well that can cause glitches.

I think that the resolution of this problem will be very interesting. To sum first try to see if the problem comes from your application, try to make a more breathy programm, then if your programm is sexy when it comes to processing power, try to up the priority of your application, if you have still the same glitch try preempt RT, if you still have those glitchs try to dedicate your kernel to your application using buildroot, and if you still have the glitchs try buildroot + PREEMPT RT. I'm only writing about PREMPT RT but you can also try true Real Time distribution such as Xenomai.

YCN-

Edit : as jamesh said writing in memory can also cause problem of that sort.

[catubc]

Hi everyone

Thanks a bunch for the detailed explanations. I'm a bit behind, but here's what I've tried:

- strobbing (C-based)
- recording @60Hz
- without saving to disk (buf.data goes into the ether)

Results: no freezes and t(LED strobe) - t(GPU timestamp) were very small. The conclusion is that passing the data from GPU->python while also tracking those time stamps is not a problem. I still would like to try the "write garbage" option where I just save some garbage from C, but I'm doubtful it will improve things.

However, before starting to get into the real time OS options, it was pointed out to me that python multi-threading does NOT allow additional cores to be used for processing due to the Global Interpreter Lock. This is a problem for my approach because I've relied exclusively on multi-threading. The code works as follows:

- script calls picamera code
- picamera code initiates threads for 2 different python functions (one to call LED strobing in C; one to call C-based saving module)
- the LED python thread then calls C-based LED strobing
- the other python thread calls C-based saving code

It seems that C-based saving code may not be running in "parallel". Can anyone comment on that? Again, I'm just using multithreading module already setup for picamera to call these two threads along the way.

-> python thread -> call C-based LED (dll/ctypes)
/
picamera -------------- ongoing picamera processing...
\
-> python thread -> call C-based saving (dll/ctypes)

Someone suggested that I should start a second thread from the C-based saving code which will then ensure that C access to the hard disk will be parallel with other goings-on. So something like this:

-> python thread -> C-based LED
/
picamera -------------- ongoing picamera processing...
\
-> python thread -> C-based saving (initialize only) -> C-thread to do saving proper

I can certainly try this, but it seems overkill and I'm not sure that all these processes are running on the same thread - as to benefit from a potential late-game C thread call.

Perhaps I need to take the real-time flags/OS options more serious.

Until then, any other suggestions related to threading/multiprocessing? I can certainly try multiprocessing - but then I run into complex memory sharing problems...

Thanks.

[YCN-]

Hi,

There's also another solution, which is DMA, I'm not sure it could fit your needs, but usualy when a data transfer between one peripheral to another is too CPU intensive it is handle via DMA. Well, I don't know if that's possible between your camera and your userspace application but that could also be something to look into.
DMA is a hardware (sort of) peripheral, that means that once it's (properly) set you'll almost use no CPU to transfer your data.
May be that the raspicam already use DMA. If that's the case you could use DMA also to strobe the light, because I imagine that this is done at high frequency, and is CPU intensive ?
(you could look at here : https://github.com/Wallacoloo/Raspberry-Pi-DMA-Example
or https://github.com/hzeller/rpi-gpio-dma-demo
or even https://github.com/626Pilot/RaspberryPi-NeoPixel-WS2812 )

You can also set the priority of your threads using pthread_setschedparam (from pthread library). For this kind of heavy treatment I also wouldn't use python since it's less efficient than C (even if I don't know how much less efficient).

YCN-

[Gavinmc42]

I have similar issues even in Pascal on baremetal, writing the files to sdcard can cause issues.
So I write to a memory stream and then when finished write the stream to card.

This works for me now because it's not big files.
Doing images, I might try multicore instead of multitasking.
I have a suspicion it might be file locking related, but I not a good enough coder to figure that out

In Linux, what else is running?
256x256x60hz =240MB of raw 8bit for 10mins? Big ram drive?

Could you save data via Ethernet to another Pi that saves to HD, er USB pipe might get in the way.
Have you tried a smaller Linux like piCore? It does not write back to SDcard, runs from ram.

Are they Pi cameras? Not USB cameras?

[KnarfB]

Writing the SD card with garbage is easy from as shell. Replace the "of=" path with a path to a new file on the disk.

Code: Select all

dd if=/dev/urandom of=/path/to/disk/x bs=1M oflag=direct status=progress

or you use the 'stress' tool.
You could also monitor the free memory + buffer cache usage for a shell and watch out for low memory situations.

Code: Select all

watch free -mh

hth Frank

[YCN-]

Writing the SD card with garbage is easy from as shell. Replace the "of=" path with a path to a new file on the disk.

Code: Select all

dd if=/dev/urandom of=/path/to/disk/x bs=1M oflag=direct status=progress

Well I don't feel like a call from the program to a shell cmd could be faster than having C writing directly to a place.
But this has to be tested that's true, some time "simple" stuffs ends up being incredibly good at doing complex stuff.

YCN-

[abramq]

Maybe would be good to write frames to RAM disk and then, when finished to SD? RAM disk works very fast, there are tools to make this in Linus, maybe in RPI too?

[rleonardi]

Hi catubc,

Strobbing code is working surprisingly well ... lighting every single frame usually within 15usec or less of a new frame.

I'm trying to do something similar.

Could you possibly share your code?

Thank you.
Ciao