jacky007
Posts: 8
Joined: Mon Jun 11, 2018 7:13 am

PiNoIR v2 mode 4 1640x1232 fast capture with python+opencv

Mon Jun 11, 2018 7:47 am

Hello pi camera experts!

I'm working on a stereo camera proof of concept using 2 RPi3 and 2 PiNoIR v2 cameras. To achieve this I need full sensor output at a good resolution (ideally the 2x2 binning 1640x1232 or above) as fast as it gets. What is possible and how do I achieve that? I've been trying so many different things, that I begin to lose track of what I did and tried, but all to no avail. above 800x600 the fps really start to drop. 800x600: ~40fps, 1024x768:~30fps, 1640x1232: forget it... :)

I don't need color, as I'm using infrared light and a visible light blocking filter, so YUV and using only the luma channel would really help me to save bandwidth and unnecessary data.

I'd like to process the images with opencv (for now in python as I'm not 100% sure of the best way to process. If it works it could later be done in c++ to further improve performance)

I'm amazed at the results with raspiraw and the high framerates (courtesy of HermannSW and 6by9), but this seems all to be derived from the 640x480 mode 7 which only uses a very limited crop of the center region of the sensor.

I'd be very happy to get stable 40fps full sensor coverage @1640x1232 single channel b/w images ready to work with in opencv. Is this possible at all? What are the bottlenecks?

I tried directly opening with opencv capture, picamera, grabbing the raspivid output and basically all the promising code snippets I could find so far.

I'd be very thankful if someone could point me into the right direction or tell me that this is impossible with the hardware, that I can stop fumbling around. If the latter is the case, what can be ideally achieved? It might be ok to drop both the resolution and the framerate a bit if necessary... :)

Cheers,
Jack
PS: if I'm correct mode 4 2x2 binned 1640x1232 8bit luma channel data @40fps would be ~77MB per second. Is that realistic to handle?
Last edited by jacky007 on Mon Jun 11, 2018 12:18 pm, edited 1 time in total.

HermannSW
Posts: 752
Joined: Fri Jul 22, 2016 9:09 pm

Re: PiNoIR v2 mode 4 1640x1232 fast capture with python+opencv

Mon Jun 11, 2018 11:10 am

> I'm amazed at the results with raspiraw and the high framerates (curtesy of HermannSW and 6by9),
> but this seems all to be derived from the 640x480 mode 7 which only uses a very limited crop of the center region of the sensor.
>
Only the very high framerate modes are based on 640x480. There is a tool for every v1 or v2 camera mode:

Code: Select all

[email protected]:~ $ grep fps= fork-raspiraw/tools/1640x*
fork-raspiraw/tools/1640x1232:fps=40
fork-raspiraw/tools/1640x922:fps=40
[email protected]:~ $ 
> I'd be very happy to get stable 40fps full sensor coverage @1640x1232 single channel b/w images ready to work with in opencv.
> Is this possible at all? What are the bottlenecks?
>
Yes, without frame skips at 40fps, up to 4.3s fint into /dev/shm.
In case you process on the fly and do not store, there is no limit in recording time, see example code here (automatic camera tilt calibration)
viewtopic.php?f=43&t=189661#p1231151
Be aware that OpenCV algorithms might not be able to process at 40fps, I tried edge detection on only 320x240 frames and it was not possible to run OpenCV with more than 50fps.

Code: Select all

[email protected]:~/fork-raspiraw/t $ 1640x1232 4300
removing /dev/shm/out.*.raw
capturing frames for 4300ms with 40fps requested
172 frames were captured at 40fps
frame delta time[us] distribution
      1 
      6 24989
     19 24990
    129 24991
     14 24992
      3 24993
after skip frame indices (middle column)
0% frame skips
[email protected]:~/fork-raspiraw/t $ df /dev/shm
Filesystem     1K-blocks   Used Available Use% Mounted on
tmpfs             448336 430688     17648  97% /dev/shm
[email protected]:~/fork-raspiraw/t $ 
bookmark list: https://stamm-wilbrandt.de/en/Raspberry_camera.html

https://github.com/Hermann-SW/fork-raspiraw      https://github.com/Hermann-SW/userland
https://github.com/Hermann-SW/wireless-control-Eachine-E52-drone      https://twitter.com/HermannSW

ethanol100
Posts: 537
Joined: Wed Oct 02, 2013 12:28 pm

Re: PiNoIR v2 mode 4 1640x1232 fast capture with python+opencv

Mon Jun 11, 2018 2:07 pm

If you are using opencv and python, do you use the v4l2 kernel module or the picamera library? The v4l2 kernel module has an options to switch between still mode and video mode, this is done by default above1280x720 resolution. You can try something like "modprobe bcm2835-v4l2 max_video_width=3280 max_video_height=2464" to force video mode for all resolution. You should be able to get 40fps images from the camera.

You will not be able to save 77MB/s to any storage for a longer duration. SD cards will handle data in order of 10MB/s, USB (network adapter or storage device) will be able to handle up to around 30MB/s max. The only possible storage would be inside RAM. For processing, you will not be able to do anything meaningful at 40fps with 1640x1232 resolution, you have only 25ms time for processing. This will be even less, as the data needs to be copy around first. It would be interesting to just measure the time needed to loop over the image data and finding i.e. the maximum value, you then can try to work out how long your needed processing will take.

jacky007
Posts: 8
Joined: Mon Jun 11, 2018 7:13 am

Re: PiNoIR v2 mode 4 1640x1232 fast capture with python+opencv

Tue Jun 12, 2018 6:37 am

Hello and thanks for your answers!

@Hermann: I did not know about the other modes of raspiraw. Thanks for the notice. I installed it and got it to work a bit after a while. I can start raspiraw manually and get a preview and set some of the command line parameters. However if I try to run tools/1640x1232 it complains that it cannot acces '/dev/shm/out.*.raw', although the folder dev/shm exists.
How would I get the images loaded to the RAM and have them accessible in python to avoid the 4.3s limitation?

@ethanol100: I was not aware of the switch. This seems indeed to be a problem somehow. 960x720 runs fine at 40fps, 1024x768 (768 beinger >720) clocks in at 3.7fps or so. However I tried your suggestion with the v4l2 setting and it didn't change anything. How can I check if the modprobe changed anything?
For the processing I think I'll manage to find a feasible solution. I can play with a lot of variables to get the best result, but stable [email protected] is a good start compared to 960x720 I have to use now. I'd rather work from top down to get the best result possible. It would help though to have a single channel image right from the camera, this would save some time as well...

Thanks for your patience, I'm getting more optimistic than I was before.

Cheers,
Jack

ethanol100
Posts: 537
Joined: Wed Oct 02, 2013 12:28 pm

Re: PiNoIR v2 mode 4 1640x1232 fast capture with python+opencv

Tue Jun 12, 2018 8:42 am

It depends on how you are loading the module. If you load it automatically you can try to add

Code: Select all

bcm2835-v4l2 max_video_width=3280 max_video_height=2464
to /etc/modules

Or you first need to remove the module with rmmod bcm2835-v4l2 and then modprobe it again with parameters.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5561
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: PiNoIR v2 mode 4 1640x1232 fast capture with python+opencv

Tue Jun 12, 2018 9:44 am

As the module loads it adds a log line to dmesg similar to

Code: Select all

[    4.677107] bcm2835-v4l2: V4L2 device registered as video2 - stills mode > 1280x720
That resolution should change.

Or

Code: Select all

cat /sys/module/bcm2835_v4l2/parameters/max_video_width
cat /sys/module/bcm2835_v4l2/parameters/max_video_height
BTW I think a bug may have crept in with the latest kernels where stills mode for any raw pixel formats (eg YU12, YV12, NV12, NV21, and RGB, but not JPEG) stalls after the first buffer. I'm investigating, and it may only be on my branches, but I think some stuff has been merged (possibly to the mainline kernel only).
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

HermannSW
Posts: 752
Joined: Fri Jul 22, 2016 9:09 pm

Re: PiNoIR v2 mode 4 1640x1232 fast capture with python+opencv

Tue Jun 12, 2018 10:36 am

jacky007 wrote:
Tue Jun 12, 2018 6:37 am
However if I try to run tools/1640x1232 it complains that it cannot acces '/dev/shm/out.*.raw', although the folder dev/shm exists.
I did use /dev/shm as it got configured from Raspbian Stretch without issues.
I know of some other people using raspiraw tools, so they have no problems with /dev/shm.
Please look whether your /dev/shm is somehow different, and whether you are able to store files as I did here:

Code: Select all

[email protected]:~ $ ls -l /dev| grep shm
drwxrwxrwt 2 root root          40 Nov  3  2016 shm
[email protected]:~ $ mount | grep /dev/shm
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
[email protected]:~ $ echo "foo" > /dev/shm/bar
[email protected]:~ $ cat /dev/shm/bar 
foo
[email protected]:~ $ 
How would I get the images loaded to the RAM and have them accessible in python to avoid the 4.3s limitation?
I pointed to the C code that I modified raspiraw with to do automatic camera tilt callibration:
viewtopic.php?f=43&t=189661#p1231151

Instead of writing a frame to /dev/shm I add my own code that does the frame analysis and motor control in the raspiraw callback function, inclusive fast conversion of 640x480 raw10 Bayer frame to 320x240 grey8 frame first.

Your code needs to get called from

Code: Select all

static void callback(MMAL_PORT_T *port, MMAL_BUFFER_HEADER_T *buffer)
in case you don't put your C code there. I don't know the best way to call python from C, but it seems that time is of the essence for you in processing frames in 25ms, so maybe C should be used instead of python (OpenCV has a C api as well).
bookmark list: https://stamm-wilbrandt.de/en/Raspberry_camera.html

https://github.com/Hermann-SW/fork-raspiraw      https://github.com/Hermann-SW/userland
https://github.com/Hermann-SW/wireless-control-Eachine-E52-drone      https://twitter.com/HermannSW

jacky007
Posts: 8
Joined: Mon Jun 11, 2018 7:13 am

Re: PiNoIR v2 mode 4 1640x1232 fast capture with python+opencv

Tue Jun 12, 2018 12:28 pm

Hi!
@ethanol100, @6by9: I got the bcm2835-v4l2 video mode working for resolutions above 1280x720. Thanks to your suggestions. However, the framerates do not hold up. I measured the fps for all resolutions of true 4:3 aspect ratio betwenn xwidth of 1024px and 1900px. Up until ~1200px 40fps is possible (depending on the actual resolution), above the performance decreases (I think the problem here is hitting a bandwidth limit somewhere). You can see this in the attached image. Funny that there seem to be two alternating modes which differ quite a lot performancewise. For example 1200x900 is ~40fps, 1192x894 is ~33fps, 1208x906 is ~32fps and 1216x912 is again ~40fps. So I think picking the right resolution might be beneficial. Are there some ideal resolutions in terms of efficiency?

Just calling opencv capture gives me the 8bit 3-channel images, where I would only need one channel or the luma image. This could be a way to alleviate the problem and extend the "40 fps zone". Is there a way to do this with v4l2 and python-opencv?

Another option or additional performance booster would be to skip every second line as I'm more interested in the horizontal resolution. Is there a possibility to do that?

If neither one channel images nor lineskip are possible with the opencv interface and v4l2, I think the raspiraw way would be better.

@HermannSW: thanks for your information, my /dev/shm/ works just fine... I just needed to add a

Code: Select all

./
at the beginning of the line

Code: Select all

raspiraw -md 4 -t $1 -ts /dev/shm/tstamps.csv ...
in the /tools/1640x1232 file.

But as I see it (while this works) I'd have to figure out how to put the info in RAM and load it directly to python. C++ is of course an option, but I find it so much more convenient do develop in python... :roll: I'd first have to setup opencv for C++ and then see if I can manage to do some work on the data. But really, I'd rather not right now. It was only planned as last resort, if the performance goals cannot be met at all...

Cheers,
Jack
Attachments
graph1.png
fps performance of different 4:3 aspect ratios (mode 4)
graph1.png (67.95 KiB) Viewed 679 times

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5561
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: PiNoIR v2 mode 4 1640x1232 fast capture with python+opencv

Tue Jun 12, 2018 12:55 pm

The performance differences are likely to be down to the GPU having constraints on the image padding, but V4L2 (or more accurately a bundle of V4L2 apps) not respecting the bytesperline setting, and therefore we repack the image.
If the resolution is a multiple of 32 horizontally, and 16 vertically, then no repacking is needed, and that should deliver the maximum performance.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

jacky007
Posts: 8
Joined: Mon Jun 11, 2018 7:13 am

Re: PiNoIR v2 mode 4 1640x1232 fast capture with python+opencv

Wed Jun 13, 2018 10:16 am

Hello!
ethanol100 wrote:
Tue Jun 12, 2018 8:42 am
It depends on how you are loading the module. If you load it automatically you can try to add

Code: Select all

bcm2835-v4l2 max_video_width=3280 max_video_height=2464
to /etc/modules

Or you first need to remove the module with rmmod bcm2835-v4l2 and then modprobe it again with parameters.
I had problems loading the v4l2 drivers with parameters. Now I know how to do it:

Code: Select all

sudo nano /etc/modprobe.d/bcm2835-v4l2.conf
add

Code: Select all

options bcm2835-v4l2 max_video_width=3280 max_video_height=2464
and save. In /etc/modules *only* append bcm2835-v4l2 and save.

@6by9:
I tried the multiple of 32/16 thing and it did indeed give another little performance boost :) See red dots in updated graph below. 1248x928 is the best I can get @40fps.

I think receiving either only one channel or the luma channel of the yuv signal would really help here. Is there a way to do that with v4l2?
Attachments
graph2.png
shown in red multiple of 32/16 horz/vert
graph2.png (60.78 KiB) Viewed 618 times

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5561
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: PiNoIR v2 mode 4 1640x1232 fast capture with python+opencv

Wed Jun 13, 2018 11:30 am

jacky007 wrote:
Wed Jun 13, 2018 10:16 am
I think receiving either only one channel or the luma channel of the yuv signal would really help here. Is there a way to do that with v4l2?
It's not possible at all. Even in raspistillyuv and raspividyuv the application is passed the full YUV frame and it is the application that chooses to only look at the luma.

There are a couple of changes to the V4L2 driver in the pipeline. The main headache is that it only passes a single buffer to the GPU regardless of how many are allocated in V4L2. So when the GPU completes a frame it has to wait for the kernel to have woken up and passed the buffer back before it can continue. The change passes as many V4L2 buffers as are allocated down to the GPU (and that is the point I think I broke the stills mode raw pixel captures).
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

jacky007
Posts: 8
Joined: Mon Jun 11, 2018 7:13 am

Re: PiNoIR v2 mode 4 1640x1232 fast capture with python+opencv

Mon Jun 18, 2018 8:05 am

Hello

I started to work with raspiraw. There are however some challenges I have troubles to solve. I successfully modified raspiraw to compile with g++ c++ compiler, so that I can use the C++ opencv API. After a lot of trial and error I'm able to get some raw data in an opencv mat.

Taking only one channel works pretty fast, albeit the resolution is then @820x616. Not bad, but could be better. I can memcpy the whole raw frame in a 2080x1232 matrix, but every 5th column contains the lower 2 bits of each the previous four bytes (to make it 10bits) and thus should be removed or extracted and added to the previous ones and that takes precious time.
I can use some pointer loops to extract only the 4 bytes and skip every fifth column.
I can call raspiraw with -b 8, but that gives ugly banding results if the image contains brightness over 255 as I think that it is a dumb conversion of 10bits->8bits. And the black regions in the image are quite gray in this mode, so it throws away some possible dynamic range.

So I have several options to play with, all of them with more or less pronounced disadvantages.

However the most serious problem is that the data seems to be unreliable. I'm accessing buffer->data in the callback function and on a regular basis do get images that have been taken earlier, even days ago. There is something wrong with the buffer. I admit that I really have my problems understanding the raspiraw source in complete, as it contains already many options and possibilities that I don't need for my specific case.

Attached you see a scaled down image (1640x1232 was too big for the board apparently) that I managed to get after several calls to my raspiraw version. But there is a four in five chance that I get some garbled noise image or another one that I took last week. Really weird!

What would be the minimal working skeleton to get to the raw buffer with the PiNoIR v2 (sony chip)? Why are there some old images in the buffer or where do they come from?

Thank you for any hints that help me!

Cheers,
Jack
Attachments
img2.png
IR image @50% scale
img2.png (198.49 KiB) Viewed 430 times

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5561
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: PiNoIR v2 mode 4 1640x1232 fast capture with python+opencv

Mon Jun 18, 2018 9:00 am

-b 8 is known to be wrong. It drops the top two bits rather than shifting down so is not actually that useful.
-b 16 would give you the full resolution but packed as the bottom 10 bits of a 16 bit word.

I have seen other reports of odd buffer behaviour, but not had a chance to fully investigate. It's on the list of things to do, but a way down it.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

jacky007
Posts: 8
Joined: Mon Jun 11, 2018 7:13 am

Re: PiNoIR v2 mode 4 1640x1232 fast capture with python+opencv

Mon Jun 18, 2018 1:38 pm

Thank you for your answer and the clarification for the 8bit mode.
I'll try if the 16bit retrieval mode and subsequent bitshift -> 8bit is faster than the pointer magic. Thanks for the hint!

I further looked into the buffer behaviour and found out the following: in my modified version I first get one captured frame, then one "old" frame A, then the next new frame and finally a different "old" frame B. This sequence repeats endlessly.
If I do the same experiment with the unmodified raspiraw (and write the images to the sd card for inspection) I get a new frame, then "old" frame A and then only new frames. This behaviour would be acceptable, even if not perfect. However I have problems understanding what exactly is going on.

I do not need to write files for my processing, so it'd suffice to use raspiraw without the file output. But then the callback function won't get called as I suspect it somehow is diverted to the screen for output. For debugging it would be perfect to both see the preview on screen as well as have the buffer data available for processing. Where do I have to modify raspiraw.c?

Cheers,
Jack

jacky007
Posts: 8
Joined: Mon Jun 11, 2018 7:13 am

Re: PiNoIR v2 mode 4 1640x1232 fast capture with python+opencv

Wed Jun 20, 2018 12:19 pm

Hello!
I still didn't find a solution for the buffer problem. :? However I did some further testing and found out, that the weird frames are changing too and do not display a static image.
Using the 8bit mode I modified the buffer in place, so all the four pixel high bytes are adjacent to each other forming an image that is 1664 Pixels inside the 2080 Pixels frame (4/5th of the image). The rest of the row I set to zero=black. Then I wrote the framenumber in the black area and let it run.
In the gif below you can see what happens (if it runs, I'm not able to upload it to the forum)

I really think that raspiraw is the way to go for quick IR imaging, but only if I can get the acquired frames without every second frame being a dud.

So if anyone has any idea what I can try, I'd be very thankful!

Cheers,
Jack

PS: the odd numbered frames >=5 are the ones that work
Image

HermannSW
Posts: 752
Joined: Fri Jul 22, 2016 9:09 pm

Re: PiNoIR v2 mode 4 1640x1232 fast capture with python+opencv

Wed Jun 20, 2018 12:37 pm

jacky007 wrote:
Wed Jun 20, 2018 12:19 pm
... but only if I can get the acquired frames without every second frame being a dud ...
The 2nd frames are no duds but carry other information and have same timestamp as those carrying the data you want.
You can lookup raspiraw.c in code accessding SD card on how to filter the frames you are interested in:

Code: Select all

if (!(buffer->flags&MMAL_BUFFER_HEADER_FLAG_CODECSIDEINFO)
bookmark list: https://stamm-wilbrandt.de/en/Raspberry_camera.html

https://github.com/Hermann-SW/fork-raspiraw      https://github.com/Hermann-SW/userland
https://github.com/Hermann-SW/wireless-control-Eachine-E52-drone      https://twitter.com/HermannSW

jacky007
Posts: 8
Joined: Mon Jun 11, 2018 7:13 am

Re: PiNoIR v2 mode 4 1640x1232 fast capture with python+opencv

Wed Jun 20, 2018 12:56 pm

D'oh! I feel really stupid right now... :lol:
Apparently I was a little overenthusiastic commenting out seemingly unnecessary stuff. The first two frames are still rubbish, but the rest is fine. Marvelous!
Thank you Hermann!

Let's see what I can manage to get performance-wise now.
Cheers,
Jack

ethanol100
Posts: 537
Joined: Wed Oct 02, 2013 12:28 pm

Re: PiNoIR v2 mode 4 1640x1232 fast capture with python+opencv

Wed Jun 20, 2018 1:28 pm

Wouldn't it be easier to modifying raspividyuv? This can output only the y from yuv and you could directly use this allocated buffer for a opencv Mat object? I don't know how much effort it would take to change this to c++ and add opencv there.
see: raspividyuv

Or you could pipe the output from raspividyuv into a program to analyse it, if you know the size you could just fill the buffer from stdin?

Return to “Camera board”

Who is online

Users browsing this forum: No registered users and 15 guests