Joose
Posts: 9
Joined: Fri Mar 21, 2014 11:10 pm

Re: New V4L2 driver and OpenCV

Fri Mar 28, 2014 3:39 pm

6by9 wrote:<slightly grumpy>...</slightly grumpy>
Sowwy :(
I had tried his code previously and it didn't work either, it did work now though, I guess running rpi-update helped.
Now I'm running in to the problem that if I run the code it works once after reboot, but the second time always fails with

Code: Select all

VIDIOC_STREAMON: Operation not permitted
and dmesg tells me

Code: Select all

[  150.532044] bcm2835-v4l2: error 0 waiting for frame completion
[  166.402420] bcm2835_v4l2: error 0 waiting for sync completion
[  166.402462] bcm2835-v4l2: Failed to enable encode tunnel - error -62
Also after the second time the camera light stays on and never turns off. Only way to get it working again is to power cycle my Pi.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 10823
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: New V4L2 driver and OpenCV

Fri Mar 28, 2014 9:10 pm

Joose wrote:Sowwy :(
I had tried his code previously and it didn't work either, it did work now though, I guess running rpi-update helped.
I'm sorry. You caught me on a bad morning and seemed to be going over stuff that has been covered already.
rpi-update should always be the first port of call as we are still developing the V4L2 driver (and the rest of Raspbian). There is an outside chance that things could get worse, but we do try hard not to introduce regressions. There have been the odd issues where extending the functionality has revealed bugs in 3rd party software though (the current fun with GStreamer being one).
Joose wrote:Now I'm running in to the problem that if I run the code it works once after reboot, but the second time always fails with

Code: Select all

VIDIOC_STREAMON: Operation not permitted
and dmesg tells me

Code: Select all

[  150.532044] bcm2835-v4l2: error 0 waiting for frame completion
[  166.402420] bcm2835_v4l2: error 0 waiting for sync completion
[  166.402462] bcm2835-v4l2: Failed to enable encode tunnel - error -62
Also after the second time the camera light stays on and never turns off. Only way to get it working again is to power cycle my Pi.
Sounds like I need to try that one out. Hitting situations with "error 0 waiting for frame completion" and the LED stays on are bad news (the GPU has got wedged) and need to be solved. Being GPU side, that means it needs to be JamesH or I.
I'm no OpenCV expert (far from it having never even tried to run it before) - what do I need to install to try your/pondruska's python code?
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Joose
Posts: 9
Joined: Fri Mar 21, 2014 11:10 pm

Re: New V4L2 driver and OpenCV

Fri Mar 28, 2014 10:51 pm

6by9 wrote: I'm sorry. You caught me on a bad morning and seemed to be going over stuff that has been covered already.
rpi-update should always be the first port of call as we are still developing the V4L2 driver (and the rest of Raspbian).
No worries.
I am aware that the firmware keeps updating regurarly, though I'm still not certain if it was the latest version that fixed it or if it was just coincidence.
6by9 wrote: Sounds like I need to try that one out. Hitting situations with "error 0 waiting for frame completion" and the LED stays on are bad news (the GPU has got wedged) and need to be solved. Being GPU side, that means it needs to be JamesH or I.
I'm no OpenCV expert (far from it having never even tried to run it before) - what do I need to install to try your/pondruska's python code?
I am using OpenCV 2.4.8 which is not available on the Raspbian repositories yet, it does take hours to compile on the Raspberry though. The way I set it up was following these steps (started from a clean install of Raspbian):

Code: Select all

sudo apt-get install build-essential cmake pkg-config python-dev libgtk2.0-dev libgtk2.0 zlib1g-dev libpng-dev libjpeg-dev libtiff-dev libjasper-dev libavcodec-dev swig

wget http://downloads.sourceforge.net/project/opencvlibrary/opencv-unix/2.4.8/opencv-2.4.8.zip
unzip opencv-2.4.8.zip
cd opencv-2.4.8

cmake -DCMAKE_BUILD_TYPE=RELEASE -DCMAKE_INSTALL_PREFIX=/usr/local -DBUILD_PERF_TESTS=OFF -DBUILD_opencv_gpu=OFF -DBUILD_opencv_ocl=OFF
make
sudo make install
Also there seems to be about a 2-3 second lag in the live camera view, even with the default size of 640x480.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 10823
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: New V4L2 driver and OpenCV

Mon Mar 31, 2014 10:59 am

Joose wrote:I am using OpenCV 2.4.8 which is not available on the Raspbian repositories yet, it does take hours to compile on the Raspberry though. The way I set it up was following these steps (started from a clean install of Raspbian):

Code: Select all

sudo apt-get install build-essential cmake pkg-config python-dev libgtk2.0-dev libgtk2.0 zlib1g-dev libpng-dev libjpeg-dev libtiff-dev libjasper-dev libavcodec-dev swig

wget http://downloads.sourceforge.net/project/opencvlibrary/opencv-unix/2.4.8/opencv-2.4.8.zip
unzip opencv-2.4.8.zip
cd opencv-2.4.8

cmake -DCMAKE_BUILD_TYPE=RELEASE -DCMAKE_INSTALL_PREFIX=/usr/local -DBUILD_PERF_TESTS=OFF -DBUILD_opencv_gpu=OFF -DBUILD_opencv_ocl=OFF
make
sudo make install
Well that will keep my Pi busy for a while then.

Once built, are you just using that little snippet of Python code to actually do the capture? Nothing else needed? I'll give it a whirl.
Joose wrote:Also there seems to be about a 2-3 second lag in the live camera view, even with the default size of 640x480.
That is unlikely to be the GPU or V4L2 driver. They run on very low latency (typically less than a frame). Converting to RGB does take a little while, but we're still only talking a couple of milliseconds at 640x480. Unless it gets too involved, I'm not intending to get involved in OpenCV in any depth - there are so many potential client apps for V4L2 and I just don't have time to look into all of them.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Joose
Posts: 9
Joined: Fri Mar 21, 2014 11:10 pm

Re: New V4L2 driver and OpenCV

Mon Mar 31, 2014 3:21 pm

6by9 wrote: Once built, are you just using that little snippet of Python code to actually do the capture? Nothing else needed? I'll give it a whirl.
This is the code I used:

Code: Select all

#!/usr/bin/python
import cv2.cv as cv
import time
import os

# Load the camera driver
os.system("sudo modprobe bcm2835-v4l2")

# Wait for modprobe
time.sleep(2)

capture = cv.CaptureFromCAM(0)
while True:
	# Grab one frame from camera
	frame = cv.QueryFrame(capture)

	# Display frame in a window
	cv.ShowImage( "image", frame )

	# Wait for ESC to end program
	c = cv.WaitKey(7) % 0x100
	if c == 27:
		break
cv.DestroyAllWindows()
6by9 wrote:That is unlikely to be the GPU or V4L2 driver. They run on very low latency (typically less than a frame). Converting to RGB does take a little while, but we're still only talking a couple of milliseconds at 640x480. Unless it gets too involved, I'm not intending to get involved in OpenCV in any depth - there are so many potential client apps for V4L2 and I just don't have time to look into all of them.
Yeah I guess it could be up to the implementation of V4L2 in OpenCV, though from what I've heard other people have used USB webcams succesfully on a Raspberry Pi with OpenCV without much lag, albeit with lower resolutions (eg. 320x240). I can't test that myself though since I don't own a working webcam currently.
Also understandably you have very limited time for these things, but I have to say OpenCV is one of the more useful pieces of software out there.

I did manage to get the capture working with much higher FPS and without lag using a Python library called Picamera, I guess I can just fall back to that instead since it seems to be readily functioning.
However if you wish to investigate this further I'm attaching an example code of the same thing done with Picamera library for reference. Picamera is readily available on the repositories, install with apt-get install python-picamera

Code: Select all

#!/usr/bin/python
import io
import picamera
import cv2
import numpy as np
import sys

# Create the in-memory stream
stream = io.BytesIO()

# Open connection to camera
with picamera.PiCamera() as camera:
	# Set camera resolution
	camera.resolution = (640, 480)

	# Start loop
	while True:
		# Capture image from camera
		camera.capture(stream, format='jpeg', use_video_port=True)

		# Convert image from camera to a numpy array
		data = np.fromstring(stream.getvalue(), dtype=np.uint8)

		# Decode the numpy array image
		image = cv2.imdecode(data, cv2.CV_LOAD_IMAGE_COLOR)

		# Empty and return the in-memory stream to beginning
		stream.seek(0)
		stream.truncate(0)

		# Display the image
		cv2.imshow('image', image)

		# Wait for ESC to end program
		c = cv2.waitKey(7) % 0x100
		if c == 27:
			break
cv2.destroyAllWindows()

Joose
Posts: 9
Joined: Fri Mar 21, 2014 11:10 pm

Re: New V4L2 driver and OpenCV

Mon Mar 31, 2014 3:27 pm

Also for the V4L2 version CPU usage seems to be maxed at 100%, and the Picamera version only uses about 50-60% and as mentioned before the V4L2 version only works once after powering up, the camera light gets stuck and I have to do a power cycle, reboot doesn't work.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 10823
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: New V4L2 driver and OpenCV

Mon Mar 31, 2014 4:31 pm

6 hours in, and it's still compiling - 88% done. One kernel panic in the mix too due to the swap file going nuts! Shame it won't cross-compile easily (I did try, but it wants oodles of libraries prebuilt)
Joose wrote:Also for the V4L2 version CPU usage seems to be maxed at 100%, and the Picamera version only uses about 50-60% and as mentioned before the V4L2 version only works once after powering up, the camera light gets stuck and I have to do a power cycle, reboot doesn't work.
It sounds like they haven't been terribly smart with their V4L2 interface.
Motion was also poor in that regard. It didn't set the V4L2 framerate, but potentially did a format conversion on every frame before dropping it to achieve the desired framerate.
I'll have a dig once things are built, but sadly the time pressures from other projects are building.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 10823
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: New V4L2 driver and OpenCV

Tue Apr 01, 2014 11:14 am

(Restarting the compile as I've reimaged my SD card. Even more gutting as it turned out I didn't need to, I'd just created a duff GPU firmware :( ).
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 10823
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: New V4L2 driver and OpenCV

Wed Apr 02, 2014 5:03 pm

It finished compiling!
So your test app is running OK.
The latency is not the GPU. From a terminal prompt run

Code: Select all

v4l2-ctl --overlay=1
and it'll add an overlay to the screen direct from the V4L2 driver. No latency there, so OpenCV is adding the delay.
v4l2-ctl -V tells you the format it is using

Code: Select all

Format Video Capture:
        Width/Height  : 640/480
        Pixel Format  : 'MJPG'
        Field         : None
        Bytes per Line: 640
        Size Image    : 307200
        Colorspace    : JPEG (JFIF/ITU601)
so it's having to JPEG decode the frames before doing anything with them - told you it would be doing something silly! That also partly explains why it maxes out the ARM.
It also gives me a pointer as to what is going wrong on the GPU - the MJPEG codec isn't used other than on RPi, so it's the likely candidate.

Real work is getting in the way of me investigating this, but I will keep it in mind.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Joose
Posts: 9
Joined: Fri Mar 21, 2014 11:10 pm

Re: New V4L2 driver and OpenCV

Thu Apr 03, 2014 1:06 am

Fair enough, I guess I'll go with the Picamera implementation for now then. Seems to work without too many problems.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 10823
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: New V4L2 driver and OpenCV

Thu Apr 03, 2014 11:15 am

Just had a look at the V4L2 setup code from OpenCV in opencv-2.4.8/modules/highgui/src/cap_v4l.cpp

Code: Select all

  if (try_palette_v4l2(capture, V4L2_PIX_FMT_BGR24) == 0)
  {
    capture->palette = PALETTE_BGR24;
  }
  else
  if (try_palette_v4l2(capture, V4L2_PIX_FMT_YVU420) == 0)
  {
    capture->palette = PALETTE_YVU420;
  }
  else
  if (try_palette_v4l2(capture, V4L2_PIX_FMT_YUV411P) == 0)
  {
    capture->palette = PALETTE_YUV411P;
  }
  else

#ifdef HAVE_JPEG
  if (try_palette_v4l2(capture, V4L2_PIX_FMT_MJPEG) == 0 ||
      try_palette_v4l2(capture, V4L2_PIX_FMT_JPEG) == 0)
  {
    capture->palette = PALETTE_MJPEG;
  }
  else
#endif

  if (try_palette_v4l2(capture, V4L2_PIX_FMT_YUYV) == 0)
  {
    capture->palette = PALETTE_YUYV;
  }
  else if (try_palette_v4l2(capture, V4L2_PIX_FMT_UYVY) == 0)
  {
    capture->palette = PALETTE_UYVY;
  }
  else
  if (try_palette_v4l2(capture, V4L2_PIX_FMT_SN9C10X) == 0)
  {
    capture->palette = PALETTE_SN9C10X;
  } else
  if (try_palette_v4l2(capture, V4L2_PIX_FMT_SBGGR8) == 0)
  {
    capture->palette = PALETTE_SBGGR8;
  } else
  if (try_palette_v4l2(capture, V4L2_PIX_FMT_SGBRG) == 0)
  {
    capture->palette = PALETTE_SGBRG;
  }
      else
  {
    fprintf(stderr, "HIGHGUI ERROR: V4L2: Pixel format of incoming image is unsupported by OpenCV\n");
    icvCloseCAM_V4L(capture);
    return -1;
  }

  return 0;
We support RGB24, not BGR24 (after I corrected the typo and people were complaining of R & B being swapped). I did look at supporting BGR24 on the GPU but it isn't trivial to keep performance up. There was the possibility of doing it as a two pass operation (convert YUV to RGB24, and then swap to BGR24), but I ran out of time to investigate.
We support YUV420, not YVU420. It's a trivial fix on the GPU, although it would also be a trivial fix in OpenCV too. It always calls yuv420_to_rgb24 and just needs to swap pU and pV before doing the conversion - I'm just trying a quick mod to that now (it's almost lunchtime!).

I will have a further play with this, but can't make promises on timescales.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 10823
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: New V4L2 driver and OpenCV

Thu Apr 03, 2014 12:41 pm

Sort of still lunchtime :)

One patch for OpenCV to support YUV420. Latency significantly reduced, although the ARM is still maxed out. OpenCV forces the framerate to 30fps, which doesn't help.
You can run v4l2-ctl -p 6 once your app is running to drop the driver to 6fps and ARM to about 50% with almost no visible difference to the display. Don't drop it below 3 as it shows up a bug where I'm not changing the GPU timeout, so a frame time exceeding 400ms (2.5fps) will cause the GPU to start timing out.

Code: Select all

From c86add1abed85400b0e52cdcbb9b5de8a8a6307a Mon Sep 17 00:00:00 2001
From: Dave Stevenson <dsteve@broadcom.com>
Date: Thu, 3 Apr 2014 11:42:01 +0000
Subject: [PATCH] Add support for V4L2_PIXEL_FORMAT_YUV420

Handling is almost identical as for YVU420, but the
computation of the pointer pU and pV is modified in
yuv420p_to_rgb24
---
 modules/highgui/src/cap_v4l.cpp |   36 +++++++++++++++++++++++++++++++-----
 1 file changed, 31 insertions(+), 5 deletions(-)

diff --git a/modules/highgui/src/cap_v4l.cpp b/modules/highgui/src/cap_v4l.cpp
index c9fca05..9053c79 100644
--- a/modules/highgui/src/cap_v4l.cpp
+++ b/modules/highgui/src/cap_v4l.cpp
@@ -288,6 +288,7 @@ static unsigned int n_buffers = 0;
 enum PALETTE_TYPE {
   PALETTE_BGR24 = 1,
   PALETTE_YVU420,
+  PALETTE_YUV420,
   PALETTE_YUV411P,
   PALETTE_YUYV,
   PALETTE_UYVY,
@@ -551,6 +552,10 @@ static int autosetup_capture_mode_v4l2(CvCaptureCAM_V4L* capture)
   {
     capture->palette = PALETTE_YVU420;
   }
+  if (try_palette_v4l2(capture, V4L2_PIX_FMT_YUV420) == 0)
+  {
+    capture->palette = PALETTE_YUV420;
+  }
   else
   if (try_palette_v4l2(capture, V4L2_PIX_FMT_YUV411P) == 0)
   {
@@ -1496,15 +1501,26 @@ move_411_block(int yTL, int yTR, int yBL, int yBR, int u, int v,
 /* Converts from planar YUV420P to RGB24. */
 static void
 yuv420p_to_rgb24(int width, int height,
-           unsigned char *pIn0, unsigned char *pOut0)
+           unsigned char *pIn0, unsigned char *pOut0,
+           bool is_yuv)
 {
     const int numpix = width * height;
     const int bytes = 24 >> 3;
     int i, j, y00, y01, y10, y11, u, v;
     unsigned char *pY = pIn0;
-    unsigned char *pU = pY + numpix;
-    unsigned char *pV = pU + numpix / 4;
+    unsigned char *pU;
+    unsigned char *pV;
     unsigned char *pOut = pOut0;
+    if (is_yuv)
+    {
+        pU = pY + numpix;
+        pV = pU + numpix / 4;
+    }
+    else
+    {
+        pV = pY + numpix;
+        pU = pV + numpix / 4;
+    }
 
     for (j = 0; j <= height - 2; j += 2) {
         for (i = 0; i <= width - 2; i += 2) {
@@ -2160,7 +2176,16 @@ static IplImage* icvRetrieveFrameCAM_V4L( CvCaptureCAM_V4L* capture, int) {
         yuv420p_to_rgb24(capture->form.fmt.pix.width,
                  capture->form.fmt.pix.height,
                  (unsigned char*)(capture->buffers[capture->bufferIndex].start),
-                 (unsigned char*)capture->frame.imageData);
+                 (unsigned char*)capture->frame.imageData,
+                 false);
+        break;
+
+    case PALETTE_YUV420:
+        yuv420p_to_rgb24(capture->form.fmt.pix.width,
+                 capture->form.fmt.pix.height,
+                 (unsigned char*)(capture->buffers[capture->bufferIndex].start),
+                 (unsigned char*)capture->frame.imageData,
+                 true);
         break;
 
     case PALETTE_YUV411P:
@@ -2239,7 +2264,8 @@ static IplImage* icvRetrieveFrameCAM_V4L( CvCaptureCAM_V4L* capture, int) {
         yuv420p_to_rgb24(capture->captureWindow.width,
              capture->captureWindow.height,
              (unsigned char*)(capture->memoryMap + capture->memoryBuffer.offsets[capture->bufferIndex]),
-             (unsigned char*)capture->frame.imageData);
+             (unsigned char*)capture->frame.imageData,
+             true);
         break;
     case VIDEO_PALETTE_YUV420:
         yuv420_to_rgb24(capture->captureWindow.width,
-- 
1.7.10.4
It looks to me like their handling of YVU420 is incorrect, as it was calculating the pointers as if it were YUV420. Probably worth raising a bug report on OpenCV - I don't know what hardware they used to test YVU420.

BTW Rather than downloading the the OpenCV source ZIP file from sourceforge, you're probably better off downloading from github with

Code: Select all

git clone https://github.com/Itseez/opencv.git
2.4.8 was released on 30th December 2013, and there have been changes since (including the specific fix for RaspberryPi discussed above, however that is now partially redundant as we support manual AWB).
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Joose
Posts: 9
Joined: Fri Mar 21, 2014 11:10 pm

Re: New V4L2 driver and OpenCV

Thu Apr 03, 2014 3:26 pm

I applied your patch and recompiled but I can't actually test the code since it seems I have managed to fry either the camera board or the CSI connector. Oops. Ohwell, Amazon Prime to the rescue, I'll continue testing tomorrow.

magnatag
Posts: 33
Joined: Tue Mar 04, 2014 8:39 pm

Re: New V4L2 driver and OpenCV

Thu Apr 03, 2014 8:39 pm

I would be very interested in seeing this fix work as I am working closely with RPi and OpenCV at the moment. At 640x380 I am getting about 1-2s lag just as Joose stated above.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 10823
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: New V4L2 driver and OpenCV

Thu Apr 03, 2014 9:03 pm

It's not zero latency, but it is certainly less than when it was running MJPEG.
I'll try to find time tomorrow to run the V4L2 overlay alongside the OpenCV image to get a better idea of what the delay actually is. If I can wangle BGR out of the GPU then it's just possible that will reduce it further.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

magnatag
Posts: 33
Joined: Tue Mar 04, 2014 8:39 pm

Re: New V4L2 driver and OpenCV

Fri Apr 04, 2014 1:19 pm

Thank you for taking the time to dig into this issue 6by9! Myself and other opencv users really appreciate your help!

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 10823
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: New V4L2 driver and OpenCV

Fri Apr 04, 2014 1:44 pm

No problems - anyone mentioning GPU lockups from the camera will get my attention.

This one looks like it was down to the MJPEG. I know there are various issues in the MJPEG codec, but it is well down my list of priorities to look at. The better solution in this case is to get one of the raw RGB or YUV pixel formats working as it makes life easier all round.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 10823
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: New V4L2 driver and OpenCV

Fri Apr 04, 2014 3:33 pm

I love it when a plan comes together!

I have changes for BGR888 :) Testing them with Joose's test code I get that mode selected, no R/B swapping, but latency :(

Run the command v4l2-ctl -p 7 (or any number below about 10 really) whilst the test code is running, the ARM utilisation drops below 100%, and the latency disappears (well given a second for buffer queues to empty).
Using v4l2-ctl --overlay=1 to bring up the GPU rendered overlay and there is no significant difference between the two :D

Sadly there appears to be no way to change the requested framerate programmatically from OpenCV, the only place it is set is from modules/highgui/src/cap_v4l.cpp, function icvSetVideoSize

Code: Select all

    /* try to set framerate to 30 fps */
    struct v4l2_streamparm setfps;
    memset (&setfps, 0, sizeof(struct v4l2_streamparm));
    setfps.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
    setfps.parm.capture.timeperframe.numerator = 1;
    setfps.parm.capture.timeperframe.denominator = 30;
    ioctl (capture->deviceHandle, VIDIOC_S_PARM, &setfps);
Not much I can do about that one, but if you're happy to recompile OpenCV then a quick tweak of that 30 should see you good. If there are discussions with OpenCV people, then being able to set that programmatically would be rather useful - I don't know how receptive they are to requests or others taking on developments such as this.

I will bite the bullet and get these changes pushed. This change depends on my ongoing changes to remove the padding from the images too, so it's a larger change than it could have been. How much can really break?!
When Dom will release them is another matter, but he's normally pretty quick. It's both a firmware and V4L2 update.

*edit* Checked with Dom, he's planning on doing an update tonight. A post will be made on https://twitter.com/RPF_Dev_Updates when it's done.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

magnatag
Posts: 33
Joined: Tue Mar 04, 2014 8:39 pm

Re: New V4L2 driver and OpenCV

Mon Apr 07, 2014 4:00 pm

Wow 6by9! What a terrific job! I ran one of my programs with the os.system(v4l2-ctl -p 7) command (after rpi-update of course) and there is barely any latency! Amazing! Thank you very much for your hard work!

Now there is no way to extract a gray channel directly from the camera now is there? I've been doing bgr to greyscale conversion and it works okay but I'm always looking for more efficient way of doing things.

Thanks again!

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 10823
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: New V4L2 driver and OpenCV

Mon Apr 07, 2014 4:56 pm

magnatag wrote:Now there is no way to extract a gray channel directly from the camera now is there? I've been doing bgr to greyscale conversion and it works okay but I'm always looking for more efficient way of doing things.
Easy, but possibly not with OpenCV.

Grey is the same as the luma (Y) channel from a YUV image. To a first approximation, the chroma channels are the difference from the luma for each of the red and blue channels. Turn to Wiki for a more detailed explanation - http://en.wikipedia.org/wiki/YUV

Ask the V4L2 driver for V4L2_PIX_FMT_YUV420, V4L2_PIX_FMT_YVU420, V4L2_PIX_FMT_NV21 or V4L2_PIX_FMT_NV12 and you'll get a single buffer that contains the three components. The luma (Y) is always first, and will be (width * height) bytes in size, 8bits/pixel - that's the bit you want.
The chroma will follow in the buffer, and the layout in memory will depend on the format chosen (YUV420 will have (w/2 x h/2) pixels of U, and then the same of V. YVU420 likewise but V first, then U. V4L2_PIX_FMT_NV12 has a single plane with effectively 16 bit values holding the U and V values for each location. NV21 is the same but with U & V swapped).

The awkward bit is how OpenCV handles V4L2. It goes back to modules/highgui/src/cap_v4l.cpp again, where autosetup_capture_mode_v4l2 will try formats in order. Once a format is chosen, icvRetrieveFrameCAM_V4L will handle the buffers as they are generated, and always converts them to RGB888 :(

To get what you want, you'll need to tell that code that you want it to produce greyscale (however that is represented in OpenCV), and then setup V4L2 appropriately, and have conversion functions as needed. I know we had some similar discussions on http://www.raspberrypi.org/forums/viewt ... 43&t=71400
You seemed to have found some properties that could be set on the capture components, so it may be possible to add the required signalling, but it may be a bigger job than you intend, and is likely to be in C rather than Python. There may be some friendly people around who may be able to help out though - this is common code to any Linux users of OpenCV, not just those on RaspberryPi. I'm happy to provide some advice, but don't have the time to dive in myself.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Return to “Camera board”