6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 4356
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Official V4L2 driver

Sat Jul 11, 2015 9:20 pm

airsports wrote:Is V4L2 driver supported with fully updated raspbian (with rpi-update)?
I was using a working setup of raspberry camera with v4l2 driver + motion software. It stopped working after I ran "rpi-update". I think there was a message during the update warning that this update is rather major and could break things.
One of the very latest releases had a change to try and sort a caching issue on transfers from the GPU due to 2836 having different cache line sizes (https://github.com/raspberrypi/firmware/issues/443). There's one bug report already logged and being investigated that it seems to have broken the V4L2 driver - https://github.com/raspberrypi/linux/issues/1055
airsports wrote:I am running this camera for security surveillance, would appreciate any suggestions on how to restore this camera to working condition.
"sudo rpi-update 19debde" should get you the previous version before that change. The regression will be investigated as soon as the relevant people have a chance.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 4356
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Official V4L2 driver

Wed Jul 15, 2015 5:25 pm

6by9 wrote:
airsports wrote:Is V4L2 driver supported with fully updated raspbian (with rpi-update)?
I was using a working setup of raspberry camera with v4l2 driver + motion software. It stopped working after I ran "rpi-update". I think there was a message during the update warning that this update is rather major and could break things.
One of the very latest releases had a change to try and sort a caching issue on transfers from the GPU due to 2836 having different cache line sizes (https://github.com/raspberrypi/firmware/issues/443). There's one bug report already logged and being investigated that it seems to have broken the V4L2 driver - https://github.com/raspberrypi/linux/issues/1055
Just for reference, this issue has been fixed with the latest releases.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 4356
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Official V4L2 driver

Wed Jul 15, 2015 11:08 pm

byterg wrote:
6by9 wrote:MMAL_PARAMETER_INPUT_CROP is using relative values, not absolute pixel counts, so a rectangle of 0,0,0x1000,0x1000 is the full field of view, having cropped the image to the output aspect ratio. This was the reason that I hadn't tackled it before - the way the Broadcom GPU code works didn't seem to quite map onto the V4L2 one, and I ran out of time. I'm not sure whether that has actually changed, or my previous reading was wrong. Perhaps I was looking to implement the selection API for cropping, composing, and scaling - I can't quite remember now.

Adding in clipping to the hardware limits, I think what you want is something along the lines of:

Code: Select all

mmal_rect.x = (min(v4l2.left, res->width) << 16 ) /res->width;
mmal_rect.y = (min(v4l2.top, res->height)  << 16 ) /res->height;
mmal_rect.w = (min(v4l2.width, res->width - v4l2.left) << 16) / res->width;
mmal_rect.h = (min(v4l2.height, res->height - v4l2.top) << 16 ) / res->height);
Thank you, 6by9!
Questions:
- you are sure in frame size limits 0x1000
- why you use shift on 16 if frame size limit is 0x1000
Thank you and excuse my bad english.

P.S. I try use your example, unfortunately result image not croped.
I've just pushed my work in progress on this to https://github.com/6by9/linux/tree/rpi-4.0.y
For some reason I'd started with your earlier patch, not the later one. With my tree I can use commands such as v4l2-ctl --set-crop=top=200,left=400,width=512,height=368 to crop sections out of the sensor.
Still things to tidy up though - the default is 0,0,0,0 which is wrong, and I don't know what the correct handling is when you change resolution.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

ricberw
Posts: 15
Joined: Mon Jul 06, 2015 10:16 pm

Re: Official V4L2 driver

Thu Jul 16, 2015 1:51 am

Is it possible to use MMAL_PARAM_EXPOSUREMETERINGMODE_MATRIX within the current version of the v4l2 driver/v4l2-ctl program in a hacky way?

I found it in the git repo, and I'm trying to build the latest version after a few edits, but am not sure if that is going to work.

(after further searching, it seems as though it is listed in v4l2-ctl as a mode, but the -c exposure_metering_mode range is still limited to 0-2, which doesn't include the "3" that would prompt it to use the matrix mode -- thoughts?)

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 4356
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Official V4L2 driver

Thu Jul 16, 2015 6:41 am

ricberw wrote:Is it possible to use MMAL_PARAM_EXPOSUREMETERINGMODE_MATRIX within the current version of the v4l2 driver/v4l2-ctl program in a hacky way?

I found it in the git repo, and I'm trying to build the latest version after a few edits, but am not sure if that is going to work.

(after further searching, it seems as though it is listed in v4l2-ctl as a mode, but the -c exposure_metering_mode range is still limited to 0-2, which doesn't include the "3" that would prompt it to use the matrix mode -- thoughts?)
Doesn't need to be hacky.
https://github.com/raspberrypi/linux/bl ... ols.c#L387
/* todo matrix weighting not added to Linux API till 3.9
Seeing as we're beyond 3.9 now, it can be added cleanly. Something like:

Code: Select all

diff --git a/drivers/media/platform/bcm2835/controls.c b/drivers/media/platform/bcm2835/controls.c
index 71f39d7..4c5e2e7 100644
--- a/drivers/media/platform/bcm2835/controls.c
+++ b/drivers/media/platform/bcm2835/controls.c
@@ -384,11 +384,9 @@ static int ctrl_set_metering_mode(struct bm2835_mmal_dev *dev,
                dev->metering_mode = MMAL_PARAM_EXPOSUREMETERINGMODE_SPOT;
                break;

-       /* todo matrix weighting not added to Linux API till 3.9
        case V4L2_EXPOSURE_METERING_MATRIX:
                dev->metering_mode = MMAL_PARAM_EXPOSUREMETERINGMODE_MATRIX;
                break;
-       */

        }

@@ -1006,7 +1004,7 @@ static const struct bm2835_mmal_v4l2_ctrl v4l2_ctrls[V4L2_CTRL_COUNT] = {
        {
                V4L2_CID_EXPOSURE_METERING,
                MMAL_CONTROL_TYPE_STD_MENU,
-               ~0x7, 2, V4L2_EXPOSURE_METERING_AVERAGE, 0, NULL,
+               ~0xF, 3, V4L2_EXPOSURE_METERING_AVERAGE, 0, NULL,
                MMAL_PARAMETER_EXP_METERING_MODE,
                &ctrl_set_metering_mode,
                false
That seems to build, but I haven't checked it works.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

ricberw
Posts: 15
Joined: Mon Jul 06, 2015 10:16 pm

Re: Official V4L2 driver

Thu Jul 16, 2015 4:34 pm

It'll be my first time compiling the kernel, but I'm in-process. If you have the compiled bcm2835-v4l2.ko for the most recent kernel with those changes made, would you mind sending it over? That'd be super helpful so I can play with it before my build finishes (probably 5-6 hours).

Thanks a ton :)

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 4356
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Official V4L2 driver

Thu Jul 16, 2015 5:12 pm

ricberw wrote:It'll be my first time compiling the kernel, but I'm in-process. If you have the compiled bcm2835-v4l2.ko for the most recent kernel with those changes made, would you mind sending it over? That'd be super helpful so I can play with it before my build finishes (probably 5-6 hours).
Cross-compiling is MUCH faster, and the instructions do work. It does mean you need a Linux PC to work with though.

I probably have the modules built at home - I'll try to push them somewhere (probably github) later this evening, and assuming that it works I'll create a pull request so it gets merged into the official releases.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 4356
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Official V4L2 driver

Thu Jul 16, 2015 8:02 pm

6by9 wrote:
ricberw wrote:It'll be my first time compiling the kernel, but I'm in-process. If you have the compiled bcm2835-v4l2.ko for the most recent kernel with those changes made, would you mind sending it over? That'd be super helpful so I can play with it before my build finishes (probably 5-6 hours).
Cross-compiling is MUCH faster, and the instructions do work. It does mean you need a Linux PC to work with though.

I probably have the modules built at home - I'll try to push them somewhere (probably github) later this evening, and assuming that it works I'll create a pull request so it gets merged into the official releases.
https://github.com/6by9/RPiTest/blob/ma ... o?raw=true
v4l2-ctl --list-ctrls-menu now lists

Code: Select all

         exposure_metering_mode (menu)   : min=0 max=3 default=0 value=0
                                0: Average
                                1: Center Weighted
                                2: Spot
                                3: Matrix
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

ricberw
Posts: 15
Joined: Mon Jul 06, 2015 10:16 pm

Re: Official V4L2 driver

Thu Jul 16, 2015 10:46 pm

Fantastic! Working perfectly for me.

My build failed (when using modprobe to load the driver, it throws: "ERROR: could not insert 'bcm2835_v4l2': Exec format error"), so I'll have to keep tinkering to get it working so I can make a few more modifications.

Thank you :)

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 4356
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Official V4L2 driver

Fri Jul 17, 2015 8:24 am

ricberw wrote:My build failed (when using modprobe to load the driver, it throws: "ERROR: could not insert 'bcm2835_v4l2': Exec format error"), so I'll have to keep tinkering to get it working so I can make a few more modifications.
Normally means that the module doesn't match the running kernel version. dmesg sometimes has some useful information, but often not. uname -a to confirm the running kernel version. Did you switch to the kernel you built, or just push the one module?
Glad you got it working with my build though.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

ricberw
Posts: 15
Joined: Mon Jul 06, 2015 10:16 pm

Re: Official V4L2 driver

Fri Jul 24, 2015 1:30 am

Ahh, just ran into another problem:

Linux raspberrypi 4.0.8+ #804 PREEMPT Tue Jul 14 12:57:44 BST 2015 armv6l GNU/Linux << this is the kernel I'm on that it was working on previously

I moved the card from my Raspberry Pi 2 B to a Raspberry Pi B+, and now it doesn't work anymore :/

Any ideas?

User avatar
rpdom
Posts: 11391
Joined: Sun May 06, 2012 5:17 am
Location: Essex, UK

Re: Official V4L2 driver

Fri Jul 24, 2015 5:26 am

The B+ and the 2B use different kernels.

When using the 2B you should see a "v7+" on the end of the kernel version like this (mine is an old kernel)

Code: Select all

Linux raspi5 3.18.11-v7+ #781 SMP PREEMPT Tue Apr 21 18:07:59 BST 2015 armv7l
while on the B+ you would see

Code: Select all

Linux raspi3 3.18.11+ #781 PREEMPT Tue Apr 21 18:02:18 BST 2015 armv6l
The 2B will use the file "kernel7.img", while the B+ uses "kernel.img". There needs to be two directories in /lib/modules, one for each kernel version

Code: Select all

pi@raspi3 ~ $ ls /lib/modules
3.18.11+  3.18.11-v7+
If you build your own kernel/modules and you are using a 2B and a B+ you need to build both versions.

ricberw
Posts: 15
Joined: Mon Jul 06, 2015 10:16 pm

Re: Official V4L2 driver

Fri Jul 24, 2015 4:15 pm

Got it - and I managed to re-compile myself and it worked!

I have one last issue that is preventing me from using the full 15fps now for I420: even with a manual exposure setting and AWB being off/using manual gain settings, I still get variation in exposure and white balance between the first shot (which is normally perfect) and all of the following shots.

It almost seems as if the camera is bracketing the shots and the exposure time is increasing as it records.

Any suggestions?

gjimenez
Posts: 6
Joined: Sat Jul 25, 2015 1:26 pm

Re: Official V4L2 driver

Sat Jul 25, 2015 1:28 pm

Hi, Is roi supported now on the official v4l2 driver?

mattday
Posts: 4
Joined: Sun Jul 26, 2015 5:00 pm

Re: Official V4L2 driver

Sun Jul 26, 2015 5:35 pm

Hope someone will be able to clarify something regarding the official driver. I'm aiming to perform some processing on a 1920x1080 stream as fast as possible (it really needs to be at least 10fps).

It seems the official driver can deliver frames at close to 30fps only when the pixel format is set to H264 or MJPG. Anything else such as YU12 or RGB3 and this drops to 5fps. Can anyone give an authoritative explanation as to why the uncompressed frame rates are so low? I can guess bandwidth limitations, but would love to know what is really going on and how I might get round it (trying to decode compressed frames on the CPU doesn't make much sense).

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 4356
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Official V4L2 driver

Sun Jul 26, 2015 8:14 pm

gjimenez wrote:Hi, Is roi supported now on the official v4l2 driver?
Not yet. I got the initial part working following on from byterg's initial patch, but need to ensure that it is the correct way within the V4L2 framework and gets advertised correctly.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 4356
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Official V4L2 driver

Sun Jul 26, 2015 8:58 pm

mattday wrote:Hope someone will be able to clarify something regarding the official driver. I'm aiming to perform some processing on a 1920x1080 stream as fast as possible (it really needs to be at least 10fps).

It seems the official driver can deliver frames at close to 30fps only when the pixel format is set to H264 or MJPG. Anything else such as YU12 or RGB3 and this drops to 5fps. Can anyone give an authoritative explanation as to why the uncompressed frame rates are so low? I can guess bandwidth limitations, but would love to know what is really going on and how I might get round it (trying to decode compressed frames on the CPU doesn't make much sense).
The imaging pipe always runs in YUV mode, and converting to any RGB format is also done as an extra step. There is almost zero performance difference between the YUV 4:2:0 formats (I420 aka YU12, YV12, NV12, and NV21). YUV 4:2:2 will be a little slower as there is more data to handle, and the interleaving of the chroma for the YUYV formats will have a very small additional overhead compared to the planar format.

Secondly, for performance reasons, the GPU always works with widths that are multiples of 32 and heights a multiple of 16. 1080 isn't a multiple of 16, so there is a postprocessing step to remove the extra padding and that takes memory bandwidth and time.

BGR3 also takes an extra step over and above RGB3 (My brain couldn't munging the coefficients at the time, so we convert I420 to RGB3, and then swap all the red and blue values. At some point I might revisit that, but very low priority).

I'll check later (camera module currently refusing to detect), but we have had on other threads reporting 2592x1944 I420 streaming at 15fps successfully.
viewtopic.php?f=43&t=107571&p=740712#p740800 for example, with the reported performance of
BGR3 - 9 fps
RGB3 - 11-12 fps
l420 - 15 fps
viewtopic.php?f=43&t=82691&p=585569#p585569 we also have 15fps 5MPix I420.

The key thing I suspect you are missing is setting the max_video_[width|height] when you load the driver ("sudo modprobe bcm2835-v4l2 max_video_width=2592 max_video_height=1944"). Without that it will be doing multiple stills captures, which include extra denoising and postprocessing to optimise image quality, and the sensor will be stopping and starting as the buffering isn't sufficient to keep the higher resolutions going continuously.

edit Got my camera working again.

Code: Select all

sudo modprobe bcm2835-v4l2 max_video_width=2592 max_video_height=1944
v4l2-ctl --set-fmt-video=width=1920,height=1080,pixelformat=[I420|YV12|RGB3|BGR3]
v4l2-ctl -p 15
v4l2-ctl --stream-mmap=3 --stream-count=1000 --stream-to=/dev/null
Works perfectly at 15fps for all four formats listed above. The framerate does drop if going up to 2592x1944, but you said you didn't need that.

I haven't checked what the field of view is. It may well be selecting the cropped field of view mode. Requesting 1920x1440 (4:3) should ensure you get the full field of view (full 5MPix resized), and a quick test implies that it still manages to deliver 15fps in all the above pixel formats.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

mattday
Posts: 4
Joined: Sun Jul 26, 2015 5:00 pm

Re: Official V4L2 driver

Sun Jul 26, 2015 10:19 pm

6by9 wrote: I'll check later (camera module currently refusing to detect), but we have had on other threads reporting 2592x1944 I420 streaming at 15fps successfully.
viewtopic.php?f=43&t=107571&p=740712#p740800 for example, with the reported performance of
BGR3 - 9 fps
RGB3 - 11-12 fps
l420 - 15 fps
viewtopic.php?f=43&t=82691&p=585569#p585569 we also have 15fps 5MPix I420.

The key thing I suspect you are missing is setting the max_video_[width|height] when you load the driver ("sudo modprobe bcm2835-v4l2 max_video_width=2592 max_video_height=1944"). Without that it will be doing multiple stills captures, which include extra denoising and postprocessing to optimise image quality, and the sensor will be stopping and starting as the buffering isn't sufficient to keep the higher resolutions going continuously.
Spot on! Thanks for a really helpful post. I had completely missed the need for these parameters (might be helpful to add to first page). Using them, the performance is quite impressive. On my Pi 2 I get similar speeds to those quoted for RGB3 and BGR3, but about 23 fps for I420. In fact, the frame rate doesn't drop off much until you start approaching the full sensor resolution. So for example 2240X1944 is still about 28 fps (I420). CPU usage is also negligible. Awesome stuff!

Unfortunately, using the OpenCV 3 VideoCapture class, the frame rate at 1920x1080 has only increased from 5 to 17 fps and it is working one CPU core hard. I doubt I'll be able to process the stream at 17 fps, but would like as much of the CPU available as possible. So does this mean I have to write something using the V4L2 API to store the frames to OpenCV Mat objects, or has somebody perhaps done that already?

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 4356
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Official V4L2 driver

Sun Jul 26, 2015 10:44 pm

mattday wrote:Spot on! Thanks for a really helpful post. I had completely missed the need for these parameters (might be helpful to add to first page). Using them, the performance is quite impressive. On my Pi 2 I get similar speeds to those quoted for RGB3 and BGR3, but about 23 fps for I420. In fact, the frame rate doesn't drop off much until you start approaching the full sensor resolution. So for example 2240X1944 is still about 28 fps (I420). CPU usage is also negligible. Awesome stuff!
Please bear in mind the sensor resolutions available. 2592x1944 @ 15fps, a cropped 1920x1080 @ 30fps, or binned 1296x976 @ 42fps. If you ask for something too odd, you may end up with either taking the 1920x1080, cropping it further to a 4:3 aspect ratio, and then upscaling. Or you could end up upscaling the binned mode. Neither will be great for the image quality. Your 2240x1944 @ 28fps is almost guaranteed to be doing something grotty as it can't be in the 2592x1944 mode.
mattday wrote:Unfortunately, using the OpenCV 3 VideoCapture class, the frame rate at 1920x1080 has only increased from 5 to 17 fps and it is working one CPU core hard. I doubt I'll be able to process the stream at 17 fps, but would like as much of the CPU available as possible. So does this mean I have to write something using the V4L2 API to store the frames to OpenCV Mat objects, or has somebody perhaps done that already?
Last time I looked, VideoCapture class is already interfacing directly to V4L2. The implementation wasn't great as it always wanted BGR3 (one of the reasons support got added to the GPU firmware) even if you want a greyscale image (easiest to take YUV and just take the luma), and also didn't support sensible framerate control. I don't know if that has been improved in OpenCV3.
If it got overloaded with frames, then it swallowed vast amounts of CPU, and ended up with huge latency too.
It'd be worth having a quick look at the implementation to see if it is still doing daft things.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

mattday
Posts: 4
Joined: Sun Jul 26, 2015 5:00 pm

Re: Official V4L2 driver

Mon Jul 27, 2015 8:11 pm

6by9 wrote: Please bear in mind the sensor resolutions available. 2592x1944 @ 15fps, a cropped 1920x1080 @ 30fps, or binned 1296x976 @ 42fps. If you ask for something too odd, you may end up with either taking the 1920x1080, cropping it further to a 4:3 aspect ratio, and then upscaling. Or you could end up upscaling the binned mode. Neither will be great for the image quality. Your 2240x1944 @ 28fps is almost guaranteed to be doing something grotty as it can't be in the 2592x1944 mode.
I haven't actually looked at any of the frames, so I'll take your word for it that they're not what I expect. Although I am still curious as to why I get 23 fps at 2592x1944...

Code: Select all

$ v4l2-ctl -v width=2592,height=1944,pixelformat=YU12
$ v4l2-ctl -p 30
Frame rate set to 30.000 fps
$ v4l2-ctl --stream-mmap=3 --stream-count=360 --stream-to=/dev/null
<<<<<<<<<<<<<<<<<<<<<<<<<< 24 fps
<<<<<<<<<<<<<<<<<<<<<<<<< 24 fps
<<<<<<<<<<<<<<<<<<<<<<<<< 24 fps
<<<<<<<<<<<<<<<<<<<<<<<< 23 fps
<<<<<<<<<<<<<<<<<<<<<<<< 23 fps
<<<<<<<<<<<<<<<<<<<<<<<< 23 fps
<<<<<<<<<<<<<<<<<<<<<<< 23 fps
<<<<<<<<<<<<<<<<<<<<<<<< 23 fps
<<<<<<<<<<<<<<<<<<<<<<< 23 fps
<<<<<<<<<<<<<<<<<<<<<<<< 23 fps
<<<<<<<<<<<<<<<<<<<<<<< 22 fps
<<<<<<<<<<<<<<<<<<<<<<< 23 fps
<<<<<<<<<<<<<<<<<<<<<<<< 23 fps
<<<<<<<<<<<<<<<<<<<<<<<< 23 fps
<<<<<<<<<<<<<<<<<<<<<<<< 23 fps
6by9 wrote: Last time I looked, VideoCapture class is already interfacing directly to V4L2. The implementation wasn't great as it always wanted BGR3 (one of the reasons support got added to the GPU firmware) even if you want a greyscale image (easiest to take YUV and just take the luma), and also didn't support sensible framerate control. I don't know if that has been improved in OpenCV3.
If it got overloaded with frames, then it swallowed vast amounts of CPU, and ended up with huge latency too.
It'd be worth having a quick look at the implementation to see if it is still doing daft things.
I appreciate OpenCV VideoCapture is just wrapping V4L2, but what is not clear for me yet, is whether it can be made to do that in a nice efficient way. As you noted, it seems to want to receive frames in BGR format by default, but I found you can request other formats by calling, for example:

Code: Select all

VideoCapture::set(CV_CAP_PROP_MODE, CV_CAP_MODE_YUYV);
Frame rates at 1920x1080 are then:

CV_CAP_MODE_YUYV, ~28 fps, with ~100% core usage.
CV_CAP_MODE_GRAY, ~28 fps, with ~50% core usage
CV_CAP_MODE_RGB, ~21 fps, with ~100% core usage.

I read something that suggested the OpenCV frame rate control may have been fixed now. However, in my experiments, it seems to want to always run at 30 fps unless you are using BGR format.

I have had a quick look at the OpenCV file cap_libv4l.cpp, but it is going to take some time to completely follow what it is doing. It might be quicker to take a different approach to figuring out where it is consuming all the CPU cycles.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 4356
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Official V4L2 driver

Mon Jul 27, 2015 9:32 pm

mattday wrote:I haven't actually looked at any of the frames, so I'll take your word for it that they're not what I expect. Although I am still curious as to why I get 23 fps at 2592x1944...

Code: Select all

$ v4l2-ctl -v width=2592,height=1944,pixelformat=YU12
$ v4l2-ctl -p 30
Frame rate set to 30.000 fps
$ v4l2-ctl --stream-mmap=3 --stream-count=360 --stream-to=/dev/null
<<<<<<<<<<<<<<<<<<<<<<<<<< 24 fps
<<<<<<<<<<<<<<<<<<<<<<<<< 24 fps
Because you've asked for 30fps. It will try to deliver that in preference over image quality, so the sensor will be running at either 1920x1080 or 1296x976 and upscaling.
mattday wrote:I appreciate OpenCV VideoCapture is just wrapping V4L2, but what is not clear for me yet, is whether it can be made to do that in a nice efficient way. As you noted, it seems to want to receive frames in BGR format by default, but I found you can request other formats by calling, for example:

Code: Select all

VideoCapture::set(CV_CAP_PROP_MODE, CV_CAP_MODE_YUYV);
Frame rates at 1920x1080 are then:

CV_CAP_MODE_YUYV, ~28 fps, with ~100% core usage.
CV_CAP_MODE_GRAY, ~28 fps, with ~50% core usage
CV_CAP_MODE_RGB, ~21 fps, with ~100% core usage.
OK, that is new then. A quick hunt back finds my previous investigations in conversation with Joose viewtopic.php?f=43&t=65026&p=529772s#p528389
mattday wrote:I read something that suggested the OpenCV frame rate control may have been fixed now. However, in my experiments, it seems to want to always run at 30 fps unless you are using BGR format.

I have had a quick look at the OpenCV file cap_libv4l.cpp, but it is going to take some time to completely follow what it is doing. It might be quicker to take a different approach to figuring out where it is consuming all the CPU cycles.
First thing to ensure is that you are using cap_libv4l.cpp, and not cap_v4l.cpp - the libv4l version has had updates, including
16th patch: Dec 16, 2014, Joseph Howse josephhowse@nummist.com
- Allow getting/setting CV_CAP_PROP_MODE. These values are supported:
- CV_CAP_MODE_BGR : BGR24 (default)
- CV_CAP_MODE_RGB : RGB24
- CV_CAP_MODE_GRAY : Y8, extracted from YUV420
- Tested successfully on these cameras:
- PlayStation 3 Eye
- Logitech C920
- Odroid USB-CAM 720P
17th patch: May 9, 2015, Matt Sandler
added supported for CV_CAP_PROP_POS_MSEC, CV_CAP_PROP_POS_FRAMES, CV_CAP_PROP_FPS
Those sound like the two features you're after.
Just a minor reservation as there is a comment that libv4l will do image conversions between the formats. It shouldn't need to as the Pi camera supports all those formats, but worth checking. "v4l-ctl -V" on the command line would tell you the current format selected.

The default is for 30fps. You can enable additional debug from the V4L2 driver by adding "debug=2" to the end of the modprobe line loading the driver - that should give you extra info to confirm whether the requests you're expecting are actually being made.
My observations from last time were that it seemingly had an internal FIFO of buffers, so if the processing wasn't running fast enough it just kept on adding frames into there and the latency went up and up with the CPU maxed out. The memcpy of the entire frame at https://github.com/Itseez/opencv/blob/m ... .cpp#L1338 won't be helping.

You don't say whether you're using a Pi1 or Pi2. All my previous tests were on a Pi1 (the 2 wasn't out), so I wonder if the updates in Pi2 may explain the extra framerate you're seeing.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

gjimenez
Posts: 6
Joined: Sat Jul 25, 2015 1:26 pm

Re: Official V4L2 driver

Mon Jul 27, 2015 10:50 pm

Hi, I am trying to livestreaming to youtube audio and video using ffmpeg.

I can do it but I have and audio-video sync problem.

At first I tried with raspivid and pipe video to ffmpeg capturing audio with alsa but I got an increasing desync between audio and video. I read that this is due to the crystal used in the rapberrry camera module and raspivid not sending timestamps to ffmpeg to make the audio-video sync.

Now I am trying with v4l2 and ffmpeg but using resolution 1280x720, fps: 25 and bitrate: 1500000, I just get out 8 fps.

Please help me,

gjimenez
Posts: 6
Joined: Sat Jul 25, 2015 1:26 pm

Re: Official V4L2 driver

Fri Jul 31, 2015 9:20 pm

I finally solved the audio-video sync with this raspivid modification: https://github.com/d3faultdotxbe/raspiv ... s/raspicam

If you want constant 25 fps you have to set it to 26, it works perfect.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 4356
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Official V4L2 driver

Fri Jul 31, 2015 10:53 pm

gjimenez wrote:I finally solved the audio-video sync with this raspivid modification: https://github.com/d3faultdotxbe/raspiv ... s/raspicam

If you want constant 25 fps you have to set it to 26, it works perfect.
Er, the firmware was fixed about 6 months back to give framerates that were accurate to within a pretty small tolerance - viewtopic.php?f=43&t=99351&p=701280 At least the person who has written your hacky code knows vaguely what they were doing and requests an I-frame at a drop point, though I wouldn't be surprised if the timing on that was pretty variable depending on how quickly the Linux app is processing the output - there's up to 2MB of FIFO available to the codec on the output side which may need to be drained before you see your requested I.

As has been said on numerous occasions, the correct solution is to use the timestamps that come back with each and every frame and store those in your container/stream.

PS Cross posting is annoying. Very few things are worth saying twice.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

mattday
Posts: 4
Joined: Sun Jul 26, 2015 5:00 pm

Re: Official V4L2 driver

Sat Aug 01, 2015 5:56 pm

6by9 wrote: The default is for 30fps. You can enable additional debug from the V4L2 driver by adding "debug=2" to the end of the modprobe line loading the driver - that should give you extra info to confirm whether the requests you're expecting are actually being made.
My observations from last time were that it seemingly had an internal FIFO of buffers, so if the processing wasn't running fast enough it just kept on adding frames into there and the latency went up and up with the CPU maxed out. The memcpy of the entire frame at https://github.com/Itseez/opencv/blob/m ... .cpp#L1338 won't be helping.

You don't say whether you're using a Pi1 or Pi2. All my previous tests were on a Pi1 (the 2 wasn't out), so I wonder if the updates in Pi2 may explain the extra framerate you're seeing.
I am using a Pi2, so that probably would explain the extra framerate.

I have given up on OpenCV VideoCapture as there was no way it was going to do what I really wanted. I instead took the example capture code from the V4L2 documentation and started to write something based on that. This gives me much more control over everything and avoids doing a memcpy of each frame. In fact, I found that just doing the memcpy at 1920x1080 uses about 50% of a CPU core and slows capture by a couple of fps. Anyhow, although this is better, it is still not really going to give me what I want, because I will then have to resize the image on the CPU which is likely to be too expensive.

What I would like, without working the CPU, is a smaller greyscale image in addition to the full size colour image. I guess I'll have to start trying to understand MMAL, unless there is a way to get multiple streams from the V4L2 driver. This scenario is fairly common for computer vision applications and I have read posts by wibble82 and ayronc from 2013 who were tackling related problems although I couldn't get wibble82's camera API to do this.

viewtopic.php?f=43&t=60239
viewtopic.php?f=43&t=59093

Return to “Camera board”

Who is online

Users browsing this forum: No registered users and 11 guests