prevoste
Posts: 32
Joined: Mon Jul 16, 2012 9:36 am

Captured frames get progressively darker

Mon May 19, 2014 12:31 pm

Hi all,

I am having a problem with Pi Camera and new V4L2 driver capturing a sequence of still images. I am using a newly built Raspbian system (3.10.25+) with the V4L2 Kernel driver (Version 0.0.2). I have written a simple c++ based program using the Opencv function ‘cvQueryFrame’ to capture each frame, which in this test program I am writing each frame to a file. Unlike the when I used the userspace version of the driver (UV4L2) last year I get some strange results. Under certain conditions the frame exposure tends to go almost completely black. Perhaps I am using the driver wrongly.

These are some of my findings (taken on a bright sunny day):

Setting the Frame Height & Width to 480 x 640 seems to be OK.

First frame is normally very over exposed but the subsequent seem reasonably exposed.

Setting the Frame Height & Width to 960 x 1280

First frame is very over exposed
Frames 2-5 completely black
Frame 6 is very over exposed
Frame 7 is a little over exposed
Subsequent frames seem reasonably exposed up until about frames 40 when each frame gets progressively darker until almost completely black and stays that way.

The same degrading exposure effect is observed at other higher fame height/width settings and the test is repeatable.

I understand that this driver is only a beta version but has anyone else observed the same problem and can anyone suggest what the problem is?

Thanks Ernie


6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6879
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Captured frames get progressively darker

Mon May 19, 2014 1:51 pm

The GPU has two processing paths - either a video capture (high framerate, minimum latency) or stills capture (lower framerate, higher latency, but better image quality)

V4L2 doesn't have a mechanism for indicating which is desired. With JPEG, H264, and MJPEG there is an obvious answer. With raw YUV or RGB frames there isn't, so it makes a best guess based on frame size. Anything with < 1280*720 pixels is judged to be video. Anything above is a still capture. (That is configured in the driver at /drivers/media/platform/bcm2835/bcm2835-camera.c, defines MAX_VIDEO_MODE_WIDTH and MAX_VIDEO_MODE_HEIGHT).

There is a known issue that doing multiple stills captures with no preview running results in the AGC results going a little crazy, normally resulting in black images. We are looking in to it, but it may be a tricky one to solve and available time is limited.
One workaround is to enable the preview overlay from the V4L2 driver using

Code: Select all

v4l2-ctl --overlay=1
. It should work even if there is no display attached, and because it is producing preview frames then AGC etc run happily.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 22695
Joined: Sat Jul 30, 2011 7:41 pm

Re: Captured frames get progressively darker

Mon May 19, 2014 2:33 pm

In Raspistill I fixed this by always having a preview running but dumping output to a NULLSINK unless it needed to be displayed.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6879
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Captured frames get progressively darker

Mon May 19, 2014 2:43 pm

jamesh wrote:In Raspistill I fixed this by always having a preview running but dumping output to a NULLSINK unless it needed to be displayed.
I know. V4L2 can't really get away with doing this though :(
Running the overlay is the only way in V4L2 to address this currently. There are a couple of bits in the MMAL/IL levels that deliberately lock AGC/AWB during captures, but I don't know how the tuner would respond anyway if those were unlocked.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

prevoste
Posts: 32
Joined: Mon Jul 16, 2012 9:36 am

Re: Captured frames get progressively darker

Mon May 19, 2014 4:47 pm

Hi,

Thanks both for your replies.

I have tried your suggestion of setting up the preview option using ‘v4l2-ctl –overlay=1’ and this has fixed the problem of the frames getting darker. I have done some more tests and found that when set the frame size to 640*480 and I run the frame capture in a loop, I get the following:

With v4l2-ctl –overlay=0 about 30 fps
v4l2-ctl –overlay=1 about 30 fps

With frame size to 1280*960

With v4l2-ctl –overlay=0 about 6 fps
v4l2-ctl –overlay=1 about 1.5 fps

I assume the difference is due your comments on it using ‘Video Mode’ or ‘Still Mode’.

I noticed in an earlier post a comment that the V4L2 driver could use hardware accelerated JPEG. How do you make sure you are using the hardware accelerated JPEG?

Thanks Ernie

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6879
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Captured frames get progressively darker

Mon May 19, 2014 5:15 pm

prevoste wrote:With frame size to 1280*960

With v4l2-ctl –overlay=0 about 6 fps
v4l2-ctl –overlay=1 about 1.5 fps
The sensor is switching between stills mode to video mode and back again for each capture. The sensor always produces a junk frame when it starts streaming, so those have to be discarded at 30-60ms each. That eats into the framerate.
If you dropped to 1024x768 you would be just under the magic threshold and still get the full 30fps of video mode. Or you could recompile the V4L2 driver with the threshold set to any value you fancied.
prevoste wrote:I noticed in an earlier post a comment that the V4L2 driver could use hardware accelerated JPEG. How do you make sure you are using the hardware accelerated JPEG?
If you ask the V4L2 driver for JPEG (or MJPEG or H264) then it is using the hardware acceleration blocks. If you ask for an RGB or YUV format then it just delivers the raw pixels and the won't use the hardware compression blocks. Use v4l2-ctl -V to show what the currently selected mode is.
When I looked into OpenCV, it was very limited on the formats it would accept, and would always convert them to RGB888 before passing it into the app. That may be the other stumbling block to your testing.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

prevoste
Posts: 32
Joined: Mon Jul 16, 2012 9:36 am

Re: Captured frames get progressively darker

Tue May 20, 2014 1:44 pm

Hi,

I have done some more tests which seem to confirm that below 768*1024 the frame rate is higher I assume from what you say because it is running in ‘Video mode’. You imply that the image quality is better in ‘Still mode’ than in ‘Video mode’, I have create a couple of images one at 704*960 (Video mode) and another at 704*1024 (still) and to my eyes the quality is much the same.

When I use the v4l2-ctl –V command I get a Pixel Format 'BGR3', I assume this is the value you were referring to that controls whether the hardware accelerator is used.

You also suggest it could be possible to recompile the video driver ‘bcm2835-camera.c’ to modify the MAX_VIDEO_MODE_WIDTH and MAX_VIDEO_MODE_HEIGHT to enable ‘Video mode’ at to be used at higher resolutions, where can I find the driver source to try this?


Thanks again your time and help


Ernie

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6879
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Captured frames get progressively darker

Tue May 20, 2014 2:13 pm

prevoste wrote:When I use the v4l2-ctl –V command I get a Pixel Format 'BGR3', I assume this is the value you were referring to that controls whether the hardware accelerator is used.
3 bytes of RGB data per pixel. http://hverkuil.home.xs4all.nl/spec/med ... packed-rgb for a description of all the formats.
No hardware compression acceleration required as it isn't a compressed format!
prevoste wrote:You also suggest it could be possible to recompile the video driver ‘bcm2835-camera.c’ to modify the MAX_VIDEO_MODE_WIDTH and MAX_VIDEO_MODE_HEIGHT to enable ‘Video mode’ at to be used at higher resolutions, where can I find the driver source to try this?
It means rebuilding part of the kernel.
Source is all at https://github.com/raspberrypi/linux The V4L2 driver is in drivers/media/platform/bcm2835/
Instructions at http://elinux.org/RPi_Kernel_Compilation or http://www.raspberrypi.org/documentatio ... uilding.md
I'd recommend cross compiling if at all possible. Admittedly I've got access to a fairly beefy build machine (24 core Xeon machine with silly amounts of RAM!), but there a kernel build takes about 10mins. On a Pi building natively you'll be looking at a good couple of hours!

Seeing as so many people are getting upset at this limit, I'll look at making it a load-time parameter for the module.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6879
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Captured frames get progressively darker

Tue May 20, 2014 5:56 pm

6by9 wrote:Seeing as so many people are getting upset at this limit, I'll look at making it a load-time parameter for the module.
OK, that was easier than I thought. Change pushed to my kernel branch, and I'll make the pull request tomorrow https://github.com/6by9/linux/commit/44 ... 6d5c77885f

To make life easier for all, I've put my prebuilt kernel module on https://github.com/6by9/RPiTest/blob/ma ... 35-v4l2.ko
The latest firmware changes haven't been pushed for removing padding, so there may be a couple of modes that aren't perfect if you deviate from the multiple of 32/16 rule.

Code: Select all

sudo modprobe bcm2835-v4l2 max_video_width=2592 max_video_height=1944
should load the driver and leave everything in video mode all the way up to 5MPix.
Streaming with v4l2-ctl at that resolution I420(YU12) to dev/null works at 15fps. Try saving it to SD card and that number will plummet (112MB/s is just a little on the high side!)
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

prevoste
Posts: 32
Joined: Mon Jul 16, 2012 9:36 am

Re: Captured frames get progressively darker

Wed May 21, 2014 3:01 pm

HI,

I would like to say how much I appreciate the help you are giving me. Since your last post I have downloaded your updated driver and tried installing it but I get the following error:

ERROR: could not insert 'bcm2835_v4l2': Exec format error

I assume this is because yours is built against the ‘3.12.20+’ kernel where I am currently running the ‘3.10.25+’ version.

I should point out that I am not an expert in Linux as most of my working life was spent working with VMS & Windows.

Yesterday before you provided the new driver I started to try to update the driver myself using your links. I first planned to just to rebuild the driver without any changes to the source, to prove I could. I downloaded the correct version of the Kernel source and setup the environment. I found the camera module ‘Makefile’ containing:


bcm2835-v4l2-objs := bcm2835-camera.o controls.o mmal-vchiq.o
obj-$(CONFIG_VIDEO_BCM2835_MMAL) += bcm2835-v4l2.o

ccflags-$(CONFIG_VIDEO_BCM2835) += -Idrivers/misc/vc04_services -Idrivers/misc/vc04_services/interface/vcos/linuxkernel -D__VCCOREVER__=0x04000000


At this point my lack of knowledge about Linux blocked my progress, I tried just typing the 'make' command but got ‘make: *** No targets. Stop.’ back.

Could you please point me in the right direction to be able to build the camera module.


Thanks Ernie

ethanol100
Posts: 581
Joined: Wed Oct 02, 2013 12:28 pm

Re: Captured frames get progressively darker

Wed May 21, 2014 3:27 pm

I was successful yesterday following this site: https://github.com/notro/rpi-source/wiki
On my pi I needed to do the following to get it working(all from the link above):

Code: Select all

sudo apt-get install sudo  gcc-4.7 g++-4.7 libncurses5-dev
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.6 60 --slave /usr/bin/g++ g++ /usr/bin/g++-4.6
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.7 40 --slave /usr/bin/g++ g++ /usr/bin/g++-4.7
sudo update-alternatives --config gcc
sudo wget https://raw.githubusercontent.com/notro/rpi-source/master/rpi-source -O "/usr/bin/rpi-source"
sudo chmod +x /usr/bin/rpi-source
/usr/bin/rpi-source -q --tag-update
rpi-source
cd linux
Then I could modify drivers/media/platform/bcm2835/bcm2835-camera.c
and build and test the module(I was still in the "linux" directory):

Code: Select all

make SUBDIRS=drivers/media/platform/bcm2835 modules
sudo make SUBDIRS=drivers/media/platform/bcm2835 modules_install
sudo depmod
sudo modprobe bcm2835-v4l2
Using lsmod I could see the new loaded module.

How have you prepared your source tree?
Last edited by ethanol100 on Wed May 21, 2014 3:31 pm, edited 3 times in total.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6879
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Captured frames get progressively darker

Wed May 21, 2014 3:28 pm

prevoste wrote:ERROR: could not insert 'bcm2835_v4l2': Exec format error

I assume this is because yours is built against the ‘3.12.20+’ kernel where I am currently running the ‘3.10.25+’ version.
Yes, mine is based on 3.12.20+ top of tree. Linux version checks all modules on load. sudo rpi-update would get you the latest kernel.
prevoste wrote:At this point my lack of knowledge about Linux blocked my progress, I tried just typing the 'make' command but got ‘make: *** No targets. Stop.’ back.

Could you please point me in the right direction to be able to build the camera module.
Whilst I'm sure there are ways to build just the camera module, I don't use them as I rebuild the entire kernel.
http://sandsoftwaresound.net/raspberry- ... el-module/ looks plausible. Or http://ivanx.com/raspberrypi/raspberrypi_kernel.html for cross compiling. (I just did a google for "raspberry pi build kernel modules", and those were in the first few hits).
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

prevoste
Posts: 32
Joined: Mon Jul 16, 2012 9:36 am

Re: Captured frames get progressively darker

Thu May 22, 2014 4:57 pm

Hi ethanol100 & 6by9,

Thanks to both of you for your help. I now have a system running with the modified camera driver and I am able to capture large enough frame at an acceptable rates.

One point that may be worth me mentioning is that after rebuilding the camera driver I did some more tests and got improved frames rates when using larger resolutions as expected, but when I used ‘v4l2-ctl –overlay=1’ (not required by me now), I got the following error:

VIDIOC_QBUF: Invalid argument
Segmentation fault

But with ‘v4l2-ctl –overlay=0’ it was fine.

Thanks again for your help

Ernie

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6879
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Captured frames get progressively darker

Thu May 22, 2014 5:07 pm

prevoste wrote: One point that may be worth me mentioning is that after rebuilding the camera driver I did some more tests and got improved frames rates when using larger resolutions as expected, but when I used ‘v4l2-ctl –overlay=1’ (not required by me now), I got the following error:

VIDIOC_QBUF: Invalid argument
Segmentation fault

But with ‘v4l2-ctl –overlay=0’ it was fine.
How large a resolution were you going for? 1920x1088 (height a multiple of 16 until Dom does a firmware release!) should be fine. The overlay gets fed the same image as you ask for, and I do know it starts going a little crazy if presented with 2592x1944 as it tries to rescale it on the fly for the screen.
It shouldn't trigger an invalid argument or seg fault though, unless we're hitting an unhandled out of memory condition (scary numbers of 5MPix buffers kicking around potentially!) and dereferencing an invalid pointer. Hmm.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

prevoste
Posts: 32
Joined: Mon Jul 16, 2012 9:36 am

Re: Captured frames get progressively darker

Thu May 22, 2014 8:36 pm

Hi,

I have done some more tests. The error seems to occur when any valid resolution (multiple of 16) has total size greater than 1024*1024. So a value like 384*2592 is fine but 416*2592 gives the error.

I hope this helps.

Ernie

Return to “Camera board”