Affe
Posts: 8
Joined: Thu Nov 26, 2015 2:39 pm

Problem with mjpeg using camera and v4l2

Thu Nov 26, 2015 2:57 pm

I'm using the v4l2 camera for at project where i'm streaming video from a Pi to a computer. The other day i found something weird going on. It seems that when i use mjpeg pixelformat, the file size of the initial frames (which looks totally fine) is much smaller than the rest of the frames.

I have reproduced the problem with v4l2-ctl with the following commands:

v4l2-ctl --set-fmt-video=width=1280,height=720,pixelformat=5
v4l2-ctl --stream-mmap=3 --stream-count=1 --stream-to=somefile.jpg

This gives a jpeg image with a size of 20-30 kb. Then if i run the second command again with --stream-count=100, then it will give a file of about 8-10 mb. I have experimentet with this both with v4l2-ctl and using v4l2 from c++, and it seems like the first few frames is always much smaller than later frames.

Is this a bug in the driver?
1 = 20kb 100 = 8,17mb

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7411
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Problem with mjpeg using camera and v4l2

Thu Nov 26, 2015 3:34 pm

The MJPEG isn't as clean as the H264 codec.
Bitrate control (and you don't appear to have specified a bitrate - default is 10MBit/s) is basically starting with a JPEG Q factor of something, and then adjusting it if we're above or below the target. 10MBit/s = 0.33MBit/frame, or 42kB, so we're starting below target, and increasing. Ending up with 8MB for 100 frames would be 80kB/frame, so it does appear to have overshot.

There have been other reports that the bitrate creeps upwards on MJPEG - it's on the list of things to look at, but so are many other things.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Affe
Posts: 8
Joined: Thu Nov 26, 2015 2:39 pm

Re: Problem with mjpeg using camera and v4l2

Thu Nov 26, 2015 4:51 pm

Thank you for the fast reply.

I have now tried to set the bitrate from my code and tried both constant and variable bitrate, but it does not seem to change anything, the bitrate still climbs to a certain level and stays there.

Just as a possible "hack" i tried to set the bitrate everytime i capture a frame to see if i could stop the climb, but it did not seem to matter.

Do you know if there is any way to bypass this climbing? Or at least to reset the climbing each frame to keep it from climbing. It seems that the JPEG Q factor can't be set through code for MJPEG pixelformat, is this correct?

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7411
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Problem with mjpeg using camera and v4l2

Thu Nov 26, 2015 5:11 pm

Affe wrote:Do you know if there is any way to bypass this climbing?
No, this is the supposed bitrate control within the codec.
Affe wrote: Or at least to reset the climbing each frame to keep it from climbing. It seems that the JPEG Q factor can't be set through code for MJPEG pixelformat, is this correct?
Totally correct that you can't set the JPEG Q factor directly - you've asked for MJPEG, so it is up to the codec to set the Q factor for the individual frames.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Affe
Posts: 8
Joined: Thu Nov 26, 2015 2:39 pm

Re: Problem with mjpeg using camera and v4l2

Thu Nov 26, 2015 5:20 pm

Thank you again for your replies.

I guess i will have to use the current state of the driver then, and hope for future updates :)

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7411
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Problem with mjpeg using camera and v4l2

Thu Nov 26, 2015 9:44 pm

Thought I'd have a look as it should have been fairly obvious what was going on.

That codec starts with a quality of 7, and steps the quality up/down by one if the frame produced is off from the target by more than 12%. The quality range is clipped to 1 to 20.
The issue appears to be that the code for computing the target is gibberish, and the quality seems to generally ramp up to 20 and stick there. Plus changing the bitrate from V4L2 appears not to update the actual bitrate unless the format is set after the bitrate - it made an attempt, but the value would seemingly only stick if actually encoding at the time.

So a few little glitches, but shouldn't be too hard to fix up.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Affe
Posts: 8
Joined: Thu Nov 26, 2015 2:39 pm

Re: Problem with mjpeg using camera and v4l2

Thu Nov 26, 2015 10:28 pm

Well, that information explains a lot. A JPEG Q of 20 would match the size that we are getting.
So a few little glitches, but shouldn't be too hard to fix up.
Does this actually mean that you are working on a fix?

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7411
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Problem with mjpeg using camera and v4l2

Thu Nov 26, 2015 11:09 pm

Affe wrote:Does this actually mean that you are working on a fix?
Yes, as I wanted something simple to occupy my evening. Think I've got it solved to a first approximation.
Rate control is simplistic in that it just takes bitrate/framerate for the size it wants for each frame, and adjusts quality if the last frame was outside target +/- 12%. It is taking no notice of whether it has spare bits left from recent frames.
I've also sorted it so you can change bitrate whilst streaming.

I suspect I'll throw the change at Pi Towers as it stands - it is significantly better than before. Refining the rate control algorithm can get added to the bottom of my to-do list.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Affe
Posts: 8
Joined: Thu Nov 26, 2015 2:39 pm

Re: Problem with mjpeg using camera and v4l2

Fri Nov 27, 2015 8:33 am

That really sounds awesome! Thank you so much for looking into my problem, if this works, you just upgraded my bachelor project significantly. I think it is very cool that you call it "something simple" :)
I suspect I'll throw the change at Pi Towers as it stands
Does this mean I can get the change from some git repository and build it?

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7411
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Problem with mjpeg using camera and v4l2

Fri Nov 27, 2015 8:51 am

Affe wrote:
I suspect I'll throw the change at Pi Towers as it stands
Does this mean I can get the change from some git repository and build it?
Not build it, but sudo rpi-update to grab the new firmware blob once Dom has done his magic (I'll post once I get the nod, or you can look at github.com/Hexxeh/rpi-firmware which where the firmware blobs are pulled from).
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Affe
Posts: 8
Joined: Thu Nov 26, 2015 2:39 pm

Re: Problem with mjpeg using camera and v4l2

Fri Nov 27, 2015 9:42 am

That sounds fantastic :)

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7411
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Problem with mjpeg using camera and v4l2

Mon Nov 30, 2015 8:53 pm

Dom's just done a release and had told me that this change was going to be in there, though he hasn't referenced it in the commit text. Worth a quick "sudo rpi-update" and retest.

If you do "vcgencmd set_logging level=4" before the run, and then "sudo vcdbg log msg" afterwards, you should get a logging message for each frame of the form:
"mjpeg_enc_stripe: eof <buffer size> (done <frame count>/<current frame size>) (target:<target frame size>) qual:<quality>"
You should see <current frame size> broadly matching <target frame size>, and <quality> being tweaked when it diverges too far (try a scene change).
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Affe
Posts: 8
Joined: Thu Nov 26, 2015 2:39 pm

Re: Problem with mjpeg using camera and v4l2

Tue Dec 01, 2015 7:44 am

I actually saw the update on the git repository and updated even though (as you said) the change wasn't in the commit message. I just tried streamin 720p, 20 fps, bitrate 10000000, and the network usage on my client application was 10mbit/s! I then tried a bitrate of 5000000, which gave me a network usage of 5mbit/s on the client application, so it works like a charm :D Thanks again!

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7411
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Problem with mjpeg using camera and v4l2

Tue Dec 01, 2015 8:01 am

Dom did two releases last night. The second did detail all my patches :)
Glad it's working for you too.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Affe
Posts: 8
Joined: Thu Nov 26, 2015 2:39 pm

Re: Problem with mjpeg using camera and v4l2

Tue Dec 01, 2015 9:11 am

Ah okay, fair enough :)

I think it's fair to say that i owe you a beer :D

Return to “Camera board”