Tw1stid
Posts: 20
Joined: Sat Apr 13, 2013 2:27 pm

Hardware H.264 Encode of Webcam

Fri Apr 26, 2013 7:59 am

Hi,

Been spending the last few weeks looking at recording video from a USB webcam. I've had some success using OpenCV and ffmpeg but I'm finding I'm getting poor FPS and resolution.

I'd like to encode my webcam stream (from /dev/video0, the webcam outputs in either YUYV or MJPEG) in hardware to H.264 - does anyone have an example of this? I've seen the encoding sample but not sure how I would go about modifying it to take video from the webcam. I'm hoping doing it in hardware means I can get better resolution and framerate.

Thanks for any help

Peace
Posts: 3
Joined: Fri Apr 26, 2013 12:06 pm

Re: Hardware H.264 Encode of Webcam

Fri Apr 26, 2013 12:12 pm

linux program called mjeg streamer

User avatar
fbutler
Posts: 302
Joined: Thu Mar 15, 2012 4:09 pm
Location: Surrey, England

Re: Hardware H.264 Encode of Webcam

Fri Apr 26, 2013 12:14 pm

Tw1stid wrote:I'd like to encode my webcam stream (from /dev/video0, the webcam outputs in either YUYV or MJPEG) in hardware to H.264 - does anyone have an example of this?
I've got some basic prototyping example code for this type of thing that you are welcome to experiment with. The v4l2capturetoH264.c source file and Makefile can be found at:

http://www.trans-omni.co.uk/capture

There is also a short two minute capture example, in the same location, which shows a capture of the big buck bunny movie using the code. This was captured with a STK1160 USB capture device using UYVY at 704x576. I believe that the corruption present in the video is probably due to the outstanding USB isochronous transfer issues being discussed in the USB Redux thread, although it could potentially be caused by something else. So hopefully (fingers crossed) once those issues have been resolved I'll know for certain.

The code is based on the hello_encode example and the v4l2 example capture program at http://linuxtv.org/downloads/v4l-dvb-ap ... ample.html. There's a few things you need to bear in mind when using the code:
  • - There are still outstanding issues with isochronous USB transfers. This may cause some corruption in your capture
    - Before encoding the input must be converted to a supported colour format and the width and height of the source must be a multiple of 32 as detailed here: http://home.nouwen.name/RaspberryPi/doc ... ncode.html. The code does a colour format conversion to OMX_COLOR_FormatYUV420PackedPlanar (YUV420P).
    - The code uses sws_scale to perform the conversion so it could potentially (Not Tested!!) be used to rescale the capture to the required width and height. However it would be best to configure the webcam input to meet the required width and height restrictions to prevent processing overheads
    - The colour format conversion is done in software and is CPU intensive. Expect ~60% usage when running the example code with the Pi overclocked at 900
    - The are #defines near the top of the source file where you can adjust various parameters such as INPUT_WIDTH, INPUT_HEIGHT, INPUT_PIX_FMT, INPUT_FRAMERATE 25, INPUT_LINESIZE
    - You will need to manually configure your input source to match the settings that you have specified in the code using v4l2-ctl
    - You will need to install the required libraries shown in the Makefile
    - You will need to build the ilclient library as described in the /opt/vc/src/hello_pi/README file
    - There are some command line options available when using the code. For instance you can specify the number of frames to capture using -c<number of frames>. It defaults to 250 frames. You can also redirect the output to stdout using the -o flag. This can be used to provide input into a GStreamer pipeline as described here: http://www.oz9aec.net/index.php/gstream ... -gstreamer
    - By default it will produce a test.h264 file. This can be played through omxplayer
    - I've only tested the code with a UYVY input at 704x576 but in theory it should work with any pixel format supported by the sws_scale function.
Let me know if you have any questions about the code. Have fun experimenting, and let us know what kind of results you get :-).

Tw1stid
Posts: 20
Joined: Sat Apr 13, 2013 2:27 pm

Re: Hardware H.264 Encode of Webcam

Fri Apr 26, 2013 12:31 pm

That looks really helpful, thanks! I'll be spending some time with this this weekend and hopefully I can get something working.

This is to be used for some sort of basic security recording so I'm fine with dropping the framerate dramatically (in all honesty, I don't really need anything above 5fps) and having no audio. Looking at your sample video the framerate looks great but there is quite a lot of corruption. Maybe with a lower framerate this will go away? Hope so.

Thanks for the help :)

User avatar
fbutler
Posts: 302
Joined: Thu Mar 15, 2012 4:09 pm
Location: Surrey, England

Re: Hardware H.264 Encode of Webcam

Fri Apr 26, 2013 12:50 pm

Yeah, I forgot to say that the capture was done at 25 FPS so it is really pushing things with the Pi's USB :-). I expect that at lower resolutions and frame rates the corruption would be considerably less, if not non-existent. Good luck with the code over the weekend.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 27432
Joined: Sat Jul 30, 2011 7:41 pm

Re: Hardware H.264 Encode of Webcam

Fri Apr 26, 2013 2:13 pm

It might be worth looking at the camera demo applications in github/userland - they encode direct from the camera, but you could supply the buffers yourself rather than connecting the camera. Uses the mmal api which is like an easier to use OpenMAX.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed.
I've been saying "Mucho" to my Spanish friend a lot more lately. It means a lot to him.

Tw1stid
Posts: 20
Joined: Sat Apr 13, 2013 2:27 pm

Re: Hardware H.264 Encode of Webcam

Fri Apr 26, 2013 2:22 pm

Thanks jamesh, I'll take a look at mmal

User avatar
fbutler
Posts: 302
Joined: Thu Mar 15, 2012 4:09 pm
Location: Surrey, England

Re: Hardware H.264 Encode of Webcam

Fri Apr 26, 2013 3:24 pm

jamesh wrote:It might be worth looking at the camera demo applications in github/userland - they encode direct from the camera, but you could supply the buffers yourself rather than connecting the camera. Uses the mmal api which is like an easier to use OpenMAX.
Thanks,

I'll also have a look at this next week when I get some time. It should be fairly straightforward to replace the omx specific code in the example v4l2 capture program, as currently there are just 3 places in the code for the OMX stuff: initialise_omx (), teardown_omx(), and process_image(). The relevant bits of code could easily be replaced with the corresponding mmal api bits, whilst leaving the v4l2 framework, the buffer preparation and the colour format transformation logic more or less as is.

Tw1stid
Posts: 20
Joined: Sat Apr 13, 2013 2:27 pm

Re: Hardware H.264 Encode of Webcam

Fri Apr 26, 2013 10:23 pm

fbutler wrote:Yeah, I forgot to say that the capture was done at 25 FPS so it is really pushing things with the Pi's USB :-). I expect that at lower resolutions and frame rates the corruption would be considerably less, if not non-existent. Good luck with the code over the weekend.
Thanks for the code. I've had a play with it tonight and have had *some* success. Had a few issues with frame timeouts but I seem to have got around them by not exiting whenever a frame times out - seems the first couple usually do for me.

The main problem I now have is the videos playback really fast. I know why this is, but can't work out how to fix it. My webcam is set up to output at 25fps however process_image is only getting called roughly 2-3 times a second. So by the time its received 25 frames for the first second of video its actually had about 10 seconds worth of real-time footage.

I'm not sure where the bottleneck in this is, I can only assume that the frame capture from the webcam is super slow. Assuming that is the case (and there is nothing I can do about it) then I see two solutions. 1) Change the output video framerate to match how fast I can grab from the webcam or 2) pad the output video with duplicate frames. I tried (1) by changing the xFramerate member but it seems whatever I change it to is ignored. Not sure if this a limitation of using H.264 or something? I also tried (2) by simply inserting the same frame 5 times as a test. This gave me really weird colour/ghosting issues.

Any ideas where I could go from here?

In the mean time, I'm going to check out this userland raspicam project although I suspect I'll run into similar issues.

Thanks for the help :D

Tw1stid
Posts: 20
Joined: Sat Apr 13, 2013 2:27 pm

Re: Hardware H.264 Encode of Webcam

Fri Apr 26, 2013 10:59 pm

jamesh wrote:It might be worth looking at the camera demo applications in github/userland - they encode direct from the camera, but you could supply the buffers yourself rather than connecting the camera. Uses the mmal api which is like an easier to use OpenMAX.
I tried this out but unfortunately it won't even connect to my camera :(

mmal: maml_vc_component_create: failed to create component 'vc.ril.camera' (1:ENOMEM)

I took a stab in the dark and tried passing in "/dev/video0" instead of MMAL_COMPONENT_DEFAULT_CAMERA (it may even evaluate it to that - maybe should have checked) but it made no difference.

Any ideas? Thanks

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 27432
Joined: Sat Jul 30, 2011 7:41 pm

Re: Hardware H.264 Encode of Webcam

Sat Apr 27, 2013 2:15 am

Tw1stid wrote:
jamesh wrote:It might be worth looking at the camera demo applications in github/userland - they encode direct from the camera, but you could supply the buffers yourself rather than connecting the camera. Uses the mmal api which is like an easier to use OpenMAX.
I tried this out but unfortunately it won't even connect to my camera :(

mmal: maml_vc_component_create: failed to create component 'vc.ril.camera' (1:ENOMEM)

I took a stab in the dark and tried passing in "/dev/video0" instead of MMAL_COMPONENT_DEFAULT_CAMERA (it may even evaluate it to that - maybe should have checked) but it made no difference.

Any ideas? Thanks
That example code is for a camera attached to the CSI-2 port, not a webcam. I probably wasn't clear that you should use the code as an example to write you own encoder code, rather than use it as is.

You'll be getting frames from the webcam (YUV format I reckon, I cannot remember which specific formats the encoder can manage though), presumably from V4L, which you will need to buffer up and pass to the mmal encoder component.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed.
I've been saying "Mucho" to my Spanish friend a lot more lately. It means a lot to him.

User avatar
fbutler
Posts: 302
Joined: Thu Mar 15, 2012 4:09 pm
Location: Surrey, England

Re: Hardware H.264 Encode of Webcam

Sat Apr 27, 2013 3:22 am

Tw1stid wrote:Thanks for the code. I've had a play with it tonight and have had *some* success. Had a few issues with frame timeouts but I seem to have got around them by not exiting whenever a frame times out - seems the first couple usually do for me.

The main problem I now have is the videos playback really fast. I know why this is, but can't work out how to fix it. My webcam is set up to output at 25fps however process_image is only getting called roughly 2-3 times a second. So by the time its received 25 frames for the first second of video its actually had about 10 seconds worth of real-time footage.

I'm not sure where the bottleneck in this is, I can only assume that the frame capture from the webcam is super slow. Assuming that is the case (and there is nothing I can do about it) then I see two solutions. 1) Change the output video framerate to match how fast I can grab from the webcam or 2) pad the output video with duplicate frames. I tried (1) by changing the xFramerate member but it seems whatever I change it to is ignored. Not sure if this a limitation of using H.264 or something? I also tried (2) by simply inserting the same frame 5 times as a test. This gave me really weird colour/ghosting issues.

Any ideas where I could go from here?
Some teething troubles using the code are to be expected :-) Some additional details on your setup would be useful to start to figure this out:
  • What webcam are you using?
    Can you post the output from a v4l2-ctl --all command?
    What are your overclocking settings?
    What class of SD card are you using?
    How are you logged into the Pi when running the code?
The first thing I would try is changing the code, and the webcam settings, to use a much lower frame rate. Try 5 fps or something like that. If you post the answers to the questions above we should be able to work out what's going on.

User avatar
fbutler
Posts: 302
Joined: Thu Mar 15, 2012 4:09 pm
Location: Surrey, England

Re: Hardware H.264 Encode of Webcam

Sat Apr 27, 2013 4:01 am

jamesh wrote:That example code is for a camera attached to the CSI-2 port, not a webcam. I probably wasn't clear that you should use the code as an example to write you own encoder code, rather than use it as is.

You'll be getting frames from the webcam (YUV format I reckon, I cannot remember which specific formats the encoder can manage though), presumably from V4L, which you will need to buffer up and pass to the mmal encoder component.
I had a quick look at the mmal camera code tonight and I think I've figured out the most of the initialisation and closedown bits for the encoder. Now I need to figure out how to separate the specific video capture buffer processing for the encoder from the rest of the RaspiVid camera code, and integrate it into the V4l2 framework example. It will probably be next week before I get a chance to have a proper look at this.

Tw1stid, keep experimenting with the V42l OMX example code for now as the mmal camera code Jamesh is talking about will only replace the OMX encoding parts of the code, and won't actually affect the overall functionality of the code. The key part of the V4l2 example code is the format conversion to prepare frames for the encoding, and the camera code doesn't do this as the input is already in the correct format.

Jamesh, out of curiosity is there any particular reason why the camera code uses "gotos" in the code?

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 27432
Joined: Sat Jul 30, 2011 7:41 pm

Re: Hardware H.264 Encode of Webcam

Sat Apr 27, 2013 6:09 am

fbutler wrote:
jamesh wrote:That example code is for a camera attached to the CSI-2 port, not a webcam. I probably wasn't clear that you should use the code as an example to write you own encoder code, rather than use it as is.

You'll be getting frames from the webcam (YUV format I reckon, I cannot remember which specific formats the encoder can manage though), presumably from V4L, which you will need to buffer up and pass to the mmal encoder component.
I had a quick look at the mmal camera code tonight and I think I've figured out the most of the initialisation and closedown bits for the encoder. Now I need to figure out how to separate the specific video capture buffer processing for the encoder from the rest of the RaspiVid camera code, and integrate it into the V4l2 framework example. It will probably be next week before I get a chance to have a proper look at this.

Tw1stid, keep experimenting with the V42l OMX example code for now as the mmal camera code Jamesh is talking about will only replace the OMX encoding parts of the code, and won't actually affect the overall functionality of the code. The key part of the V4l2 example code is the format conversion to prepare frames for the encoding, and the camera code doesn't do this as the input is already in the correct format.

Jamesh, out of curiosity is there any particular reason why the camera code uses "gotos" in the code?
re: Goto's : ease of error handling. Didn't have a huge amount of time to write the code, so took some easy routes. Although in this case the code would look a lot more complex without goto's. C++ exception handling would provide a better looking result. First goto's I used in years...

With regard to the buffer handling in to the encoder, look how its done for the output of the encoder (that's the point the buffers are moved from the GPU to the Arm and become visible to the code), and do something similar for the input.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed.
I've been saying "Mucho" to my Spanish friend a lot more lately. It means a lot to him.

User avatar
fbutler
Posts: 302
Joined: Thu Mar 15, 2012 4:09 pm
Location: Surrey, England

Re: Hardware H.264 Encode of Webcam

Sat Apr 27, 2013 6:33 am

jamesh wrote:With regard to the buffer handling in to the encoder, look how its done for the output of the encoder (that's the point the buffers are moved from the GPU to the Arm and become visible to the code), and do something similar for the input.
Thanks for the guidance.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 27432
Joined: Sat Jul 30, 2011 7:41 pm

Re: Hardware H.264 Encode of Webcam

Sat Apr 27, 2013 7:26 am

It's worth knowing that when using YUV420/2 as the input, then you need to align on 16 byte boundaries in x and y. So each line must be multiple of 16 pixels width, and so must EACH height of YU and V, so the Y part must be %16 = 0, as must U and V, so you may need to pad the data with 0 to ensure this. This is down to the way the encoder code in blocks of 16x16 and makes the HW easier to design if you put on these constraints.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed.
I've been saying "Mucho" to my Spanish friend a lot more lately. It means a lot to him.

User avatar
fbutler
Posts: 302
Joined: Thu Mar 15, 2012 4:09 pm
Location: Surrey, England

Re: Hardware H.264 Encode of Webcam

Sat Apr 27, 2013 8:14 am

jamesh wrote:It's worth knowing that when using YUV420/2 as the input, then you need to align on 16 byte boundaries in x and y. So each line must be multiple of 16 pixels width, and so must EACH height of YU and V, so the Y part must be %16 = 0, as must U and V, so you may need to pad the data with 0 to ensure this. This is down to the way the encoder code in blocks of 16x16 and makes the HW easier to design if you put on these constraints.
OK, good to know. I was using multiples of 32.

Tw1stid
Posts: 20
Joined: Sat Apr 13, 2013 2:27 pm

Re: Hardware H.264 Encode of Webcam

Sat Apr 27, 2013 8:34 am

Thanks for clearing that up jamesh.

So, back to fbutlers' sample code :)

The webcam is a Logitech C270, connected via USB hub although I did try connecting directly which made no difference
Output from v4l2-ctl --all is at the bottom of this post
Overclocked to 900MHz
Class 10 SD Card
I have been logged in to the UI (running startx) and without being logged into the UI, doesn't seem to make any difference.

As for changing the framerate, I have been trying but nothing seems to work! I've tried using v4l2-ctl --set-parm 5 (confirmed working via calling v4l2-ctl --get-parm). Then in the code I've tried changing the framerate define at the top to 5, and the xframerate to 5 >>16...but it makes no difference. Using omxplayer on the resulting .h264 always reports a framerate of 25 fps regardless of those lines I change in the code.

Thanks

Code: Select all

Driver Info (not using libv4l2):
	Driver name : uvcvideo
	Card type : UVC Camera (046d:0825)
	Bus info : usb-bcm2708_usb-1.3.3
	Driver version: 3.6.11
	Capabilities : 0x04000001
		Video Capture
		Streaming
Format Video Capture:
	Width/Height : 640/480
	Pixel Format : 'YUYV'
	Field : None
	Bytes per Line: 1280
	Size Image : 614400
	Colorspace : SRGB
Crop Capability Video Capture:
	Bounds : Left 0, Top 0, Width 640, Height 480
	Default : Left 0, Top 0, Width 640, Height 480
	Pixel Aspect: 1/1 Video input : 0 (Camera 1: ok)
Streaming Parameters Video Capture:
	Capabilities : timeperframe
	Frames per second: 5.000 (5/1)
	Read buffers : 0

User avatar
fbutler
Posts: 302
Joined: Thu Mar 15, 2012 4:09 pm
Location: Surrey, England

Re: Hardware H.264 Encode of Webcam

Sat Apr 27, 2013 8:49 am

Tw1stid wrote:As for changing the framerate, I have been trying but nothing seems to work!
Ahh, my fault. The line in the initialise_omx function():
def.format.video.xFramerate = 25 << 16;
should be changed to:
def.format.video.xFramerate = INPUT_FRAMERATE << 16;
I did warn you that it was prototype code :-)

Tw1stid
Posts: 20
Joined: Sat Apr 13, 2013 2:27 pm

Re: Hardware H.264 Encode of Webcam

Sat Apr 27, 2013 10:07 am

Thanks, but I'd already spotted that and changed it :(. It seems to have no effect on the output, omxplayer still shows the fps as 25. Have you had any success exporting video at 5 fps?

Cheers

Tw1stid
Posts: 20
Joined: Sat Apr 13, 2013 2:27 pm

Re: Hardware H.264 Encode of Webcam

Sat Apr 27, 2013 10:19 am

Its probably also worth mentioning that when I set the webcam to 5 fps using v4l2-ctl --set-parm 5 then I just get select timeout errors

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 27432
Joined: Sat Jul 30, 2011 7:41 pm

Re: Hardware H.264 Encode of Webcam

Sat Apr 27, 2013 10:57 am

Tw1stid wrote:Thanks, but I'd already spotted that and changed it :(. It seems to have no effect on the output, omxplayer still shows the fps as 25. Have you had any success exporting video at 5 fps?

Cheers
25 is the default in omxplayer, and a raw H264 stream has no fps data in it, so will play at the default speed. You need to add a container around the H264 if you want to specify it's fps. Note there is no way to pass the fps to OMX player on the command line (or there wasn't...)
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed.
I've been saying "Mucho" to my Spanish friend a lot more lately. It means a lot to him.

User avatar
fbutler
Posts: 302
Joined: Thu Mar 15, 2012 4:09 pm
Location: Surrey, England

Re: Hardware H.264 Encode of Webcam

Sat Apr 27, 2013 11:50 am

Tw1stid wrote:Its probably also worth mentioning that when I set the webcam to 5 fps using v4l2-ctl --set-parm 5 then I just get select timeout errors
Even when you have adjusted the code to use 5fps? I only see that error when there is a mismatch between the frame rate in the code and that set on the input device.

If you install gstreamer you should be able to control the frame rate that the file plays at with a pipeline of the form:

gst-launch-1.0 -v -e filesrc location=test.h264 ! video/x-h264,width=704, height=576, framerate=25/1 ! h264parse ! omxh264dec ! eglglessink

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 27432
Joined: Sat Jul 30, 2011 7:41 pm

Re: Hardware H.264 Encode of Webcam

Sat Apr 27, 2013 12:22 pm

fbutler wrote:
jamesh wrote:It's worth knowing that when using YUV420/2 as the input, then you need to align on 16 byte boundaries in x and y. So each line must be multiple of 16 pixels width, and so must EACH height of YU and V, so the Y part must be %16 = 0, as must U and V, so you may need to pad the data with 0 to ensure this. This is down to the way the encoder code in blocks of 16x16 and makes the HW easier to design if you put on these constraints.
OK, good to know. I was using multiples of 32.

Ooo. You've made we wonder now. Whichever ones give the right result will be the right one....
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed.
I've been saying "Mucho" to my Spanish friend a lot more lately. It means a lot to him.

User avatar
fbutler
Posts: 302
Joined: Thu Mar 15, 2012 4:09 pm
Location: Surrey, England

Re: Hardware H.264 Encode of Webcam

Sat Apr 27, 2013 12:30 pm

Ooo. You've made we wonder now. Whichever ones give the right result will be the right one....
You were quite right.... From:

http://home.nouwen.name/RaspberryPi/doc ... ncode.html

"nSliceHeight must be the same as nFrameHeight rounded up to the nearest multiple of 16. nStride must be a multiple of 32"

Return to “Graphics, sound and multimedia”