ayronc
Posts: 3
Joined: Wed Nov 06, 2013 8:35 pm

GPU scaling of camera frames [Solved]

Wed Nov 06, 2013 9:14 pm

Hello all,

I've been doing some simple real-time image processing using the camera module.

To meet other optical requirements, I need to be using as much of the sensor size as possible. That is, I need to initially capture the full 2592 x 1952 resolution.
But because it would take too long to do the image processing on an image of this size, I need scale the image down to a more reasonable size.

Currently, I am doing this using the following method:
Using the MMAL API, I have created a callback that is connected directly to the camera component's video output port. In the callback function, I wrap the buffer data in an OpenCV Mat and perform the scaling using cv::resize.

With this method, the scaling is performed on the main CPU, and takes a good chuck of the amount of time I have for other calulations.

What I would really like, and am 99% sure is possible, is to perform the scaling on the GPU, but I have not figured out how to implement this.

I imagine it taking the form of just another mmal component that I can insert between the camera component and the callback function, just like the encode_video compoment in RaspiVid.c. In fact, there is a component listed in the mmal library called video_convert which may be the answer, but it seems to be undocumented with no example. I tried to implement it like video_encode is, but with no luck. Does anyone know how to implement that component, or if it is what I want.

Another option may be using OpenMax IL directly in addition to mmal, because it looks like OpenMax IL has a component called 'resize', which I'm sure does what I want, but I would like to find out if there is a simple option before putting the effort into implement this.

The thought has occured to me also, that I could chain encode_video to decode_video and have it output the resolution I want, but the thought of that cause me to cringe, and I refuse to do it.

Another possibly that I have tried is to get the camera_component to give me the scaled frame directly, as it appears to have some built-in scaling capability. But I have also been unable to get this to work in the way I wish. Most attempts ended up with scaled images, but with significant unwanted cropping.

Does anyone have any pointers?

On a similar note. Does anyone know if it is possible to run arbituary code on the gpu? Like a custom mmal component? This would be for the purpose of running the remainder of the image processing code.

Thanks.
Last edited by ayronc on Thu Nov 07, 2013 4:29 pm, edited 1 time in total.

MacHack
Posts: 21
Joined: Sat May 19, 2012 1:00 pm

Re: GPU scaling of camera frames

Thu Nov 07, 2013 12:36 am

Hi,

I think what you are looking for is the "vc.ril.resize" component. You can check my code from http://www.raspberrypi.org/phpBB3/viewt ... &start=100.
you should be able to attach it to the video port of the camera and have a callback function on the resizer to get the output image.

poing
Posts: 1131
Joined: Thu Mar 08, 2012 3:32 pm

Re: GPU scaling of camera frames

Thu Nov 07, 2013 12:47 am

Maybe I misunderstand what you're trying to do, but if you do something like

Code: Select all

raspivid -w 640 -h 480 -o test.h264
you get the full field of view only in a very small number of pixels.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 24956
Joined: Sat Jul 30, 2011 7:41 pm

Re: GPU scaling of camera frames

Thu Nov 07, 2013 9:22 am

As poing said, or use the resize component also mentioned.


You cannot run your own code on the GPU (except GPGPU stuff use OGLES shaders). The GPU code is closed source, and also requires custom compilers.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I own the world’s worst thesaurus. Not only is it awful, it’s awful."

wibble82
Posts: 66
Joined: Sun Jan 27, 2013 5:06 pm
Contact: Website

Re: GPU scaling of camera frames

Thu Nov 07, 2013 12:29 pm

Hi there. I'm doing exactly that in this code, which I released a while back: http://robotblogging.blogspot.co.uk/201 ... i-for.html. It is running on a video feed, but you could take the use of the resizer component and plug it into your own capture code. All the important stuff is in camera.cpp

Specifically, info on the resizer here: http://robotblogging.blogspot.co.uk/201 ... age-4.html

As an added bonus the resizer converts to RGB as well!

Hope that helps

-Chris

ayronc
Posts: 3
Joined: Wed Nov 06, 2013 8:35 pm

Re: GPU scaling of camera frames

Thu Nov 07, 2013 4:28 pm

Thanks guys.

vc.ril.resize certainly does the trick. Thanks wibble82 for all the work you've done on this. I wish I had discovered your blog a week ago.

Poing: Perhaps it is different for you, but running the command you suggested does introduce undesired cropping for me.

But you did make me realize that that raspistill always appears to capture using the entire sensor then scales to the desired resolution. I suppose that is the actaully difference between the camera component's still port and its video port, which I've never been too sure about before now.

Maybe someone can confirm this, but here's how my guess:
Capturing from the still port alway captures 2592x1952 pixels from the sensor, then scales to the requested resolution.
The video_port on the other hand captures 1920x1080 pixels from the sensor, which decreases the FoV, then scales to the desired resolution. **Unless you explicitly request 2592x1952 pixels, which it will then provide for you, without scaling.

Next I am going to see if capturing from the still port at 800x600, is equivalent to capturing from the video_port at 2592x1952 then resizing to 800x600.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 24956
Joined: Sat Jul 30, 2011 7:41 pm

Re: GPU scaling of camera frames [Solved]

Thu Nov 07, 2013 4:35 pm

Pretty much. Although that might change. We currently have two camera mode, full sensor and 1080p cropped. Unless told otherwise the preview/video port will use the 1080p mode, and capture port will use full res. Setting preview port using the full resolution preview option will force the preview to use the full frame, but consequently reduces the framerate to 15fps (but the FOV is the same for preview and capture)

If I add some new modes, the choosing of which mode to use for which resolution will likely change - but that's in the future.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I own the world’s worst thesaurus. Not only is it awful, it’s awful."

Return to “Camera board”