I've been doing some simple real-time image processing using the camera module.
To meet other optical requirements, I need to be using as much of the sensor size as possible. That is, I need to initially capture the full 2592 x 1952 resolution.
But because it would take too long to do the image processing on an image of this size, I need scale the image down to a more reasonable size.
Currently, I am doing this using the following method:
Using the MMAL API, I have created a callback that is connected directly to the camera component's video output port. In the callback function, I wrap the buffer data in an OpenCV Mat and perform the scaling using cv::resize.
With this method, the scaling is performed on the main CPU, and takes a good chuck of the amount of time I have for other calulations.
What I would really like, and am 99% sure is possible, is to perform the scaling on the GPU, but I have not figured out how to implement this.
I imagine it taking the form of just another mmal component that I can insert between the camera component and the callback function, just like the encode_video compoment in RaspiVid.c. In fact, there is a component listed in the mmal library called video_convert which may be the answer, but it seems to be undocumented with no example. I tried to implement it like video_encode is, but with no luck. Does anyone know how to implement that component, or if it is what I want.
Another option may be using OpenMax IL directly in addition to mmal, because it looks like OpenMax IL has a component called 'resize', which I'm sure does what I want, but I would like to find out if there is a simple option before putting the effort into implement this.
The thought has occured to me also, that I could chain encode_video to decode_video and have it output the resolution I want, but the thought of that cause me to cringe, and I refuse to do it.
Another possibly that I have tried is to get the camera_component to give me the scaled frame directly, as it appears to have some built-in scaling capability. But I have also been unable to get this to work in the way I wish. Most attempts ended up with scaled images, but with significant unwanted cropping.
Does anyone have any pointers?
On a similar note. Does anyone know if it is possible to run arbituary code on the gpu? Like a custom mmal component? This would be for the purpose of running the remainder of the image processing code.