wibble82
Posts: 66
Joined: Sun Jan 27, 2013 5:06 pm
Contact: Website

is there a way to get from mmal to opengl texture

Sat Oct 26, 2013 10:59 pm

Hey

I've been tinkering with the innards of mmal to make a little camera api, but want to mess with accelerating it using opengl. Currently I have to take the buffer from mmal, into cpu accesible memory, then hand it to opengl to put in a texture. It strikes me though that mmal probably already has a gpu accesible copy.

Does anybody know if there's a way of getting it directly to opengl in some way, or just any other more optimal means to get from an mmal port output into opengl for use by a shader.

Thanks

-Chris

ejarosek
Posts: 29
Joined: Mon Aug 27, 2012 6:39 pm

Re: is there a way to get from mmal to opengl texture

Mon Oct 28, 2013 5:03 am

If you look over in the Graphics Programming / OpenMAX forum area there are a few people who have captured from the camera using the OpenMAX framework instead of MMAL. ..I think.. that if you work in that framework it is much easier to send the video over into the OpenGL framework.

drhastings
Posts: 113
Joined: Wed Feb 06, 2013 11:38 pm

Re: is there a way to get from mmal to opengl texture

Mon Oct 28, 2013 5:25 am

If I understand what you are working on than you should be able to use the openmax camera component and the egl_render component to achieve what you are looking for.

You can find a set of openmax demo programs in the openmax forum. I've had success combining the camera capture example with the hello_teapot example found in /opt/vc/src/hello_pi to render the camera capture to an opengl texture.

You could probably copy the video data out in your filled buffer callback before sending it to opengl. If not I imagine you could use the video_splitter component to achieve that instead. You can definitely use glreadpixels to get the info after your shaders have done their work.

I don't know how you could do this using mmal. I'll pare what I've got down to a simple render to fullscreen texture and post it to github in the next few days if that would be helpful. I need to write some shaders but from your other post in this sub forum we are working on exactly the same thing. I've had a sobel filter running at 1920x1080 @30fps with maybe 100ms lag, but thats just an eyeball based guess on the latency.

glreadpixels is the bottleneck it seems.
http://www.dansrobotprojects.com/

drhastings
Posts: 113
Joined: Wed Feb 06, 2013 11:38 pm

Re: is there a way to get from mmal to opengl texture

Mon Oct 28, 2013 6:58 am

Its a bit hacky and the opengl code needs to be cleaned up but it work: https://github.com/drhastings/cameragl.
http://www.dansrobotprojects.com/

wibble82
Posts: 66
Joined: Sun Jan 27, 2013 5:06 pm
Contact: Website

Re: is there a way to get from mmal to opengl texture

Mon Oct 28, 2013 7:27 am

Thanks - that looks great! I am currently copying to a buffer from mmal, then copying the pixels into a texture as you suggest. Performance isn't bad but it's frustratingly slow given the gpu can handle the image fine.

I suspected ditching mmal and going straight to openmax might be the best approach so your work will be a massive help. I've already got a bunch of filter and conversion shaders working, so once the gpu has it ill be in business!

Thanks

- chris

wibble82
Posts: 66
Joined: Sun Jan 27, 2013 5:06 pm
Contact: Website

Re: is there a way to get from mmal to opengl texture

Mon Oct 28, 2013 8:22 am

wow - on closer inspection that looks greater than great! Was going to have a go at writing it but that should work just with a few tweaks. I'm going to build it into another friendly camera api like my previous mmal based one - I'm guessing you'd have no issue with me sharing it around once done? - obviously I'd credit you for it :)

fyi, the readpixels code is definitely going to be the bottleneck - basically the first bit of cpu work is where it'll slow down with a big image. The arm can not handle a 1080p image at 30hz (maybe at 15hz, with nothing else going on). However what you can do (and what I intend to do) is image processing on a lower resolution image, then when you know the 'interesting' areas, only read those from the hi res image. So for example psuedo code with face detection:

lo_res_bytes = readpixels(0,0,lo_res_width,lo_res_height,lo_res_texture)
facerectangles = getthefacerectangles(lo_res_bytes)
foreach rect in facerectangles:
hi_res_bytes = readpixels(rect.x,rect.y,rect.width,rect.height,hi_res_texture)

Once you have the hi res bytes you can just show them, or submit them for further processing. Obviously this process gets less efficient if you have lots of / big rectangles, but still useful.

-Chris

p.s. my filters are up on that site if you want to save a bit of work writing new ones

drhastings
Posts: 113
Joined: Wed Feb 06, 2013 11:38 pm

Re: is there a way to get from mmal to opengl texture

Mon Oct 28, 2013 3:33 pm

Sure do with it what you will, that's why I posted it! I'll definitely be taking a look at your shaders.

I was also thinking to do all the shader work at high res then rendering to a smaller buffer than the one used for the capture to do the pixel read on the whole thing. I don't know how you will get a lower res copy otherwise. I'm not really looking at face detection as my robot can't really look up that far off the ground, I'm more looking for blob detection. I'm trying to see how much I can get done in shaders for that.

Let me know if you have any questions about the program, the comments may be deceiving in places.

Now I can't be sure but from my brief experimenting with this the glreadpixels call seemed slower that reading out an mmal buffer full of pixel data.
http://www.dansrobotprojects.com/

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23062
Joined: Sat Jul 30, 2011 7:41 pm

Re: is there a way to get from mmal to opengl texture

Mon Oct 28, 2013 4:24 pm

We are looking in to keeping the whole image path to texture on the GPU - should have something for next week (I'm not doing the work although I started looking at it some time ago), there's another guy here at brcm taking a shufti. No promises.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

drhastings
Posts: 113
Joined: Wed Feb 06, 2013 11:38 pm

Re: is there a way to get from mmal to opengl texture

Mon Oct 28, 2013 4:42 pm

jamesh wrote:We are looking in to keeping the whole image path to texture on the GPU
Isn't this already accomplished using the egl_render openmax component?
http://www.dansrobotprojects.com/

wibble82
Posts: 66
Joined: Sun Jan 27, 2013 5:06 pm
Contact: Website

Re: is there a way to get from mmal to opengl texture

Mon Oct 28, 2013 4:47 pm

@jamesh - awesome! Dom mentioned it on my blog - very excited. Maybe I'll hold off OMX for a few days then and see what happens

@drhastings I had a blast at running the filters on full res images, but found at 720p the more complex filters (like 5x5 gaussian blur) started to hit framerate. That said, I was attempting to do multiple stages in a single frame. If you accepted lag and pipelined them over multiple frames it might work better. fyi, a technique I've used at work for gpu stuff might help you a bit with your blob detection. It goes something like this:

1. Run your image processing algorithms on a high res image. You should ultimately end up with a threshold image.
2. Presumably (as you've been using opengl) you're still writing to rgba textures. Use a shader to compress this to 1/4 size by using each rgba value to store 4 pixels in the space of 1 in the render target.
3. Now that you have a grey scale image you can effectively treat your future shaders as doing 4 pixels at once (as you have 4 grey pixels per single rgba pixel value).
4. However because it's a threshold image, it's really just black/white. Now you can compress this by taking 8 colour texture samples per fragment shader execution, which gives you a total of 32 grey scale pixels that are either 0 or 1. Encode these as individual bits in the colour channel just by multiplying each entry by a power of 2 divided by 255.

Basically that allows you to take the original render target for the threshold image and compress it to 1 bit per pixel, which can be read by the cpu much faster. It also has the added advantage of allowing your cpu analysis to load 32 pixels into a single uint, and if the uint is 0 ignore them all. If not, split it into 2 ushorts and check those etc etc.

The other nice trick is quadrant down sampling, so you make progressive downsamples of an image where each pixel in a 1/2 size downsample is the max of the 4 pixels it corresponds to in the larger image. Repeating this process say 4 times allows you to eliminate big empty regions of a large image by just checking the pixels in a smaller image.

That sounds a bit rambly when I write it down - I might do the code myself soon though so I can send it over. The tricks work pretty well though - we've used it for evaluating signed distance fields on a gpu (basically counting how many voxels are 'within' a CGS object).

drhastings
Posts: 113
Joined: Wed Feb 06, 2013 11:38 pm

Re: is there a way to get from mmal to opengl texture

Mon Oct 28, 2013 5:39 pm

@wibble82: The multiple texture reads in a shader seems to be the fastest way to degrade performance that I've found but that may be common knowledge, this is really my first foray into opengl and gpu programming.

Thanks for the tips on packing in the data for the read, that is good food for thought. I was hoping to go even a step beyond perhaps and see if I can't end up with a 'pixel' that encodes where the center of the largest blob is, then just read that one back but that will take staring at my whiteboard and some research before I have anything more interesting to say about it.
http://www.dansrobotprojects.com/

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5282
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: is there a way to get from mmal to opengl texture

Mon Oct 28, 2013 6:15 pm

drhastings wrote:Isn't this already accomplished using the egl_render openmax component?
Yes. If you are using openmax, then you can already efficiently convert camera data to a texture with an omx pipeline.
(and thanks to your cameragl work there is already example code).

We'd like to make that possible through mmal too, as most of the camera example code uses that interface.

drhastings
Posts: 113
Joined: Wed Feb 06, 2013 11:38 pm

Re: is there a way to get from mmal to opengl texture

Mon Oct 28, 2013 6:30 pm

dom wrote:
drhastings wrote:Isn't this already accomplished using the egl_render openmax component?
Yes. If you are using openmax, then you can already efficiently convert camera data to a texture with an omx pipeline.
(and thanks to your cameragl work there is already example code).

We'd like to make that possible through mmal too, as most of the camera example code uses that interface.
Gotcha, I'd rather work with the mmal stuff myself. The interfaces seem conceptually similar though openmax is a bit more verbose and arcane to work with in my brief experience.
http://www.dansrobotprojects.com/

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23062
Joined: Sat Jul 30, 2011 7:41 pm

Re: is there a way to get from mmal to opengl texture

Mon Oct 28, 2013 9:08 pm

drhastings wrote:
dom wrote:
drhastings wrote:Isn't this already accomplished using the egl_render openmax component?
Yes. If you are using openmax, then you can already efficiently convert camera data to a texture with an omx pipeline.
(and thanks to your cameragl work there is already example code).

We'd like to make that possible through mmal too, as most of the camera example code uses that interface.
Gotcha, I'd rather work with the mmal stuff myself. The interfaces seem conceptually similar though openmax is a bit more verbose and arcane to work with in my brief experience.
I agree, I find OpenMAX fairly impenetrable. The principles are simple, the implementation and use are not.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

wibble82
Posts: 66
Joined: Sun Jan 27, 2013 5:06 pm
Contact: Website

Re: is there a way to get from mmal to opengl texture

Mon Oct 28, 2013 11:12 pm

Hey

I was just thinking the other day when I looked into this. Isn't mmal just a wrapper around OMX?

This thread makes me wonder if simply allowing the user to interface OMX with mmal would be a useful approach - it'd mean that if a feature hadn't been implemented yet in mmal we could revert to the harder stuff until it was there.

I'm guessing mmal source is proprietary and that's why it doesn't come downloadable? Or have I just not found the source?

-Chris

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23062
Joined: Sat Jul 30, 2011 7:41 pm

Re: is there a way to get from mmal to opengl texture

Tue Oct 29, 2013 7:54 am

Quite a bit of the source is in userland, but there is stuff on the GPU as well.

I think you can write a MMAL component wrapper over OMX components (which is sort of what it is), to its usable in the MMAL framework, but whether that can be done entirely in ARM code I don't know.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

wibble82
Posts: 66
Joined: Sun Jan 27, 2013 5:06 pm
Contact: Website

Re: is there a way to get from mmal to opengl texture

Tue Oct 29, 2013 8:14 am

I think last time I looked I could find the mmal source, but no examples of how to actually create a component. Maybe they're buried deep in there somewhere, but it strikes me that an api not much more complex than:

mmal_connect_port_to_omx_tunnel(bla)

Could be written.

One for later though I guess - once I can get data in an eglimage I can reproduce most things in a shader anyhoo :)

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23062
Joined: Sat Jul 30, 2011 7:41 pm

Re: is there a way to get from mmal to opengl texture

Tue Oct 29, 2013 9:31 am

wibble82 wrote:I think last time I looked I could find the mmal source, but no examples of how to actually create a component. Maybe they're buried deep in there somewhere, but it strikes me that an api not much more complex than:

mmal_connect_port_to_omx_tunnel(bla)
No, its much much more complicated than that. MMAL handles a lot of complexity in the OMX handling (that's why it was written), and there is quite a lot of MMAL code....(.../userland/interface/mmal/...)
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

wibble82
Posts: 66
Joined: Sun Jan 27, 2013 5:06 pm
Contact: Website

Re: is there a way to get from mmal to opengl texture

Tue Nov 05, 2013 7:27 pm

Hey - just wondering if there's any update on that mmal support for render-to-egl-image? I've kind of got things working in OMX but its very unpleasant, and I can't get things to shutdown properly.

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5282
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: is there a way to get from mmal to opengl texture

Tue Nov 05, 2013 11:44 pm

wibble82 wrote:Hey - just wondering if there's any update on that mmal support for render-to-egl-image? I've kind of got things working in OMX but its very unpleasant, and I can't get things to shutdown properly.
I did see camera on a gl surface on a colleague's Pi (using mmal), and he was talking about getting James to okay the raspicam/texture demo and how to check it in, so I think it should be there in a day or two.

wibble82
Posts: 66
Joined: Sun Jan 27, 2013 5:06 pm
Contact: Website

Re: is there a way to get from mmal to opengl texture

Wed Nov 06, 2013 6:04 pm

Awesome! Looking forward to it

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23062
Joined: Sat Jul 30, 2011 7:41 pm

Re: is there a way to get from mmal to opengl texture

Wed Nov 06, 2013 6:56 pm

I'm hoping to have a look at the code and try it out tomorrow, hopefully for release next week. It will require a lot of updates to firmware and userland.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5282
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: is there a way to get from mmal to opengl texture

Sun Nov 10, 2013 5:52 pm

It's here! rpi-update to get it.
raspistil has been updated to make use of it. Thanks to my colleague Tim Gover for getting this working:

Examples:

raspistill -t 9999999 --preview '0,0,1280,720' --gl --glwin '0,0,1920,1080' --glscene teapot
raspistill -t 9999999 --preview '0,0,1280,720' --gl --glwin '0,0,1920,1080' --glscene square
# GL ES2.0 shader demo
raspistill -t 9999999 --preview '0,0,1280,720' --gl --glwin '0,0,1920,1080' --glscene mirror

Note: using camera and GL requires a fair bit of GPU memory. If you want both GL and preview at 1080p you may need more than 128M of gpu_mem.

wibble82
Posts: 66
Joined: Sun Jan 27, 2013 5:06 pm
Contact: Website

Re: is there a way to get from mmal to opengl texture

Sun Nov 10, 2013 5:57 pm

Brilliant! Is it just latest userland I need to get it going?

wibble82
Posts: 66
Joined: Sun Jan 27, 2013 5:06 pm
Contact: Website

Re: is there a way to get from mmal to opengl texture

Sun Nov 10, 2013 5:58 pm

Oops. - answerd my own question by reading your post properly :)

Return to “Camera board”