timgover
Posts: 8
Joined: Thu Nov 07, 2013 7:40 pm

Re: is there a way to get from mmal to opengl texture

Sun Nov 10, 2013 6:18 pm

Thanks Dom !

btw: In-case anyone's wondering there's a couple more changes in the pipeline. First, a simple tweak that allows the GL frame-buffer to be saved as the still in RaspiStill. Second, a fast-path for mapping the Y plane of the MMAL buffer to a texture as GL_LUMINANCE which should be useful for some image processing shaders. However, there's still a few bits left to do so I thought it best to release these changes separately.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23311
Joined: Sat Jul 30, 2011 7:41 pm

Re: is there a way to get from mmal to opengl texture

Tue Nov 12, 2013 10:09 am

Bump, 'cos this is good stuff!
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

lagurus
Posts: 46
Joined: Wed Aug 07, 2013 8:02 am

Impact on H264 framerate

Thu Nov 14, 2013 11:43 am

Thanks Dom for great work,

I just adapted GL texture feature into raspivid.

Everything seems to work, but when I activate GL (--gl switch), it has impact on H264 performance.

In normal state for 1920x1080 I can get h264 file with 30 fps, but with GL only 24 fps (RaspiTex report approx.: 41 FPS)
I use following command:

Code: Select all

raspivid -v -w 1920 -h 1080 -t 60000 -b 1000000  --preview '0,0,1280,720'  --gl --glwin '0,0,1920,1080' --glscene square -o /tmp/test.h264
Is it possible? Have someone similar experience?

Btw, is also possible, that I made some mistake.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23311
Joined: Sat Jul 30, 2011 7:41 pm

Re: is there a way to get from mmal to opengl texture

Thu Nov 14, 2013 1:40 pm

Its actually Tim Gover who did all the work on this - Dom just pushed the code to the repo.

As for performance, you are hitting the limits of what the Raspi is capable of. Encoding 1080p30 H264, whilst converting to a texture and mapping it to a 3D object. That's an awful lot of calculation, and an awful lot of memory bandwidth.

You can try overclocking to get a bit more speed out.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5307
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: is there a way to get from mmal to opengl texture

Thu Nov 14, 2013 3:56 pm

jamesh wrote:Its actually Tim Gover who did all the work on this - Dom just pushed the code to the repo.
I enouraged him. That's the important bit. :-P

@lagurus if you link to your code we might be able to tell if the code is right or wrong.
But as James says, if you are doing h264 encode at the same time as convert to tetxure and rendering, it will go slower than not having to h264 encode.
Overclocking (especially sdram) will help.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23311
Joined: Sat Jul 30, 2011 7:41 pm

Re: is there a way to get from mmal to opengl texture

Thu Nov 14, 2013 4:09 pm

dom wrote:
jamesh wrote:Its actually Tim Gover who did all the work on this - Dom just pushed the code to the repo.
I enouraged him. That's the important bit. :-P
:D
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

lagurus
Posts: 46
Joined: Wed Aug 07, 2013 8:02 am

Re: is there a way to get from mmal to opengl texture

Thu Nov 14, 2013 4:33 pm

dom wrote:
jamesh wrote:Its actually Tim Gover who did all the work on this - Dom just pushed the code to the repo.
I enouraged him. That's the important bit. :-P
Oh, I didn't know it, then thanks again to Tim Gover.
dom wrote: @lagurus if you link to your code we might be able to tell if the code is right or wrong.
I included my RaspiVid.c modification. It's the only file which was needed to modify.
dom wrote: But as James says, if you are doing h264 encode at the same time as convert to tetxure and rendering, it will go slower than not having to h264 encode.
Overclocking (especially sdram) will help.
I thought that. I'll try Overclocking.
Thanks for help.

BTW: I slowed down raspitex_draw function to 15FPS -> now is h264 video again on 30fps.
Attachments
RaspiVid.tar.gz
(12.62 KiB) Downloaded 245 times

timgover
Posts: 8
Joined: Thu Nov 07, 2013 7:40 pm

Re: is there a way to get from mmal to opengl texture

Thu Nov 14, 2013 6:38 pm

lagurus wrote:
I thought that. I'll try Overclocking.
Thanks for help.

BTW: I slowed down raspitex_draw function to 15FPS -> now is h264 video again on 30fps.
Thanks for posting the update to RaspiVid.c it looks good. As Dom and James say this is quite a tough use-case because the texture convert and GL render requires a 720p read, 720p write + 1080 write for GL framebuffer. It might be possible to tweak the RaspiTex to skip conversions if there is > 1 MMAL buffer available to give smoother GL animation.

lagurus
Posts: 46
Joined: Wed Aug 07, 2013 8:02 am

Re: is there a way to get from mmal to opengl texture

Thu Nov 14, 2013 8:41 pm

timgover wrote: Thanks for posting the update to RaspiVid.c it looks good. As Dom and James say this is quite a tough use-case because the texture convert and GL render requires a 720p read, 720p write + 1080 write for GL framebuffer. It might be possible to tweak the RaspiTex to skip conversions if there is > 1 MMAL buffer available to give smoother GL animation.
I just want to make some image processing stuff, so for me is not smooth animation and also high resolution necessary, but on the other hand for writing video file is maximal resolution and 30 fps important.
timgover wrote: btw: In-case anyone's wondering there's a couple more changes in the pipeline. First, a simple tweak that allows the GL frame-buffer to be saved as the still in RaspiStill. Second, a fast-path for mapping the Y plane of the MMAL buffer to a texture as GL_LUMINANCE which should be useful for some image processing shaders. However, there's still a few bits left to do so I thought it best to release these changes separately.
If I understand to it correctly you are planing to make fast mapping the Y plane of the MMAL buffer to a texture?
It will be definitely better for my purposes. If it is so, I will wait on it.

timgover
Posts: 8
Joined: Thu Nov 07, 2013 7:40 pm

Re: is there a way to get from mmal to opengl texture

Thu Nov 14, 2013 11:17 pm

lagurus wrote: If I understand to it correctly you are planing to make fast mapping the Y plane of the MMAL buffer to a texture?
It will be definitely better for my purposes. If it is so, I will wait on it.
Yes, I'm working on it at the moment and got first frames out this evening. Should be quite a lot faster than bouncing the buffers via the ARM especially if the only texture required for the shader is the Y texture. Probably another week or two though as the "correct" approach is not 100% clear just yet and I don't have much time to work on this.

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5307
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: is there a way to get from mmal to opengl texture

Thu Nov 28, 2013 10:42 pm

timgover wrote:Yes, I'm working on it at the moment and got first frames out this evening. Should be quite a lot faster than bouncing the buffers via the ARM especially if the only texture required for the shader is the Y texture. Probably another week or two though as the "correct" approach is not 100% clear just yet and I don't have much time to work on this.
I've just pushed Tim's updates out. See:
http://www.raspberrypi.org/phpBB3/viewt ... 40#p461142

lagurus
Posts: 46
Joined: Wed Aug 07, 2013 8:02 am

Re: is there a way to get from mmal to opengl texture

Mon Dec 02, 2013 3:53 pm

dom wrote:
timgover wrote:Yes, I'm working on it at the moment and got first frames out this evening. Should be quite a lot faster than bouncing the buffers via the ARM especially if the only texture required for the shader is the Y texture. Probably another week or two though as the "correct" approach is not 100% clear just yet and I don't have much time to work on this.
I've just pushed Tim's updates out. See:
http://www.raspberrypi.org/phpBB3/viewt ... 40#p461142
Many thanks Tim! Now is possible to reach 30fps for 1920x1080 H264 and at the same time process Y-texture at 30fps.

Just one question. On http://timgover.blogspot.co.uk/2013/11/ ... cdiag.html is mentioned
mmal_vc_diag command which can be used for performance analysis. But I am not able to use it (returns command not found), it means, that it wasn't yet released or is needed something to install?

timgover
Posts: 8
Joined: Thu Nov 07, 2013 7:40 pm

Re: is there a way to get from mmal to opengl texture

Mon Dec 02, 2013 5:28 pm

lagurus wrote:
dom wrote:
timgover wrote:Yes, I'm working on it at the moment and got first frames out this evening. Should be quite a lot faster than bouncing the buffers via the ARM especially if the only texture required for the shader is the Y texture. Probably another week or two though as the "correct" approach is not 100% clear just yet and I don't have much time to work on this.
I've just pushed Tim's updates out. See:
http://www.raspberrypi.org/phpBB3/viewt ... 40#p461142
Many thanks Tim! Now is possible to reach 30fps for 1920x1080 H264 and at the same time process Y-texture at 30fps.
No problem, glad to see this is giving a useful performance boost. I think the next thing I plan to look at is providing an alternative to glReadPixels by providing an API to create framebuffer-objects from memory mapped GPU memory. If there's other more useful improvements that could be made then please post suggestions here !
lagurus wrote: Just one question. On http://timgover.blogspot.co.uk/2013/11/ ... cdiag.html is mentioned
mmal_vc_diag command which can be used for performance analysis. But I am not able to use it (returns command not found), it means, that it wasn't yet released or is needed something to install?
mmal_vc_diag is included in the RPI firmware https://github.com/raspberrypi/firmware ... opt/vc/bin in /opt/vc/bin which probably isn't in your path by default. I think the more commonly used commands have symlinks from /usr/bin to /opt/vc/bin

User avatar
jbeale
Posts: 3474
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: is there a way to get from mmal to opengl texture

Mon Dec 02, 2013 5:58 pm

The ability to get Y camera data as a GL texture sounds very useful for processing tasks. I was able to make a simple motion detector using the Python interface, http://www.raspberrypi.org/phpBB3/viewt ... 75#p461876 but of course the pixels-per-second performance is limited. Looking at the various elements needed for GPU code makes me feel like a bear of very little brain. Are there any beginner tutorials for getting GL code running on R-Pi ? Is the change-detection algorithm (building up a long-term average "background" image, then for each new frame detecting and extracting areas of difference from that background) amenable to the vertex/fragment shader model?

Notes to self:
* Some OpenGL examples for R-Pi here: https://github.com/openFrameworks-arm/R ... 13Workshop although maybe not self-contained.
* The 'pyopengles' project (from 2012) would be nice if it worked, but it does not. (needs library update maybe? ) https://github.com/benosteen/pyopengles
* rpigl demos do work. After you move the 'rpigl' directory into the 'demos' directory, and run demos from there.

lagurus
Posts: 46
Joined: Wed Aug 07, 2013 8:02 am

Re: is there a way to get from mmal to opengl texture

Fri Jan 10, 2014 3:50 pm

timgover wrote: No problem, glad to see this is giving a useful performance boost. I think the next thing I plan to look at is providing an alternative to glReadPixels by providing an API to create framebuffer-objects from memory mapped GPU memory. If there's other more useful improvements that could be made then please post suggestions here !
Replace glReadPixels with another faster function will be very useful, any progress on this?

Very handy feature will be the possibility of connection already processed EGL buffers/textures back to MMAL (e.g.to jpeg/h264 encoder).


timgover
Posts: 8
Joined: Thu Nov 07, 2013 7:40 pm

Re: is there a way to get from mmal to opengl texture

Sun Jan 12, 2014 10:23 am

Sorry, I've been a bit busy over the last month and haven't had time to do any RPI work. However, we found a way of simplifying the Kernel driver support for this so I'll start hacking it next week. My current plan is to implement and release these changes in order.

The first is some enhancements to the vcio driver to enable GL to render to shared memory.

The second is tying in EGL fence sync to avoid too much synchronisation with the GL driver.

Then finally, it should be possible to accelerate the RGB to the YUV internal format required by the encoder. I'm hoping to do this in such a way that it works with either OpenMax or MMAL and will add the example code in RaspiVid.

If this fits (or doesn't) with what people want to do in this area please post feedback here !

silviodelgado70
Posts: 41
Joined: Sun Dec 01, 2013 10:36 pm

Re: is there a way to get from mmal to opengl texture

Mon Jan 13, 2014 5:46 pm

I'm working in autonomous mobile robotics, where the image processing is very important.
What i need is that the GPU perform the following task. Get the image, convert the color space to another different from YUV and downsample to lower resolution. Apply some gl shaders (gaussian, dilate, erode, etc). And then i can get the result image for further processing in openCV. This way the processing of the image wont load so much the CPU.
I think the first change you propose may help me to get the image to openCV, Am I wrong?
Is there already an efficient way to do this?

By the way, congratulations for your amazing results. I think your work is going to help many people.

Thanks in advance.
Silvio

lagurus
Posts: 46
Joined: Wed Aug 07, 2013 8:02 am

Re: is there a way to get from mmal to opengl texture

Fri Jan 17, 2014 10:13 pm

I want to use data directly from "RGBX" texture (EGL_IMAGE_BRCM_MULTIMEDIA) and simultaneosly save H264 video.

I have found that change of MMAL_PARAMETER_CAPTURE on camera_video_port also change orientation of RGBX texture - result is vertically flipped and it depends if is MMAL_PARAMETER_CAPTURE set to 0 or 1.
Other texture (y_texture,u_texture,v_texture) are not affected.

Has it any solution?

Thanks

timgover
Posts: 8
Joined: Thu Nov 07, 2013 7:40 pm

Re: is there a way to get from mmal to opengl texture

Sat Jan 18, 2014 8:53 pm

silviodelgado70 wrote:I'm working in autonomous mobile robotics, where the image processing is very important.
What i need is that the GPU perform the following task. Get the image, convert the color space to another different from YUV and downsample to lower resolution. Apply some gl shaders (gaussian, dilate, erode, etc). And then i can get the result image for further processing in openCV. This way the processing of the image wont load so much the CPU.
I think the first change you propose may help me to get the image to openCV, Am I wrong?
Is there already an efficient way to do this?

By the way, congratulations for your amazing results. I think your work is going to help many people.

Thanks in advance.
Silvio
Yep, the first change should help with that. There's probably some other things that could be done to run multiple gl shader passes (gaussian, dilate, erode, etc) in a chain, but one problem at a time :)

unphased
Posts: 23
Joined: Tue Apr 08, 2014 2:44 pm

Re: is there a way to get from mmal to opengl texture

Wed Apr 16, 2014 10:01 pm

I just wanted to comment that due to the foresight of the hardware designers, we have a Composite out port on the Raspberry Pi which allows us bypass the realtime H.264 encoding requirement for autonomous robotic operations.

FPV and other types of similar affordable analog video transmission technology can be used without modification with the Composite out. This allows us to free up the significant GPU workload that is full HD H.264 encoding, and we can then utilize those precious GPU resources for further computer vision acceleration.

I would consider this a very good compromise as it continues to enable realtime software processing of high resolution camera data, and may even allow us to push even greater resolution. This can definitely help control system quality by not limiting resolution based on streaming limits. In addition, it allows for independently driving a minimal latency video feed that is sufficient for almost all ground station purposes using a very nice point of data separation via video out (what I mean by this is that the same feed can then be plugged in to an HDMI display. Extremely convenient, and not addressed by a streaming-only solution).

User avatar
Gavinmc42
Posts: 3605
Joined: Wed Aug 28, 2013 3:31 am

Re: is there a way to get from mmal to opengl texture

Tue Apr 22, 2014 1:53 am

I did some testing using composite TV output.

Python code ..display.list modes() returns screen size as 656 x 416
This is good enough for a 320x240 video window with room for OSD around the edges.
Should have low latency using an analog TV RF transmitter, better than digital any way.

Wonder how to overlay OSD onto video?
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

kaikeraut
Posts: 30
Joined: Tue Apr 29, 2014 8:18 am

Re: is there a way to get from mmal to opengl texture

Tue May 27, 2014 5:49 am

Can anyone post the bare code just for camera preview using mmal or omx.

Thanks

Return to “Camera board”