kingbox1
Posts: 15
Joined: Fri Mar 29, 2019 6:21 am

OpenMax video covers all X window and keeps topmost

Fri Mar 29, 2019 6:30 am

Hi all
I am a newbie to openmax programming and i downloaded some openmax sample code here:https://github.com/popcornmix/omxplayer and i found that openmax video always keep topmost and covers all x windows on my platform. Is there anyway to render openmax video to an x window?

I checked that there is one flag called OMX_DISPLAY_SET_LAYER and tried to set it to 0, still no luck , x window still being coverred, what does this flag mean?

thanks

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7150
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: OpenMax video covers all X window and keeps topmost

Fri Mar 29, 2019 9:39 am

OMX video_render has no concept of X, it just applies an overlay to be composed by the hardware (the pixels don't get written to the framebuffer).

Conversely X has no concept of overlays, therefore composition into an X window on the RGBX framebuffer is a hugely processor intensive operation (format conversion, resize, and blit), and is one of the reasons that so much work has gone into making things like Chromium and VLC perform vaguely reasonably.

OMX_DISPLAY_SET_LAYER does what it says in setting the layer, however you need to appreciate that the framebuffer is on layer -127. You can alter that with a mailbox call, but without alpha being enabled, and something sane in the aplha plane of the framebuffer, it isn't going to help.

You probably want to look at OMX_DISPLAY_SET_DEST_RECT and set dest_rect in your OMX_CONFIG_DISPLAYREGIONTYPE, and also set OMX_DISPLAY_SET_FULLSCREEN and clear fullscreen. You can then move the overlay whereever you want on the screen.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

kingbox1
Posts: 15
Joined: Fri Mar 29, 2019 6:21 am

Re: OpenMax video covers all X window and keeps topmost

Fri Mar 29, 2019 9:51 am

6by9 wrote:
Fri Mar 29, 2019 9:39 am
OMX video_render has no concept of X, it just applies an overlay to be composed by the hardware (the pixels don't get written to the framebuffer).

Conversely X has no concept of overlays, therefore composition into an X window on the RGBX framebuffer is a hugely processor intensive operation (format conversion, resize, and blit), and is one of the reasons that so much work has gone into making things like Chromium and VLC perform vaguely reasonably.

OMX_DISPLAY_SET_LAYER does what it says in setting the layer, however you need to appreciate that the framebuffer is on layer -127. You can alter that with a mailbox call, but without alpha being enabled, and something sane in the aplha plane of the framebuffer, it isn't going to help.

You probably want to look at OMX_DISPLAY_SET_DEST_RECT and set dest_rect in your OMX_CONFIG_DISPLAYREGIONTYPE, and also set OMX_DISPLAY_SET_FULLSCREEN and clear fullscreen. You can then move the overlay whereever you want on the screen.
Thanks for your reply! by setting OMX_DISPLAY_SET_DEST_RECT, the video is rendered to part of the screen but it does not solve the issue that the video will cover the x window under it even if i have clicked the x window under it. Is there anyway to solve this?

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7150
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: OpenMax video covers all X window and keeps topmost

Fri Mar 29, 2019 10:03 am

kingbox1 wrote:
Fri Mar 29, 2019 9:51 am
Thanks for your reply! by setting OMX_DISPLAY_SET_DEST_RECT, the video is rendered to part of the screen but it does not solve the issue that the video will cover the x window under it even if i have clicked the x window under it. Is there anyway to solve this?
Not using video_render. You are not rendering into the window, you are adding an overlay on top of the frame buffer.

You can pull the pixels back to the ARM and blit them actually into the window surface, but as I have said that is CPU and memory bandwidth intensive.
You haven't said what content you're presenting to video_render, but you could look at the Raspbian patches to Chromium or VLC which resize video decode data to exactly the right size for X to "simply" have to do a blit into the appropriate buffer. With a moderately sized window you are still taking a fair hit with the blit. Those patches are using MMAL instead of IL as it is the better supported API and has various features that allow for better performance.

There are rendering paths available via OpenGL to render external content into the frame buffer, but even there there are significant overheads.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

kingbox1
Posts: 15
Joined: Fri Mar 29, 2019 6:21 am

Re: OpenMax video covers all X window and keeps topmost

Wed Apr 03, 2019 7:40 am

6by9 wrote:
Fri Mar 29, 2019 10:03 am
kingbox1 wrote:
Fri Mar 29, 2019 9:51 am
Thanks for your reply! by setting OMX_DISPLAY_SET_DEST_RECT, the video is rendered to part of the screen but it does not solve the issue that the video will cover the x window under it even if i have clicked the x window under it. Is there anyway to solve this?
Not using video_render. You are not rendering into the window, you are adding an overlay on top of the frame buffer.

You can pull the pixels back to the ARM and blit them actually into the window surface, but as I have said that is CPU and memory bandwidth intensive.
You haven't said what content you're presenting to video_render, but you could look at the Raspbian patches to Chromium or VLC which resize video decode data to exactly the right size for X to "simply" have to do a blit into the appropriate buffer. With a moderately sized window you are still taking a fair hit with the blit. Those patches are using MMAL instead of IL as it is the better supported API and has various features that allow for better performance.

There are rendering paths available via OpenGL to render external content into the frame buffer, but even there there are significant overheads.
Thanks for your patient explanation. I am presenting a decoded h264 stream to video_render. You are mentioning that framebuffer is of layer -127. Can you explain a little bit here about this concept layer? or you can point me some material that i can learn myself, I googled a lot using keywords"framebuffer -127", but i did not find much information about this.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23399
Joined: Sat Jul 30, 2011 7:41 pm

Re: OpenMax video covers all X window and keeps topmost

Wed Apr 03, 2019 8:45 am

kingbox1 wrote:
Wed Apr 03, 2019 7:40 am
6by9 wrote:
Fri Mar 29, 2019 10:03 am
kingbox1 wrote:
Fri Mar 29, 2019 9:51 am
Thanks for your reply! by setting OMX_DISPLAY_SET_DEST_RECT, the video is rendered to part of the screen but it does not solve the issue that the video will cover the x window under it even if i have clicked the x window under it. Is there anyway to solve this?
Not using video_render. You are not rendering into the window, you are adding an overlay on top of the frame buffer.

You can pull the pixels back to the ARM and blit them actually into the window surface, but as I have said that is CPU and memory bandwidth intensive.
You haven't said what content you're presenting to video_render, but you could look at the Raspbian patches to Chromium or VLC which resize video decode data to exactly the right size for X to "simply" have to do a blit into the appropriate buffer. With a moderately sized window you are still taking a fair hit with the blit. Those patches are using MMAL instead of IL as it is the better supported API and has various features that allow for better performance.

There are rendering paths available via OpenGL to render external content into the frame buffer, but even there there are significant overheads.
Thanks for your patient explanation. I am presenting a decoded h264 stream to video_render. You are mentioning that framebuffer is of layer -127. Can you explain a little bit here about this concept layer? or you can point me some material that i can learn myself, I googled a lot using keywords"framebuffer -127", but i did not find much information about this.
Imagine layers of graphics in the same way as layers of sheet of paper. The sheet at the bottom is layer -127, the piece of paper above that is -126 etc. Now imagine those sheets of paper are offset from each other, or have areas cut out of them to you can see bits fo the images on the sheets below.. This is sort of how the compositor works. See https://en.wikipedia.org/wiki/Layers_(d ... e_editing) and https://en.wikipedia.org/wiki/Alpha_compositing
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

kingbox1
Posts: 15
Joined: Fri Mar 29, 2019 6:21 am

Re: OpenMax video covers all X window and keeps topmost

Wed Apr 03, 2019 8:52 am

jamesh wrote:
Wed Apr 03, 2019 8:45 am
kingbox1 wrote:
Wed Apr 03, 2019 7:40 am
6by9 wrote:
Fri Mar 29, 2019 10:03 am


Not using video_render. You are not rendering into the window, you are adding an overlay on top of the frame buffer.

You can pull the pixels back to the ARM and blit them actually into the window surface, but as I have said that is CPU and memory bandwidth intensive.
You haven't said what content you're presenting to video_render, but you could look at the Raspbian patches to Chromium or VLC which resize video decode data to exactly the right size for X to "simply" have to do a blit into the appropriate buffer. With a moderately sized window you are still taking a fair hit with the blit. Those patches are using MMAL instead of IL as it is the better supported API and has various features that allow for better performance.

There are rendering paths available via OpenGL to render external content into the frame buffer, but even there there are significant overheads.
Thanks for your patient explanation. I am presenting a decoded h264 stream to video_render. You are mentioning that framebuffer is of layer -127. Can you explain a little bit here about this concept layer? or you can point me some material that i can learn myself, I googled a lot using keywords"framebuffer -127", but i did not find much information about this.
Imagine layers of graphics in the same way as layers of sheet of paper. The sheet at the bottom is layer -127, the piece of paper above that is -126 etc. Now imagine those sheets of paper are offset from each other, or have areas cut out of them to you can see bits fo the images on the sheets below.. This is sort of how the compositor works. See https://en.wikipedia.org/wiki/Layers_(d ... e_editing) and https://en.wikipedia.org/wiki/Alpha_compositing
Yeah, i can understand that layer with higher number appears on top of layer with smaller number. I mean i did not find any material saying that framebuffer's layer number is -127 and i did not find material saying that GPU is presenting the image by compoisting different layers

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23399
Joined: Sat Jul 30, 2011 7:41 pm

Re: OpenMax video covers all X window and keeps topmost

Wed Apr 03, 2019 9:38 am

kingbox1 wrote:
Wed Apr 03, 2019 8:52 am
jamesh wrote:
Wed Apr 03, 2019 8:45 am
kingbox1 wrote:
Wed Apr 03, 2019 7:40 am

Thanks for your patient explanation. I am presenting a decoded h264 stream to video_render. You are mentioning that framebuffer is of layer -127. Can you explain a little bit here about this concept layer? or you can point me some material that i can learn myself, I googled a lot using keywords"framebuffer -127", but i did not find much information about this.
Imagine layers of graphics in the same way as layers of sheet of paper. The sheet at the bottom is layer -127, the piece of paper above that is -126 etc. Now imagine those sheets of paper are offset from each other, or have areas cut out of them to you can see bits fo the images on the sheets below.. This is sort of how the compositor works. See https://en.wikipedia.org/wiki/Layers_(d ... e_editing) and https://en.wikipedia.org/wiki/Alpha_compositing
Yeah, i can understand that layer with higher number appears on top of layer with smaller number. I mean i did not find any material saying that framebuffer's layer number is -127 and i did not find material saying that GPU is presenting the image by compoisting different layers
Not sure there is any specific documenation about that.

The framebuffer is given the lowest layer, which is -127.

Most GPU's use compositing, in our case the VC4 does real time compositing of multiple bitmaps directly to the output devices all in hardware. Note that in usual linux use that feature is not used, as X by default does all its rendering in sotware. However OMXplayer or the camera apps preview will use the HW compositor as it provides by far the fastest way of getting the image on the display.

Using the beta graphics driver will make better use of the GPU compositor.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

kingbox1
Posts: 15
Joined: Fri Mar 29, 2019 6:21 am

Re: OpenMax video covers all X window and keeps topmost

Mon Jul 22, 2019 6:52 am

jamesh wrote:
Wed Apr 03, 2019 9:38 am
kingbox1 wrote:
Wed Apr 03, 2019 8:52 am
jamesh wrote:
Wed Apr 03, 2019 8:45 am


Imagine layers of graphics in the same way as layers of sheet of paper. The sheet at the bottom is layer -127, the piece of paper above that is -126 etc. Now imagine those sheets of paper are offset from each other, or have areas cut out of them to you can see bits fo the images on the sheets below.. This is sort of how the compositor works. See https://en.wikipedia.org/wiki/Layers_(d ... e_editing) and https://en.wikipedia.org/wiki/Alpha_compositing
Yeah, i can understand that layer with higher number appears on top of layer with smaller number. I mean i did not find any material saying that framebuffer's layer number is -127 and i did not find material saying that GPU is presenting the image by compoisting different layers
Not sure there is any specific documenation about that.

The framebuffer is given the lowest layer, which is -127.

Most GPU's use compositing, in our case the VC4 does real time compositing of multiple bitmaps directly to the output devices all in hardware. Note that in usual linux use that feature is not used, as X by default does all its rendering in sotware. However OMXplayer or the camera apps preview will use the HW compositor as it provides by far the fastest way of getting the image on the display.

Using the beta graphics driver will make better use of the GPU compositor.
Hi james
Recently, we got our new pi4 installed with buster image and i found something about graphics is different from what it was on pi3.
Previously, when I play video using openmax on pi3, I noticed that the cursor will be covered by the video. which means that the cursor GPU render layer is lower then openmax video which you explained to me before, however, I noticed that on pi4, the cursor will not be covered by openmax video. So the cursor layer is changed on pi4? Is there anyting else different about graphics on pi4?

thanks

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23399
Joined: Sat Jul 30, 2011 7:41 pm

Re: OpenMax video covers all X window and keeps topmost

Mon Jul 22, 2019 8:31 am

kingbox1 wrote:
Mon Jul 22, 2019 6:52 am
jamesh wrote:
Wed Apr 03, 2019 9:38 am
kingbox1 wrote:
Wed Apr 03, 2019 8:52 am

Yeah, i can understand that layer with higher number appears on top of layer with smaller number. I mean i did not find any material saying that framebuffer's layer number is -127 and i did not find material saying that GPU is presenting the image by compoisting different layers
Not sure there is any specific documenation about that.

The framebuffer is given the lowest layer, which is -127.

Most GPU's use compositing, in our case the VC4 does real time compositing of multiple bitmaps directly to the output devices all in hardware. Note that in usual linux use that feature is not used, as X by default does all its rendering in sotware. However OMXplayer or the camera apps preview will use the HW compositor as it provides by far the fastest way of getting the image on the display.

Using the beta graphics driver will make better use of the GPU compositor.
Hi james
Recently, we got our new pi4 installed with buster image and i found something about graphics is different from what it was on pi3.
Previously, when I play video using openmax on pi3, I noticed that the cursor will be covered by the video. which means that the cursor GPU render layer is lower then openmax video which you explained to me before, however, I noticed that on pi4, the cursor will not be covered by openmax video. So the cursor layer is changed on pi4? Is there anyting else different about graphics on pi4?

thanks
The cursor is now on its own HW layer (in fact, I did do a HW accelerated cursor some years ago that was never merged which had the same effect).

The graphics on the Pi4 are quite difference - new GPU (VC6), plus lots of work on the FKMS driver, MESA and DRM. However, from an end users point of view, there should be minimal differences.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

pik33
Posts: 173
Joined: Thu Sep 10, 2015 4:26 pm

Re: OpenMax video covers all X window and keeps topmost

Tue Jul 30, 2019 8:16 pm

The framebuffer is given the lowest layer, which is -127.
I experimented with layer <-127 It worked, if the main fb had the alpha channel enabled in config.txt and it was visible via transparent sections in the framebuffer. It was done in Ultibo, but it seems its framebuffer (using standard RPi mailbox interface to create a framebuffer) is also layer #-127

(or maybe the fb with alpha enabled is not layerr -127?)

User avatar
dividuum
Posts: 168
Joined: Sun Jun 16, 2013 1:18 pm
Location: Germany
Contact: Website

Re: OpenMax video covers all X window and keeps topmost

Thu Aug 08, 2019 3:33 pm

You should be able to find that out if you run `vcgencmd dispmanx_list`. You'll see the various dispmanx parameters used to compose the final video output.
info-beamer hosted - A user and programmer friendly digital signage platform for the Pi: https://info-beamer.com/hosted

Return to “OpenMAX”