piense
Posts: 27
Joined: Thu Jun 09, 2016 4:11 am

Memory & compositing architecture questions

Fri Feb 02, 2018 7:40 pm

Just had a couple quick architecture related questions as I work on structuring my app. It's basically going to be a digital signage app based on OpenMAX and a buildroot image. I'm trying to understand how the GPU interacts with memory and the buffers for the OpenMAX components. Does the gpu essentially have it's own channel to access the GPU reserved memory areas? I'm currently allocating frame buffers with the cpu and moving things around with memcopy there, compositing a frame by blanking the buffer, then copying images and drawing text with the CPU and then pushing that buffer to video_render component (attaching the memory with OMX_UseBuffer). I'm just trying to see if there's a better way to leverage the gpu for those tasks. I toyed with leveraging OpenGL and OpenVG but ran into problems running OpenMAX components at the same time - waiting for the the asynchronous callbacks and events from OpenMAX gets disrupted somehow and I haven't quite figured out how or why that happens. I've seen similar reports from a few people years ago but didn't seem to get very far. The other thing I can't figure out is any good way to figure out how to line things up with the displays frame sync using OpenMAX.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5069
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Memory & compositing architecture questions

Mon Feb 05, 2018 11:01 am

The GPU and ARM cores share the same memory interface.

How many things are you compositing to make your full scene?
The GPU has a very powerful block called the Hardware Video Scaler (HVS) that is given a list of items to put together into the frame output to the HDMI, each with their own cropping, location, and layer information. There are limits to the amount of downscaling and layers that can be composed in realtime, but typically you'll get a couple of overlapping layers quite merrily.
The IL video_render component just adds one extra item into the list of things the HVS is told to display, along with the appropriate parameters.

You can also add items directly using the DispmanX API. Examples at https://github.com/raspberrypi/userland ... dispmanx.c and https://github.com/AndrewFromMelbourne/raspidmx amongst others. Unless you are layering vast numbers of objects, then I'd suggest you investigate DispmanX.

The other suggestion I'd make is to switch from IL to MMAL.
IL was written when the chip was intended to be used as a graphics co-processor, and all data had to be copied across from the main memory to the co-pro. MMAL is using exactly the same underlying components, but was written becasue (a) IL was an absolute swine to get right, and (b) there was a shift to applications processors (APs) where CPU and GPU were more tightly combined. MMAL has tricks that avoid a load of the copying by mapping the GPU memory allocation into the ARM virtual memory space (and then managing the caches carefully).
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

piense
Posts: 27
Joined: Thu Jun 09, 2016 4:11 am

Re: Memory & compositing architecture questions

Tue Feb 06, 2018 3:54 am

Thanks for the response. I'm really not trying to use all that many layers and the HVS sounds like what I really want to leverage. Basically just putting some text on top of a picture or video frame and dissolving between them. The main challenge I'm having is to do it in a way that allows me to do the dissolve nicely. For example in the case of two slides each with a jpeg and text on top. I'd like to just fade up the second slide on top of the first one. My approach with the cpu is to render to two frame buffers, set one as layer 0 and one as layer 1, fade up layer one, then push it back to 0 when the transition is done. If I broke that out into 4 layers I'm not sure how to dissolve the second one up without the text rendering funny on top of the jpeg during the transition. Seems like I need the HVS to render both slides into a buffer and then use those two buffer for the layers, just trying to figure out how to wrangle the APIs to do that. Looks like dispmanx might be organized in a way to do that with offscreen buffers(?), I'll have to dig through Andrew's examples and understand how that all fits together.

As frustrating as the IL is I've gotten to the point of wrangling the components fairly well and figure out how to poke at the ports and buffers to find their idiosyncrasies without too much profanity. It does have the advantage of having documentation on what the API is at least supposed to be doing. Though I still haven't figured out why components fail to start executing every so often, it's on the todo list. I've heard MMAL is supposed to be simpler but I haven't been able to find any docs on it.

piense
Posts: 27
Joined: Thu Jun 09, 2016 4:11 am

Re: Memory & compositing architecture questions

Tue Feb 06, 2018 4:56 am

actually dispmanx looks to do exactly what I need to, cool.

piense
Posts: 27
Joined: Thu Jun 09, 2016 4:11 am

Re: Memory & compositing architecture questions

Wed Feb 07, 2018 3:01 am

So I've got the basics working but one part I'm not understanding is why src_rect in vc_dispmanx_element_add is multiplied by 2^16. Is it just to be able to grab fractions of it?

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5069
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Memory & compositing architecture questions

Wed Feb 07, 2018 7:11 am

piense wrote:
Wed Feb 07, 2018 3:01 am
So I've got the basics working but one part I'm not understanding is why src_rect in vc_dispmanx_element_add is multiplied by 2^16. Is it just to be able to grab fractions of it?
Yes. It uses a fixed point representation so you can grab fractional pixels in your source rectangle.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

piense
Posts: 27
Joined: Thu Jun 09, 2016 4:11 am

Re: Memory & compositing architecture questions

Wed Feb 07, 2018 10:16 pm

Any advice on error handling with dispmanx? Seems like if a function isn't happy with something it just hangs and never returns. I'm seeing that for invalid parameters, which I'll just check for, and when the gpu is out of memory for creating resources. GPU memory a bit harder to check for in advanced. I'll certainly bump up the memory split but it'd be nice to be able to get an error instead - or I'll have to wrap it in a timeout.

Return to “OpenMAX”

Who is online

Users browsing this forum: No registered users and 1 guest