oleg11
Posts: 5
Joined: Sun Feb 23, 2020 12:03 pm

Capturing screen on RPi3 without glReadPixels

Wed Mar 25, 2020 6:51 pm

Hi, All!

I have an application which does some intensive graphics using the OpenGL ES (without X) and I need to capture screen and encode it into h264 video.

Currently I did it using MMAL, the screen data is obtained using the glReadPixels. It is highly inefficient, so here is the question - are there any way to feed screen content to MMAL directly without using glReadPixels?

Thanks!

User avatar
dividuum
Posts: 228
Joined: Sun Jun 16, 2013 1:18 pm
Location: Germany
Contact: Website

Re: Capturing screen on RPi3 without glReadPixels

Wed Mar 25, 2020 8:05 pm

oleg11 wrote:
Wed Mar 25, 2020 6:51 pm
Currently I did it using MMAL, the screen data is obtained using the glReadPixels. It is highly inefficient, so here is the question - are there any way to feed screen content to MMAL directly without using glReadPixels?
Some ideas, but I guess all are still too slow for any usable frame rate:
  • You could use dispmanx and vc_dispmanx_snapshot to capture the screen output into memory and send that to MMAL.
  • It is possible to create textures that are mappable to userspace (see https://github.com/raspberrypi/userland/pull/344 for example). Then use those as framebuffer target. Then send the userspace snapshot to MMAL. This probably won't work on the Pi4. You'll have to use dma_bufs for that, I guess.
I imagine both methods have their problems are they still use one round-trip to userspace and back and they will not be very fast. I'm curious if there's any other method.
info-beamer hosted - A user and programmer friendly digital signage platform for the Pi: https://info-beamer.com/hosted

oleg11
Posts: 5
Joined: Sun Feb 23, 2020 12:03 pm

Re: Capturing screen on RPi3 without glReadPixels

Thu Mar 26, 2020 8:33 am

Thank you for the reply and information.

I tried vc_dispmanx_snapshot before, but had no luck initially - it just hung on the second call. After some searching and trial and errors it appeared that vc_dispmanx_snapshot works ONLY with the VC_IMAGE_RGB565, VC_IMAGE_RGB888 and VC_IMAGE_RGBA32 formats.

So I was able to use it with RGBA32 and then moved to RGB565 for performance reasons. It works much better then glReadPixel (and it can be run in a separate thread, so has much less influence on the main graphics thread, especially when the sheduler type and priorities are carefully chosen). But it seems it still does some unnecessary memory moves. I have to investigate it further. Currenty it runs with 1024x600pixel RGB565 frames, and it looks like there are some dropped frames. The goal is to capture 1280x800 RGB565 frames at 30fps.

I saw another approach here https://www.raspberrypi.org/forums/view ... 8&t=222930 . I moved a bit into those direction, so I successfully created the input port with the necessary parameters and it requests the expected buffer with 24bytes size ( OMX_BRCMVEGLIMAGETYPE structure, there is not equivalent in the MMAL, at least I could not find it). The only problem is getting the necessary EGL image memory handle. I saw how that was done in those thread, but as far as I understand the OpenGL part it that thread draws on the offscreen framebuffer and then that framebuffer has to be copied to the screen FB (if you want to show it also, not just encode). It probably will compromise the performance of the main graphics thread that is not acceptable for me.

Anyway I will probably try it too a bit later.

oleg11
Posts: 5
Joined: Sun Feb 23, 2020 12:03 pm

Re: Capturing screen on RPi3 without glReadPixels

Fri Mar 27, 2020 8:59 am

I moved a bit into those direction, so I successfully created the input port with the necessary parameters and it requests the expected buffer with 24bytes size
One more step and I have a working test program that encodes directly the content of the EGL image.

So, now it is time to think and experiment how to render OpenGL into screen and EGL image simultaneously with the max. performance. I see no other options except rendering all drawing to the FBO and then rendering FBO texture to the screen. But it involves unnecessary memory copy (or even more processing) again... So, it may appear to perform not better than the current code that uses vc_dispmanx_snapshot...

milianw
Posts: 5
Joined: Tue Apr 21, 2020 1:15 pm

Re: Capturing screen on RPi3 without glReadPixels

Tue Apr 21, 2020 2:52 pm

Hey,

could you maybe share your test program? It sounds like something I may leverage myself.

Return to “OpenMAX”