Doub
Posts: 24
Joined: Fri Apr 07, 2017 9:48 pm

Screen capture and video encoding

Wed Jul 17, 2019 4:56 am

I've written a C application which primary feature is to display the camera input full screen, with some UI overlay, with minimal lag. I've used MMAL to capture the camera, and OpenVG to draw the overlay.

Now I'd like to add as a secondary feature (as in, it shouldn't compromise the primary feature) recording of the whole screen to a video file for archiving purposes. The video should ideally record exactly what was shown (minus compression artifacts).

Is there any way I could feed the whole screen to some video encoder which ideally uses the hardware encoder? For example can I write some MMAL component that would capture the screen and that I'd connect to the MMAL H.264 encoder?

jdonald
Posts: 391
Joined: Fri Nov 03, 2017 4:36 pm

Re: Screen capture and video encoding

Tue Jul 30, 2019 6:17 pm

Someone recently asked the related question of casting RetroPie's direct rendering in this thread in the Gaming section.

You can start with VNC direct capture mode, which can obtain legacy OpenVG or dispmanx overlay pixels directly from the framebuffer. Go to VNC Server tray icon -> Options -> Troubleshooting -> Enable direct capture mode. You probably also want to go to Options -> Security -> Authentication: VNC password (instead of UNIX password) for maximum portability.

Then take code for some existing VNC->video streaming tool and modify it to do the MMAL encode. The main compromise would be the overhead of frame copies could slow down your primary application. Also, RealVNC uses compression so you might have a second layer of artifacts. To improve that you could try some of the alternatives to VNC I listed in the other thread, although those are generally more costly or time-consuming.

Doub
Posts: 24
Joined: Fri Apr 07, 2017 9:48 pm

Re: Screen capture and video encoding

Tue Jul 30, 2019 8:31 pm

jdonald wrote:
Tue Jul 30, 2019 6:17 pm
You can start with VNC direct capture mode, which can obtain legacy OpenVG or dispmanx overlay pixels directly from the framebuffer.
This sounds quite inefficient to run a VNC server and a VNC client just to capture the framebuffer. Do you happen to know how that direct capture mode is implemented? Otherwise is the VNC server you're suggesting open source?

jdonald
Posts: 391
Joined: Fri Nov 03, 2017 4:36 pm

Re: Screen capture and video encoding

Wed Jul 31, 2019 1:52 am

You understood it perfectly.

While RealVNC does maintain some open-source offerings, RealVNC Server is proprietary. For some sample code, I dug up the thread with an implementation that predates RealVNC's:
- Dispmanx VNC Server

Those posts contain a link to the dispmanx_vnc project on GitHub which you can use as a reference. When searching for the topic I came across a few other projects which basically operate using the same vc_dispmanx_snapshot() mechanism: rpi-fbcp, raspi2fb, raspi2png.

I also noticed posts from late 2016 when UV4L added uv4l-raspidisp which apparently is able to get the screen capture from HDMI out.
Last edited by jdonald on Thu Aug 01, 2019 12:45 pm, edited 1 time in total.

Doub
Posts: 24
Joined: Fri Apr 07, 2017 9:48 pm

Re: Screen capture and video encoding

Wed Jul 31, 2019 1:28 pm

Thanks for the pointers. It looks like it might be possible through the dispmanx API. I need to spend some time with it (and MMAL) to see if I can manage to get an uncompressed video stream efficiently and encode it. I have not used MMAL extensively for encoding, my main focus has been getting the camera feed so far, so it's still not entirely clear to me how the memory is architectured and what's relationship between the CPU and the hardware encoder.

User avatar
dividuum
Posts: 165
Joined: Sun Jun 16, 2013 1:18 pm
Location: Germany
Contact: Website

Re: Screen capture and video encoding

Thu Aug 01, 2019 10:53 am

Doub wrote:
Wed Jul 31, 2019 1:28 pm
Thanks for the pointers. It looks like it might be possible through the dispmanx API.
At least from my understanding (of up to the Pi3, no idea about what the Pi4 might do at some point), there's no way to efficiently get any kind of realtime screen capturing. The vc_dispmanx_snapshot function results in 3-5 snapshots per second max and at that point you still only have the raw screen pixels and not a (for example) H264 encoded stream. I don't think there's any way to setup up a hardware accelerated pipeline for any of that. Happy to be corrected..
info-beamer hosted - A user and programmer friendly digital signage platform for the Pi: https://info-beamer.com/hosted

Return to “Graphics programming”