Page 1 of 1

Screen capture and video encoding

Posted: Wed Jul 17, 2019 4:56 am
by Doub
I've written a C application which primary feature is to display the camera input full screen, with some UI overlay, with minimal lag. I've used MMAL to capture the camera, and OpenVG to draw the overlay.

Now I'd like to add as a secondary feature (as in, it shouldn't compromise the primary feature) recording of the whole screen to a video file for archiving purposes. The video should ideally record exactly what was shown (minus compression artifacts).

Is there any way I could feed the whole screen to some video encoder which ideally uses the hardware encoder? For example can I write some MMAL component that would capture the screen and that I'd connect to the MMAL H.264 encoder?

Re: Screen capture and video encoding

Posted: Tue Jul 30, 2019 6:17 pm
by jdonald
Someone recently asked the related question of casting RetroPie's direct rendering in this thread in the Gaming section.

You can start with VNC direct capture mode, which can obtain legacy OpenVG or dispmanx overlay pixels directly from the framebuffer. Go to VNC Server tray icon -> Options -> Troubleshooting -> Enable direct capture mode. You probably also want to go to Options -> Security -> Authentication: VNC password (instead of UNIX password) for maximum portability.

Then take code for some existing VNC->video streaming tool and modify it to do the MMAL encode. The main compromise would be the overhead of frame copies could slow down your primary application. Also, RealVNC uses compression so you might have a second layer of artifacts. To improve that you could try some of the alternatives to VNC I listed in the other thread, although those are generally more costly or time-consuming.

Re: Screen capture and video encoding

Posted: Tue Jul 30, 2019 8:31 pm
by Doub
jdonald wrote:
Tue Jul 30, 2019 6:17 pm
You can start with VNC direct capture mode, which can obtain legacy OpenVG or dispmanx overlay pixels directly from the framebuffer.
This sounds quite inefficient to run a VNC server and a VNC client just to capture the framebuffer. Do you happen to know how that direct capture mode is implemented? Otherwise is the VNC server you're suggesting open source?

Re: Screen capture and video encoding

Posted: Wed Jul 31, 2019 1:52 am
by jdonald
You understood it perfectly.

While RealVNC does maintain some open-source offerings, RealVNC Server is proprietary. For some sample code, I dug up the thread with an implementation that predates RealVNC's:
- Dispmanx VNC Server

Those posts contain a link to the dispmanx_vnc project on GitHub which you can use as a reference. When searching for the topic I came across a few other projects which basically operate using the same vc_dispmanx_snapshot() mechanism: rpi-fbcp, raspi2fb, raspi2png.

I also noticed posts from late 2016 when UV4L added uv4l-raspidisp which apparently is able to get the screen capture from HDMI out.

Re: Screen capture and video encoding

Posted: Wed Jul 31, 2019 1:28 pm
by Doub
Thanks for the pointers. It looks like it might be possible through the dispmanx API. I need to spend some time with it (and MMAL) to see if I can manage to get an uncompressed video stream efficiently and encode it. I have not used MMAL extensively for encoding, my main focus has been getting the camera feed so far, so it's still not entirely clear to me how the memory is architectured and what's relationship between the CPU and the hardware encoder.

Re: Screen capture and video encoding

Posted: Thu Aug 01, 2019 10:53 am
by dividuum
Doub wrote:
Wed Jul 31, 2019 1:28 pm
Thanks for the pointers. It looks like it might be possible through the dispmanx API.
At least from my understanding (of up to the Pi3, no idea about what the Pi4 might do at some point), there's no way to efficiently get any kind of realtime screen capturing. The vc_dispmanx_snapshot function results in 3-5 snapshots per second max and at that point you still only have the raw screen pixels and not a (for example) H264 encoded stream. I don't think there's any way to setup up a hardware accelerated pipeline for any of that. Happy to be corrected..