We have all seen the excellent demo where a video is rendered on a teapot by passing an EGLImage as an output buffer to the egl_render component. I thought that using an EGLImage as an input to the video_encode component would be as simple as that. I was wrong. UseEGLImage() is not implemented in video_encode.
So here's the question: how much work would it be for the Foundation to implement passing EGLImage to video_encode? I would think that it would be less than 20 lines of code. However, somebody created an entire new component egl_render, instead of implementing UseEGLImage() in video_render, which makes me believe that the matter might not be as trivial as I thought.
Just one example of how using EGLImage would help a big part of the community, is colorspace conversion. Right now, if video_encode does not support your camera's colorspace (which it usually doesn't), colorspace conversion will severely slow down your framerates. Even if someone created a component that uses GPU for colorspace conversion, its output could still not be fed directly to video_encode's input without passing through CPU. I see lots of people complaining about lack of good colorspace conversion not only on Raspberry Pi, but on other ARM boards as well.
For a more detailed example, consider my use case. I have an USB camera that has no encoder on board. From the camera, I get a 1280x960 image in Bayer pattern. I then use Morgan McGuire's shaders from http://graphics.cs.williams.edu/papers/BayerJGT09/
to demosaic the image into a 1280x960 RGBA image. I have adapted these shaders to OpenGL ES, so they work on Raspberry Pi. Then I download
the image to CPU and immediately re-upload
to the GPU for video_encode, using glReadPixels and OMX_EmptyThisBuffer, respectively. This useless last step takes around 92 ms, which reduces the overall framerate to 10 FPS. And it's not even Full HD.
Shaders are a very powerful concept. Instead of colorspace conversion or demosaicing, people could add all kinds of special effects to videos before encoding and saving to disk. Combining different filters, various real time machine vision tasks could be implemented on the Pi etc. For this reason, I would ask that all OpenMAX graphics components should have EGLImage input and output. For selfish reasons
, I suggest that video_encode should be the next one getting this feature (after the already implemented egl_render, which is wonderful).