JeDy111
Posts: 4
Joined: Tue Sep 03, 2019 8:35 pm

Overlay a video layer with per pixel transparency / alpha blending

Tue Sep 03, 2019 9:59 pm

Hi all, I need to overlay a video having an alpha channel on top of an other video.

The overlay will be triggered by an event while the background video is played, thus I cannot prepare the blended video output upfront.

This post mentions that the hardware can compose multiple layers with per pixel alpha, but I don't figure how to get it from video players or tools.
https://www.raspberrypi.org/forums/view ... a#p1394751

Results of my searches so far:
vlc mosaic module => seems to be only for png still images, not videos
omxplayer --alpha option => produces a flat and uniform transparency, not per pixel
ffmpeg -filter_complex overlay => I don't see how the ffmpeg filter can be launched and displayed on the fly while playing the background video

Do you have any suggestion?

JeDy111
Posts: 4
Joined: Tue Sep 03, 2019 8:35 pm

Re: Overlay a video layer with per pixel transparency / alpha blending

Wed Sep 11, 2019 7:05 am

I haven't found any solution as of now.
I share my findings and thoughts.

Pngview is a nice solution for overlaying transparent images.
https://github.com/AndrewFromMelbourne/ ... er/pngview

Does the equivalent of pngview exists for videos? or would it be possible to build it?
Transparency could be in the RGBA video alpha channel, or chroma-key.

I am wondering if this already exists or has to be developed, maybe starting from https://github.com/raspberrypi/userland ... ot/video.c and use dispmanx to draw the frames from the decoded video? The drawback is that it would not benefit from hardware capabilities.

Hardware can be PI3 or PI4.
The foreground video is small.
I am not sure if videocore version has an impact for this use case.
As 6by9 mentioned, the hardware can handle per pixel alpha compositing.

I would greatly appreciate any suggestion or advice. Thanks!

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7124
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Overlay a video layer with per pixel transparency / alpha blending

Wed Sep 11, 2019 9:49 am

I'm unaware of any video codecs that also encode an alpha channel, so where is your per-pixel alpha source coming from?

Per-pixel alpha isn't implemented on any of the YUV formats, only RGB. The alpha value is interleaved with the RGB values (there are no options for an independent alpha plane to the image data), so inserting your alpha into a video frame is going to be expensive in terms of CPU.

There are hardware options for a single mask value within some RGB formats, so a specific RGB value is treated as transparent with all other values as opaque. I did look into it a while back but failed to work out whether the detail of how that is currently configured. My recollection again is that it was only applicable to some RGB formats though, not YUV.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

JeDy111
Posts: 4
Joined: Tue Sep 03, 2019 8:35 pm

Re: Overlay a video layer with per pixel transparency / alpha blending

Wed Sep 11, 2019 1:23 pm

Thanks 6by9 for your reply.

The per-pixel alpha source could be a webm VP8 encoded video file like those used in https://simpl.info/videoalpha/. By the way, for some reason, this URL freezes Chromium and FirefoxESR on my PI4.
The per-pixel alpha source could also one listed on Video Codecs and Image Sequences with Alpha Channels

I understand the single mask value within RGB format would be like chroma-key. For instance replace green color with transparency. It could be a solution, but not as good as a 8 bits value for the opacity level, which allows for smooth edges and transitions.
Do you know a software, library, or piece of code which can do that?

By the way, I have found this OpenMAX blending of video and alpha mask discussion which ends up with:
Saw in VideoCore IV docs that there is a Texture Datatype of RGBA32R, "Raster format 8-bit per channel red, green, blue, alpha", which is exactly what I need in order to avoid tiling/swizzling huge overhead. But as far as I can see, this is not exposed by the firmware. Why? Could this be added?
I don't know if that is something to investigate.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7124
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Overlay a video layer with per pixel transparency / alpha blending

Wed Sep 11, 2019 4:05 pm

JeDy111 wrote:
Wed Sep 11, 2019 1:23 pm
The per-pixel alpha source could be a webm VP8 encoded video file like those used in https://simpl.info/videoalpha/. By the way, for some reason, this URL freezes Chromium and FirefoxESR on my PI4.
Hmm, so a thing that Google crowbarred into the existing VP8 spec and hoped didn't break existing players.
JeDy111 wrote:The per-pixel alpha source could also one listed on Video Codecs and Image Sequences with Alpha Channels
I'd summarise those as weird formats, or what I'd consider image formats (PNG, Targa, and TIFF).
None of the video formats have any form of hardware accelerated decode on the Pi (or probably any SBC).
JeDy111 wrote:I understand the single mask value within RGB format would be like chroma-key. For instance replace green color with transparency. It could be a solution, but not as good as a 8 bits value for the opacity level, which allows for smooth edges and transitions.
Do you know a software, library, or piece of code which can do that?
Your only real hope is to get something like FFMpeg to decode the YUVA and format convert to RGBA.

Downloading their example VP8a file from https://cs.chromium.org/chromium/src/me ... -vp8a.webm appears not to render with alpha using ffplay on an x86 box, but that may be because the default (only?) decode format is yuv420p (no alpha).
Wiki claims that ffmpeg can do YUV4:2:0 with alpha

Code: Select all

ffmpeg -i bear-vp8a.webm -pix_fmt yuva420p bear.yuv
seems to dump out something in between the YUV frames, but I haven't analysed it. Likewise requesting -pix_fmt rgba gives something that my normal raw image viewer (Vooya) seems to accept as RGBA. The RGBA should be easy enough to pass to DRM, MMAL video_render, or DispmanX.
JeDy111 wrote:By the way, I have found this OpenMAX blending of video and alpha mask discussion which ends up with:
Saw in VideoCore IV docs that there is a Texture Datatype of RGBA32R, "Raster format 8-bit per channel red, green, blue, alpha", which is exactly what I need in order to avoid tiling/swizzling huge overhead. But as far as I can see, this is not exposed by the firmware. Why? Could this be added?
I don't know if that is something to investigate.
Those are only potentially applicable if you are rendering through GL/GLES, not using the HVS to add an overlay.
I've just checked the datasheet for the HVS - there are no YUVA formats supported.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

JeDy111
Posts: 4
Joined: Tue Sep 03, 2019 8:35 pm

Re: Overlay a video layer with per pixel transparency / alpha blending

Wed Sep 11, 2019 9:35 pm

Thanks a lot for your detailed answer. I very much appreciate your help.

I can use yuva420p instead of webm VP8.

I am new to DRM, MMAL video_render, and I discovered DispmanX recently while working on this issue.
Is DispmanX the easiest way to go among the three?

Do you have any guidance on how to pass the RGBA to DispmanX?

I have found these lines of code: AVFrame YUV420P to dispmanx
Do you think it is a good starting point?

Return to “Graphics, sound and multimedia”