ksharindam
Posts: 41
Joined: Sat Jan 09, 2016 4:16 pm

HW acclerated playback (rendering) on xwindows like chromium

Tue Oct 17, 2017 4:49 am

As far as I know, OpenGL ES can not be integrated with xwindows. So every app that uses hw accelerated playback, overlays on screen. But I am quite surprised Chromium browser is able to play video with hw accleration without overlaying.
Can anyone explain how is it possible?
Last edited by ksharindam on Tue Oct 17, 2017 7:04 am, edited 1 time in total.

gkreidl
Posts: 5326
Joined: Thu Jan 26, 2012 1:07 pm
Location: Germany

Re: HW acclerated playback xwindows like chromium

Tue Oct 17, 2017 5:54 am

ksharindam wrote:
Tue Oct 17, 2017 4:49 am
As far as I know, OpenGL ES can not be integrated with xwindows. So every app that uses hw accelerated playback, overlays on screen. But I am quite surprised Chromium browser is able to play video with hw accleration without overlaying.
Can anyone explain how is it possible?
OpenGL (including ES) is for 3D (and some 2D) graphics hardware acceleration and has nothing to do with video decoding/encoding.
There is an alpha OpenGL driver, but it grabs tho whole GPU for itself and keeps all other GPU related stuff from working, including video decoding (omxplayer), for example.
Minimal Kiosk Browser (kweb)
Slim, fast webkit browser with support for audio+video+playlists+youtube+pdf+download
Optional fullscreen kiosk mode and command interface for embedded applications
Includes omxplayerGUI, an X front end for omxplayer

ksharindam
Posts: 41
Joined: Sat Jan 09, 2016 4:16 pm

Re: HW acclerated playback xwindows like chromium

Tue Oct 17, 2017 7:03 am

I am not saying about hw accelerated decoding, I am saying about hwacclerated rendering.
Raspberry Pi2 processor is strong enough to decode 720p video but it is not strong enough to render video on fullscreen. You can not play smoothly a hw decoded video on x11 output. I have confirmed this by playing a video with mpv using ffmpeg compiled for raspberry pi (that supports hw accelerated decoding). But mpv can not play that video smoothly on fullscreen using x11 output (thouth decoding was hw acclerated).
But chromium can play video smoothly.

gkreidl
Posts: 5326
Joined: Thu Jan 26, 2012 1:07 pm
Location: Germany

Re: HW acclerated playback xwindows like chromium

Tue Oct 17, 2017 7:22 am

ksharindam wrote:
Tue Oct 17, 2017 7:03 am
I am not saying about hw accelerated decoding, I am saying about hwacclerated rendering.
Raspberry Pi2 processor is strong enough to decode 720p video but it is not strong enough to render video on fullscreen. You can not play smoothly a hw decoded video on x11 output. I have confirmed this by playing a video with mpv using ffmpeg compiled for raspberry pi (that supports hw accelerated decoding). But mpv can not play that video smoothly on fullscreen using x11 output (thouth decoding was hw acclerated).
But chromium can play video smoothly.
There is no "hwacclerated rendering" (except by using OpenGL, perhaps). The video is decoded on the GPU (if HW decoding is used) and the result has to be copied back into the frame buffer in real time, which includes a costly colour space conversion (some kind of YUV to RGB(A)).
How efficient this process is depends on the software components being used.
The method used by the specially patched chromium version works best so far, but cannot play video on a full HD 1920x1080 screen without loss of frames. It may look fluent, but as soon as the video includes camera movements you will see, that quite a lot of frames are missing. And adding a little bit of "stress", e. g. by moving the mouse around, will result in a very jerky playback.
Minimal Kiosk Browser (kweb)
Slim, fast webkit browser with support for audio+video+playlists+youtube+pdf+download
Optional fullscreen kiosk mode and command interface for embedded applications
Includes omxplayerGUI, an X front end for omxplayer

ksharindam
Posts: 41
Joined: Sat Jan 09, 2016 4:16 pm

Re: HW acclerated playback xwindows like chromium

Tue Oct 17, 2017 7:33 am

which includes a costly colour space conversion (some kind of YUV to RGB(A)).
How efficient this process is depends on the software components being used.
The method used by the specially patched chromium version works best so far.
Thats what I wanted to know.
So, you are saying that chromium does this color conversion faster.
One more thing that the frames are scaled also, which is also costly process. mpv runs smooth when in 1x scale, bt when scale is changed, it becomes painfully slow.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 4604
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: HW acclerated playback (rendering) on xwindows like chromium

Tue Oct 17, 2017 9:33 am

Chromium is using video_decode on the GPU, followed by the vc.ril.isp component to resize so the composition is just a memcpy, and format convert to RGBA.
It tends to be adding any overlays that take it over the performance limit, and it's tending to be either task scheduling or SDRAM bandwidth that are the bottlenecks. 1920x1080 RGBA frames are about 8MB a shot, and that hurts.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

ksharindam
Posts: 41
Joined: Sat Jan 09, 2016 4:16 pm

Re: HW acclerated playback (rendering) on xwindows like chromium

Tue Oct 17, 2017 9:54 am

Thanks 6by9, it is now clear. Your answer is the most perfect.

pagenotfound
Posts: 5
Joined: Mon Mar 14, 2016 12:44 pm

Re: HW acclerated playback (rendering) on xwindows like chromium

Fri Oct 20, 2017 2:21 pm

Recently, Eric Anholt wrote on his blog:
"Dave Stevenson has VCSM successfully importing dma-bufs. This is a major step for displaying or GL rendering from of the media decode engine’s buffers."

Does that have anything to do with video rendering?
More generally, is there any hope that the whole situation will improve and VLC or MPlayer will be able to use the GPU just like they do on other systems? Full screen, with their own transport controls on top. Or Firefox use the GPU for page rendering?

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 4604
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: HW acclerated playback (rendering) on xwindows like chromium

Fri Oct 20, 2017 2:43 pm

pagenotfound wrote:
Fri Oct 20, 2017 2:21 pm
Recently, Eric Anholt wrote on his blog:
"Dave Stevenson has VCSM successfully importing dma-bufs. This is a major step for displaying or GL rendering from of the media decode engine’s buffers."

Does that have anything to do with video rendering?
More generally, is there any hope that the whole situation will improve and VLC or MPlayer will be able to use the GPU just like they do on other systems? Full screen, with their own transport controls on top. Or Firefox use the GPU for page rendering?
Sort of on MPlayer and VLC. Firefox page rendering sounds like only composition to me which has nothing to do with it.
I've left the ball in Eric's court really, although there is work half scheduled to investigate Chromimum again (software rendering and EGL rendering are two entirely different code paths).

https://github.com/6by9/drm_mmal is my simple demo app passing an H264 ES into video_decode, format converting using the ISP to write into dmabufs allocated by vc4, and then using libdrm to render those dmabufs.
I have very limited knowledge of how the rendering side works, and it seems that (lib)DRM only allows a single client at a time. If you have X running then that is your client, so it only works from the command line.
I have got a couple of trivial patches to vc4_plane.c to support RGB888, BGR888, NV21 and NV61, but those aren't essential.

If there is anyone else out there who wants to experiment then feel free.
At the moment I'm doing further hacking around on vc-sm to do dma-buf export (among other things), so don't expect direct support on this.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

pagenotfound
Posts: 5
Joined: Mon Mar 14, 2016 12:44 pm

Re: HW acclerated playback (rendering) on xwindows like chromium

Fri Oct 20, 2017 4:47 pm

I'm afraid I'm not really qualified for this type of work (it's been 11 years since I last touched any C code). But thanks for answering. Maybe someone else will read it and get into action..

Return to “Graphics programming”

Who is online

Users browsing this forum: No registered users and 3 guests