VLC 3.0 with hardware acceleration

[gkreidl]

VLC has always been my favourite media player on all kinds of operating systems (Windows, Linux, even Android, which I don't use any more). It has always been on Raspbian, but without hardware accleration it was useless for video. 5 years ago I started to compile my own versions with hardware accleration enabled and I have posted two tutorials about it here on the forum. The latest Raspbian image now contains a new version 3.0.3 of VLC with hardware accleration for video and of course it can also be installed on older Stretch systems with apt etc.

First I want to thank the Foundation's developers for creating this great new version. It contains two new output modules, "MMAL" to play videos in a screen overlay, and "MMAL X11 splitter" to play HW accelerated video inside the video window, which is really what people have been looking for. I have spent hours testing it and it is really working well.

I suggest, that we use the this thread for user experiences and bug reports (and I will start with my first report in the following post)

[gkreidl]

While video works great, there is a real problem with audio files. They take a long time to start (even minutes). VLC is doing something in the background which seems to take a long time. I fiddled around with a lot of settings but could not find anything which helps.
Edit: fixed! This only happened with files on samba shares, not with local files. Solution: add "vers=1.0" to all samba mount commands.

When using MMAL (overlay) output, the video size is not stretched to fill the screen. A 720p50 video will be displayed in the upper left side using the original size. And it looks even worse when you play SD video. This may not be called a "bug", but it really should be changed.

Some people have reported crashes in another thread, but I cannot confirm this at the moment.

[gkreidl]

I have tested a lot today and it really works well for video. But there is a problem with audio files: they need a lot of time to start (tested mp3 and flac)

Is this a general problem with vlc on pi, or is this an issue purely in the new accelerated version included in raspbian?

The self-compiled versions of VLC have all been using VLC 2.2.x versions and the audio problems did not exist in these versions. It may be caused by something new in VLC 3.0 or by some missing dependencies.

@dom thank you for paying attention to VLC. Would it be possible to include in the repository version also the "OpenMAX IL video output" module that @gkreidl has been kindly working on for years now? That option has proved to work well for years and it should be kept available.

Full screen performance of the "OpenMAX IL video output" available in this thread is better, and it also allows software decoding of MPEG2 using the VLC software routines and relying on hardware acceleration for rendering. The repository version completely hampers that software decoding.

@gkreidl Thank you for all your work in this area. Please do not abandon your version until the "official" version in the repositories provides full funtionallity.

Martin

"OpenMAX IL video output" was never completely stable and was completely broken in VLC 3.0.

I'm not sure that it could be combined with software decoding of MPEG2, but I cannot really prove it, as I bought the Codecs for all my RPis.

There's one problem with the MMAL overlay output as described above: it does not scale up the video to use the full screen area.

[Gadgetguy]

While video works great, there is a real problem with audio files. They take a long time to start (even minutes). VLC is doing something in the background which seems to take a long time. I fiddled around with a lot of settings but could not find anything which helps.

When using MMAL (overlay) output, the video size is not stretched to fill the screen. A 720p50 video will be displayed in the upper left side using the original size. And it looks even worse when you play SD video. This may not be called a "bug", but it really should be changed.

Some people have reported crashes in another thread, but I cannot confirm this at the moment.

Trying out the new vlc I noticed some audio /video sync problems for example on this recent youtube video:


https://www.youtube.com/watch?v=LM9862GCl5o

While pressing the back arrow navigation key seemed to put things back in sync i was looking for a more satisfactory solution; using the menu/tools/preferences/audio/ choose alsa audio output and then for audio device choose BCM2835 Alsa Direct Hardware Device without any conversions. This seems to resolve a/v sync problems, and moreover with this settings my mp3 files start playing quickly.

[gkreidl]

While video works great, there is a real problem with audio files. They take a long time to start (even minutes). VLC is doing something in the background which seems to take a long time. I fiddled around with a lot of settings but could not find anything which helps.

When using MMAL (overlay) output, the video size is not stretched to fill the screen. A 720p50 video will be displayed in the upper left side using the original size. And it looks even worse when you play SD video. This may not be called a "bug", but it really should be changed.

Some people have reported crashes in another thread, but I cannot confirm this at the moment.

Trying out the new vlc I noticed some audio /video sync problems for example on this recent youtube video:


https://www.youtube.com/watch?v=LM9862GCl5o

While pressing the back arrow navigation key seemed to put things back in sync i was looking for a more satisfactory solution; using the menu/tools/preferences/audio/ choose alsa audio output and then for audio device choose BCM2835 Alsa Direct Hardware Device without any conversions. This seems to resolve a/v sync problems, and moreover with this settings my mp3 files start playing quickly.

Unfortunately this did not solve my audio problems. A few times a file was started immediately, but usually not and it never worked for m3u playlists. The delay (about 10 to 60 seconds) is between the following lines in the messages window:

Code: Select all

main debug: looking for meta reader module matching "any": 2 candidates
main debug: using meta reader module "taglib"

[Gadgetguy]

The new vlc is a wonderful addition to the repository and seems to excel at many things- the ability to play high def h264 in an infinitely resalable window is particularly nice. I notice also that vlc plays without a hiccup a high def 1080p youtube live stream eg. France24 ( thats still high def to these eyes!) via G. Kreidl's method described at:

( viewtopic.php?f=66&t=40860&start=1450#p1392789)

Vlc is also the default player for livestreamer and its' more up to date fork -streamlink and from which it also plays such livestreams without problem.

But at the moment at least it doesn't play all file formats as well as mpv-mmal (or smplayer/mpv ) (which uses overlay and unfortunately doesn't have an infinitely re-sizeable dragable window. For example mpv will play high def mpeg2 without a codec license albeit at higher cpu resouce usage.

I happened to have a video file I downloaded from youtube for which vlc will only play the audio portion without the video:



https://www.youtube.com/watch?v=kG6QW8Cu7oo


The file format that I downloaded has the following specifications as shown by Mediainfo app:

General
Complete name : /media/pi/586C3CA36C3C7E36/other vids/Bobby Darin Sandra Dee Once Upon A Time.mp4
Format : MPEG-4
Format profile : Base Media
Codec ID : isom (isom/iso2/avc1/mp41)
File size : 9.57 MiB
Duration : 3 min 48 s
Overall bit rate mode : Variable
Overall bit rate : 351 kb/s
Encoded date : UTC 1970-01-01 00:00:00
Tagged date : UTC 1970-01-01 00:00:00
Writing application : Lavf52.32.0

Video
ID : 1
Format : AVC
Format/Info : Advanced Video Codec
Format profile : [email protected]
Format settings, CABAC : Yes
Format settings, ReFrames : 1 frame
Codec ID : avc1
Codec ID/Info : Advanced Video Coding
Duration : 3 min 48 s
Bit rate : 271 kb/s
Width : 320 pixels
Height : 240 pixels
Display aspect ratio : 4:3
Frame rate mode : Constant
Frame rate : 29.970 (30000/1001) FPS
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Progressive
Bits/(Pixel*Frame) : 0.118
Stream size : 7.37 MiB (77%)
Encoded date : UTC 1970-01-01 00:00:00
Tagged date : UTC 1970-01-01 00:00:00

Audio
ID : 2
Format : AAC
Format/Info : Advanced Audio Codec
Format profile : LC
Codec ID : 40
Duration : 3 min 48 s
Bit rate mode : Variable
Bit rate : 75.5 kb/s
Channel(s) : 2 channels
Channel(s)_Original : 1 channel
Channel positions : Front: C
Sampling rate : 48.0 kHz
Frame rate : 46.875 FPS (1024 spf)
Compression mode : Lossy
Stream size : 2.06 MiB (21%)
Encoded date : UTC 1970-01-01 00:00:00
Tagged date : UTC 1970-01-01 00:00:00



The strange and annoying thing is when I stop the file playing, I cannot play another file in vlc and the only way I can quit vlc and start it again is by killing it in task manager.


In Vlc or any media player if one is playing a video in full screen It is nice to have on screen display (osd) functionality for example how many minutes of the video have elapsed and the playing length of the video in question (just like with omxplayer).
For better and for worse vlc has such a bewildering array of options it is sometimes hard to fin the option you need ,buried deep within sub menus and with cryptic sounding names. Doing a little research I found one way to enable an on demand osd rendering of the aforementioned time parameters was to enable vlc's " marquee display "
This can be accomplished by selecting tools/ preferences in vlc's menu and at the bottom select show all settings.
Then in the search box that appears at the top type osd .In the results you may if you wish go into the video section and check show title of video and select the position and duration in milliseconds you wish it to appear in osd. Next in the search results enter the osd menu and enable osd then further down in the overlays subsection
select marquee display. Then go back up to the search box and type font. In the results enter marquee and in the text box erase vlc unless you want vlc appearing in all your videos. Next in the font subsection of the marquee menu pick the pixel size you want for your osd display, too small and it may be invisible.
( I picked 25 for my 32 inch 720p tv monitor.) You may also wish to pick your font sise in the text renderer menu that appears in the font search results.

There is also a custom vlc plugin named time that purports to show time elapsed etc. I personally did not find it satisfactory. It did not appear to me to be configurable.For that matter I did not find the osd renderings of time activated by the marquee
display option of vlc to be particularly satisfying either because the time information only flashed on screen for less than a second..(The osd time display is activated by presing the arrow navigation key)
Perhaps someone can advise a better method of activating osd time display.

[Gadgetguy]

G.Kreidl wrote:

"Unfortunately this did not solve my audio problems. A few times a file was started immediately, but usually not and it never worked for m3u playlists. The delay (about 10 to 60 seconds) is between the following lines in the messages window:
Code: Select all

main debug: looking for meta reader module matching "any": 2 candidates
main debug: using meta reader module "taglib""


Just a thought in the dark but perhaps if you have embedded data such as a jpg or for example some of the metadata you can embed in an mp3 file via MS windows perhaps this compiled vlc has difficulty interpreting it?

[gkreidl]

The audio delay problem could be solved!

It only appeared, when I accessed audio files from a locally mounted samba share (all my audio files are on this share!). It did not happen with local audio files.

Solution: added "vers=1.0" to the mount commands of all my samba shares.

[6by9]

We probably ought to publicise it more, but the new option is "MMAL X11 splitter".

"MMAL X11 splitter" should support all the bells and whistles that you expect (overlays/subtitles, resizeable/draggable windows, etc, etc).
There's little point in reporting any issues with the "MMAL-based vout plugin for Raspberry Pi" output pipe, unless you are reporting them upstream to the VLC developers. (Perhaps we should even go the stage further to remove that option).

For info, "MMAL X11 splitter" is using video_decode to produce the raw frames, passing it to the new HVS component to do the overlays, resize to the exact window size, and format convert to RGBA. Those frames are then passed to the ARM to blit into frame buffer.
If you go full screen then the HVS is dropped out of the pipeline and the video frames and overlays are sent to independent video_render components to create the final screen. The one thing that you do lose at that point is the mouse cursor and the control bar overlay as those are still rendered to the framebuffer that is totally obscured.

Using "MMAL X11 splitter" with the KMS or F-KMS drivers (aka GL drivers) will cause some odd behaviours.
KMS will perform badly as the plugin will resize to the correct size and RGBA (has to use the ISP as the HVS is under the control of the kms driver, so no overlays), but then passes it to GL to render which is likely to struggle.
F-KMS can use the HVS, but still passes the frame to GL to render.
There are plans coming to fruition that should improve performance here, but not yet. The V4L2 M2M (memory to memory) is the way most boards are going for codecs, and our implementation is almost merged. That gives a load of zero copy paths around X11, although it still uses GL for composition. It also provides support for the GStreamer v4l2[h264|mpeg2|mpeg4|h263]dec and v4l2h264enc components, and supporting passing dmabufs around. Doing so should give a moderate performance gain over the older omx[h264|mpeg4|h263|mpeg2]dec and omxh264enc components.

[6by9]

@GadgetGuy, I suspect the fallback path is failing for MPEG2 should the codec licence not be purchased. Error paths are always a nuisance to handle and test them all.
I don't think you'd manage full software decode of MPEG2 HD with resize and overlays - that's a pretty heavy load for the CPUs. There isn't an option within VLC to do an offload of the resize and conversion whilst using a software codec, so it's an all or nothing. mpv-mmal is offloading the resize and display to the hardware, hence the CPUs are near 100% available for the MPEG2 decode.

[MartinLaclaustra]

@GadgetGuy, I suspect the fallback path is failing for MPEG2 should the codec licence not be purchased. Error paths are always a nuisance to handle and test them all.
I don't think you'd manage full software decode of MPEG2 HD with resize and overlays - that's a pretty heavy load for the CPUs. There isn't an option within VLC to do an offload of the resize and conversion whilst using a software codec, so it's an all or nothing. mpv-mmal is offloading the resize and display to the hardware, hence the CPUs are near 100% available for the MPEG2 decode.

For the sake of comparison, with former gkreidl's version (rendering fullscreen through "OpenMAX IL video output"), VLC can reproduce a DVD folder structure and its main title without stutter (no GPU codec, software decode?), and CPU load stays < 10%.

I need to re-test with the new repositories version, will report...

[6by9]

For the sake of comparison, with former gkreidl's version (rendering fullscreen through "OpenMAX IL video output"), VLC can reproduce a DVD folder structure and its main title without stutter (no GPU codec, software decode?), and CPU load stays < 10%.

I need to re-test with the new repositories version, will report...

A DVD will be standard definition (ie 576i/720x576 PAL or thereabouts). GadgetGuy had referenced HD (High Definition - 1280x720, or 1920x1080). 5 times as many pixels per frame!

I haven't tested it, but I'd be surprised if MPEG2 SD decode was <10% for running a software codec. Deinterlacing would typically take up more processing power than that!

[pagenotfound]

Nice.Thanks for making this happen.

I still wish that fullscreen mode with mouse triggered popup transport bar was possible. I mean, if the VC4 architecture doesn't allow it, what was Broadcom thinking?

What's more, after experimenting with fullscreen mode I noticed a tendency of all windows, even those of other programs, to become effectively undecorated. By which I mean, right clicking title bars does nothing. Left clicking on minimize, close etc. in the upper right corner does nothing anymore. Minimizing or closing via the panel still works. I have to reboot to get everything working again.

[6by9]

Nice.Thanks for making this happen.

I still wish that fullscreen mode with mouse triggered popup transport bar was possible. I mean, if the VC4 architecture doesn't allow it, what was Broadcom thinking?

No, X11 is such a pile of rubbish that it can't make use of hardware overlays. It forces VLC to have two options:
- waste a HUGE amount of processing passing everything through GL to do a simple composition.
- make VLC resize the image (expensive on the CPU) and blit it into the frame buffer (even the blit will stress a Pi0 or 1).
The non-fullscreen option is blitting into framebuffer, having used the hardware blocks to do the resize and overlays to reduce the load on the ARMs.
The full-screen option passes the decoded streams straight to the hardware to replace the frame buffer. The transport bar is still rendered to the framebuffer as there is no way to tell X to float it on top of another layer. Blitting 1080p60 is almost always going to fail as it sucks up a huge amount of memory bandwidth.

The hardware itself can quite happily compose multiple layers (with per pixel or per plane alpha) with resizing on each source image, transpose if you want it, format convert, and drive multiple displays. Even if you switch to the mainline GL/DRM/KMS driver then X uses GL to do a simple composition as it doesn't have a way to use the hardware.

What's more, after experimenting with fullscreen mode I noticed a tendency of all windows, even those of other programs, to become effectively undecorated. By which I mean, right clicking title bars does nothing. Left clicking on minimize, close etc. in the upper right corner does nothing anymore. Minimizing or closing via the panel still works. I have to reboot to get everything working again.

This is on a standard Raspbian install? Which Pi variant? Using the "MMAL X11 splitter for Raspberry Pi" under Preferences/Video/Output? Any tweaks to gpu_mem? Lots of things running in parallel?
I am seeing a few oddities around how it decides what size the screen is and therefore where it is putting subtitles and the like, but I have yet to see any issues on other windows. My systems get pretty hacked around though, so that may be something in my setup.
It has been tested fairly thoroughly, but we need to have a way to reproduce any failures.

[gkreidl]

Running a number of tests confirms everything 6by9 has explained above:

1)Playing a 1080p60 video in window mode I'm getting a huge number of lost frames. If I switch to full screen no frames are lost any more. Even with a 720p50 TV live stream I'm getting a certain amount of lost frames in window mode, while it works perfect in full screen mode.

2) For this test I've been using a 1080i50 live TV stream and I'm running top from a ssh connection:

Full screen mode: idle at about 90-11%, which is almost as good as omxplayer (93-95%)
Window mode: idle between 70 and 75%. Both VLC and XORG needing a lot more processing power.

Full screen mode is far more effective than window mode. There might be a solution though how to access the control elements during full screen video playback. I've had a similar problem with omxplayerGUI when it is running in window mode (the video area inside the window ist still an overlay). If the user clicks one of the popup menues, they often pop up upwards which means that they are hidden by the video overlay. To make them usable, I set the transparency of the video to 50% for two seconds: the menues become visible behind the video area and the user can make his selection.

If we could get a toggle switch (keyboard command) in VLC which sets transparency to 50%, the user could acces the interface (e. g. move to another play position). Pressing the same toggle key a second time would restore normal view. VLC is a rather complex system and I'm not sure if something like this could be easily implemented.

[6by9]

Running a number of tests confirms everything 6by9 has explained above:

1)Playing a 1080p60 video in window mode I'm getting a huge number of lost frames

If you resize the window down to something smaller then you shouldn't get dropped frames. The resize is done in hardware (actually the same block that composes the screen).

If I switch to full screen no frames are lost any more. Even with a 720p50 TV live stream I'm getting a certain amount of lost frames in window mode, while it works perfect in full screen mode.

2) For this test I've been using a 1080i50 live TV stream and I'm running top from a ssh connection:

1080i? I'm not sure deinterlacing is plumbed in anywhere along the pipe.

Full screen mode: idle at about 90-11%, which is almost as good as omxplayer (93-95%)
Window mode: idle between 70 and 75%. Both VLC and XORG needing a lot more processing power.

It's almost all memory bandwidth. A 1080P RGBA frame is 8MB, and you're shifting 60 of those a second, so that's nearly 4Gbit/s of data that you're shovelling through solely to get it on the screen. Full screen also leaves the decoded image as YUV, so it's nearly a third of the size.
(Hang on, the codec is only rated at 1080P30 - were you really testing on 1080p60 clips?)

Full screen mode is far more effective than window mode. There might be a solution though how to access the control elements during full screen video playback. I've had a similar problem with omxplayerGUI when it is running in window mode (the video area inside the window ist still an overlay). If the user clicks one of the popup menues, they often pop up upwards which means that they are hidden by the video overlay. To make them usable, I set the transparency of the video to 50% for two seconds: the menues become visible behind the video area and the user can make his selection.

If we could get a toggle switch (keyboard command) in VLC which sets transparency to 50%, the user could acces the interface (e. g. move to another play position). Pressing the same toggle key a second time would restore normal view. VLC is a rather complex system and I'm not sure if something like this could be easily implemented.

Adding an extra command into VLC is going to be a pain, and then becomes a maintenance issue.
It may be possible to detect that VLC is trying to pop up the control panel and drop to an opaque mode, but knowing how much the guy who has been doing this has been cursing the way that VLC keeps everything in the dark about everything else I don't know how easy it would be to achieve.

[gkreidl]

1080p60 runs fine if you enhance the GPU (or H264) clock. I believe this is done automatically in omxplayer. I've always set the GPU clock to 500MHz in all my systems.

[pagenotfound]

So transport bar overlay might be possible with more CPU power? Let's hope for the upcoming hypothetical Pi 4 to be up to the task.

pagenotfound wrote:
What's more, after experimenting with fullscreen mode I noticed a tendency of all windows, even those of other programs, to become effectively undecorated. By which I mean, right clicking title bars does nothing. Left clicking on minimize, close etc. in the upper right corner does nothing anymore. Minimizing or closing via the panel still works. I have to reboot to get everything working again.

This is on a standard Raspbian install? Which Pi variant? Using the "MMAL X11 splitter for Raspberry Pi" under Preferences/Video/Output? Any tweaks to gpu_mem? Lots of things running in parallel?

Pi 3 (not Pi 3+, not Pi 3A) with standard Raspbian, some colors changed
MMAL X11 splitter for Raspberry Pi
128 MB gpu_mem
Firefox ESR 52 running in parallel (PLEASE, we need an update to FF ESR 60)

It happened to me three times. Can't give you a recipe to reproduce it, though. Then again, I'm that person who can reliably crash every Linux video editor in existence just by figuring out how to concatenate two video clips with a transition. On a big fat Intel PC. VLC hates me, too. So don't loose any sleep over this for the time being.

[ksharindam]

I still wish that fullscreen mode with mouse triggered popup transport bar was possible.

You can do that by selecting X11 video output (XCB) . Performance will be similar to that of x11 splitter for RPi in windowed mode

[6by9]

I still wish that fullscreen mode with mouse triggered popup transport bar was possible.

You can do that by selecting X11 video output (XCB) . Performance will be similar to that of x11 splitter for RPi in windowed mode

I suspect that depends on the resolution of the clip. The video will have to be resized to match the display, and that's likely to be all on the ARMs. A 1080p clip on a 1080p screen may work as no resize is required.

[Gadgetguy]

In an earlier post in this thread I had noted for those that might have been unaware that by enabling vlc's marquee display option, vlc will be able to display elapsed and total time in osd.I should also have mentioned that once this display is thus enabled. it can be activated by pressing the "t" key and that if the " t" key is pressed and held it will also display a yellow progress bar. bar at the bottom of the screen.

[aBUGSworstnightmare]

Hi,
can anybody tell me how to use (configure) the video splitter? Want to have one single video displayed over two monitors (testing the dual framebuffer beta).

Is it correct understanding that VLC is splitting the video (and not the GPU)? What is the maximum input file resolution?

[6by9]

Hi,
can anybody tell me how to use (configure) the video splitter? Want to have one single video displayed over two monitors (testing the dual framebuffer beta).

Is it correct understanding that VLC is splitting the video (and not the GPU)? What is the maximum input file resolution?

You don't configure it. VLC is generating a bitmap of the appropriate size to do a simple blit into the window's area in the frame buffer. If that window happens to be split over two frame buffers then I would hope that X does the right thing, but I don't know for certain.

Going fullscreen with multiple displays may cause some oddballs. I suspect it will always go to the primary display, whereas VLC would normally go fullscreen on whichever display had the majority of the window.

Max input file resolution is 1080P - it's the same video decoding block as has been used since the start, we haven't magically changed your silicon with something else.

[ksharindam]

I still wish that fullscreen mode with mouse triggered popup transport bar was possible.

You can do that by selecting X11 video output (XCB) . Performance will be similar to that of x11 splitter for RPi in windowed mode

I suspect that depends on the resolution of the clip. The video will have to be resized to match the display, and that's likely to be all on the ARMs. A 1080p clip on a 1080p screen may work as no resize is required.

"x11 splitter for Raspberry Pi" video output uses mmal to resize the image. the "x11 video output (XCB)" also uses mmal to resize the image. So resizing is done in GPU.

"x11 splitter for Raspberry Pi" switches between "x11 video output (XCB)" and mmal video output when fullscreen mode is changed.

[aBUGSworstnightmare]

You don't configure it. VLC is generating a bitmap of the appropriate size to do a simple blit into the window's area in the frame buffer. If that window happens to be split over two frame buffers then I would hope that X does the right thing, but I don't know for certain.

It's working, you can move the window around over two framebuffers.

Going fullscreen with multiple displays may cause some oddballs. I suspect it will always go to the primary display, whereas VLC would normally go fullscreen on whichever display had the majority of the window.

Yes, that's inline with my testings. If you try to go fullscreen on the secondary frame buffer video is moved over to the primary, leaving the secondary display all-black.

Max input file resolution is 1080P - it's the same video decoding block as has been used since the start, we haven't magically changed your silicon with something else.

Understood.

So, maybe I need to re-phrase my question.
VLC offers a '--wall' command which allows to split a single video over different screens. Is this supported on RPi too?
So, if the decoding block is used in this case as well the input file resolution is 1080P, correct? Has anybody tested already if you can go full-screen from that (by douible-clicking each window)?