HEVC/H265 Decoding

[terraspace]

What are the timelines or proposed approach for h/w accelerated H.265 decoding on the PI, will this still be possible via IL or would I need two use two different decoder APIs ?

[jamesh]

We intend to provide a V4L2 driver, and integration with FFMPEG. We won't be doing an IL/OpenMAX component.

[terraspace]

So I assume then once that is available I would be able to do something like:

IF (SourceStream IS H264) {
... USE IL Client
}
ELSE IF (SourceStream IS H265) {
... USE V4L2 Client
}

?

I specifically do not want to rely on FFMPEG and would rather be able to go directly to the OS/Driver/API level.

[6by9]

So I assume then once that is available I would be able to do something like:

IF (SourceStream IS H264) {
... USE IL Client
}
ELSE IF (SourceStream IS H265) {
... USE V4L2 Client
}

?

I specifically do not want to rely on FFMPEG and would rather be able to go directly to the OS/Driver/API level.

Start reading the V4L2 stateless decoder API spec then - https://hverkuil.home.xs4all.nl/codec-a ... coder.html
It is not a trivial feed bitstream in, get frames out. The state is being held in the client (ie FFmpeg or your app).

I've been involved in the discussions for a while, and it's still not clear to me. Offloading the problem onto the FFmpeg or GStreamer folks has a lot of benefits.
The only realy client implementation that I'm aware of (running against Allwinner hardware) is https://github.com/bootlin/libva-v4l2-request

[terraspace]

I will have a look at that,

It just seems to me that at the level of gstreamer/ffmpeg we're talking about a product and putting the ability to use h/w there makes it considerably less appealing or more difficult for someone who wanted to write an alternative to their existing product. It's a bit like an "Internet Explorer" type lock-in, although they are open source and can be utilised it feels like the wrong place to be offloading this effort to imho.

[terraspace]

The V4L2 interface doesn't look "too" complicated. I guess with at least one comprehensive example built against it, it should be fairly easy to make use of.

[6by9]

If you can get your head around the V4L2 request API and exactly how data is required to be passed in and out of the codec, then you're probably in the top 0.001%.
The older stateful codec API, where you did just pass chunks of bitstream in and you got frames out, was many orders of complexity easier to understand.
The newer API seems to shift a load of bitstream parsing and reference frame control onto the client. This is not trivial stuff, and FFmpeg and the likes will generally implement it sensibly. IL and MMAL have shielded you from that lot up until now.

There is a MMAL wrapper around libavcodec (the libav fork annoyingly), so it may be possible to pull FFmpeg into MMAL pipelines. That's a way down the line though.

[terraspace]

Agreed, The state management and knowing exactly how to use the Request API is the "fun" part. As usual the documentation is found sorely wanting too (something I find a common annoyance about Linux and derivatives).

IL deceased or on it's way definitely has simplicity as a merit.

[terraspace]

Some more useful info

https://events.static.linuxfound.org/im ... debski.pdf
https://github.com/rawoul/v4l2-decode

[6by9]

Some more useful info

https://events.static.linuxfound.org/im ... debski.pdf
https://github.com/rawoul/v4l2-decode

No. The first is the stateful codec api. The second is Qualcomm specific stuff.

The stateless api is implemented for Allwinner chips and that is about all at present. I can't recall what Amlogic have done with their chips at present.

[terraspace]

Oh dear.. I thought that code looked quite simple.. guess that explains why

Well hopefully we land up with a simple / clean API for this.. it would be nice if there were some proper standards in the Linux world that were actually well thought out and fully supported..

I don't need H.265 yet anyway, but it would be nice to offer it in the next year or so.

[tvjon]

...
IL deceased or on it's way definitely has simplicity as a merit.

Agreed, after all the effort to try to understand it, seems such a waste...

...
There is a MMAL wrapper around libavcodec (the libav fork annoyingly), so it may be possible to pull FFmpeg into MMAL pipelines. ...

Apparently, in ffmpeg, threading & hardware acceleration are mutually exclusive, so would libav get around that?

[pik33]

Now I don't understand this.

I understand the hardware codec is a set of registers at defined addresses. You put commands and parameters there, you get results from there. Why has it to be wrapped in a complex library/api which only 0.001% of people can understand?

[jamesh]

Now I don't understand this.

I understand the hardware codec is a set of registers at defined addresses. You put commands and parameters there, you get results from there. Why has it to be wrapped in a complex library/api which only 0.001% of people can understand?

Because you need a standard API, otherwise every app that wants to use it needs to write its own code to do so.

That doesn't explain why these standard API's are so hard to understand, but that could simply be down to the fact that the job they need to do is complex. Driver writing (which is what this is) has never been easy since OS's became complex.

[terraspace]

I think most of the issue comes down to good documentation. If things are well documented and thus enough people can use the API effectively, improvements and issues can be worked out and the API improved over time from it's initial design. I find the issue with a lot of this stuff is that it becomes so arcane from the start that very few people even try and hence the API doesn't get the visibility or adoption required.

That is one area where M$ have gotten things right, solid APIs, tons of documentation and examples and the odds are that someone has done what you want before and you're likely to find an example, tutorial, S/O post about it etc.. Quite often in the Linux world you venture down a black hole of 1990's esque text-only web sites and repositories, trawling through git repos trying to figure out what you're suppose to do..

ALSA is a prime example of this.. It's not the worst API ever, but the documentation might lead you to believe so.

[6by9]

To those complaining, then this is all open source development. Feel free to subscribe to the linux-media mailing list and join in the discussion of the APIs and contribute your expertise.

[terraspace]

I read in another thread that V4L2 was going to be built ontop of MMAL. Looking at the MMAL Encode/Decode examples they seem very similar to the lower IL interface, surely then MMAL would have access to the H265 encoder ?

[trejan]

I read in another thread that V4L2 was going to be built ontop of MMAL.

No? What thread did you read that said that?

Looking at the MMAL Encode/Decode examples they seem very similar to the lower IL interface, surely then MMAL would have access to the H265 encoder ?

MMAL is built on top of IL so it looking similar isn't that surprising.

[6by9]

I read in another thread that V4L2 was going to be built ontop of MMAL.

No? What thread did you read that said that?

The H264 legacy codecs stateful V4L2 drivers are built on top of MMAL (as is the V4L2 camera driver).
H265 is an entirely separate block, and requires a different style driver.

Looking at the MMAL Encode/Decode examples they seem very similar to the lower IL interface, surely then MMAL would have access to the H265 encoder ?

MMAL is built on top of IL so it looking similar isn't that surprising.

Not quite.
IL has a large block of code that would be duplicated. All of that is abstracted out into a core (referred to as ILCS). The components implement a reduced IL API, known as RIL.
MMAL reuses the RIL API to allow reuse of the component code with minimal modification.

MMAL does allow ARM side components, and indeed there is a libavcodec wrapper component. It hasn't been used for a long time, but it may be possible to resurrect it to allow use of H265 from MMAL.
No such option really exists in IL, therefore this is not an option with that API.

[trejan]

The H264 legacy codecs stateful V4L2 drivers are built on top of MMAL (as is the V4L2 camera driver).
H265 is an entirely separate block, and requires a different style driver.

Ah. I was only thinking about h.265.

Not quite.
IL has a large block of code that would be duplicated. All of that is abstracted out into a core (referred to as ILCS). The components implement a reduced IL API, known as RIL.
MMAL reuses the RIL API to allow reuse of the component code with minimal modification.

MMAL does allow ARM side components, and indeed there is a libavcodec wrapper component. It hasn't been used for a long time, but it may be possible to resurrect it to allow use of H265 from MMAL.
No such option really exists in IL, therefore this is not an option with that API.

Thanks for explaining how it all fits together.

[terraspace]

Useful to know the structure of the stack there, thanks 6by9!

One thing that is very useful with IL H264 implementation is the ability to configure it with Dispmanx, so for example a UI on layer 1 and the video render running on layer 2. Would that still be possible with MMAL? (Assuming H265 is at some point implemented via MMAL).

How does the V4L2 driver and Dispmanx work together or do they not ?

[dividuum]

MMAL does allow ARM side components, and indeed there is a libavcodec wrapper component. It hasn't been used for a long time, but it may be possible to resurrect it to allow use of H265 from MMAL.
No such option really exists in IL, therefore this is not an option with that API.

Isn't the ALSA support in omxplayer implemented with a custom OMX component? Or is that not what's happening there? Or is that considered hacky?

Custom MMAL components seem easier and cleaner to implement though and a H265 component would be superb.

[6by9]

MMAL does allow ARM side components, and indeed there is a libavcodec wrapper component. It hasn't been used for a long time, but it may be possible to resurrect it to allow use of H265 from MMAL.
No such option really exists in IL, therefore this is not an option with that API.

Isn't the ALSA support in omxplayer implemented with a custom OMX component? Or is that not what's happening there? Or is that considered hacky?

That would be a hack in my view.
https://github.com/popcornmix/omxplayer ... .cpp#L1433 intercepts component create calls for components starting "OMX.alsa." and sends them to an alternate set of create functions.
If I recall the IL spec correctly then technically you can have multiple component providers, but reality is that trying to get that to work isn't trivial.

Custom MMAL components seem easier and cleaner to implement though and a H265 component would be superb.

Yes, MMAL is set up for all component providers to call mmal_component_supplier_register with the appropriate search terms. The core then keeps track of all the providers, and on component create will search for the appropriate provider.

[terraspace]

So will the V4L2 interface inter-op with DISPMANX ? IE: Video Render to a particular Dispmanx layer ?
Or would the idea be that v4l2 would have no ability to render, simply decode the frame and which point you could render it with MMAL or another API?