terraspace
Posts: 76
Joined: Mon Dec 03, 2018 3:56 pm

HEVC/H265 Decoding

Mon Aug 05, 2019 2:40 pm

What are the timelines or proposed approach for h/w accelerated H.265 decoding on the PI, will this still be possible via IL or would I need two use two different decoder APIs ?

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23338
Joined: Sat Jul 30, 2011 7:41 pm

Re: HEVC/H265 Decoding

Mon Aug 05, 2019 3:14 pm

We intend to provide a V4L2 driver, and integration with FFMPEG. We won't be doing an IL/OpenMAX component.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

terraspace
Posts: 76
Joined: Mon Dec 03, 2018 3:56 pm

Re: HEVC/H265 Decoding

Mon Aug 05, 2019 4:42 pm

So I assume then once that is available I would be able to do something like:

IF (SourceStream IS H264) {
... USE IL Client
}
ELSE IF (SourceStream IS H265) {
... USE V4L2 Client
}

?

I specifically do not want to rely on FFMPEG and would rather be able to go directly to the OS/Driver/API level.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7135
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: HEVC/H265 Decoding

Mon Aug 05, 2019 4:57 pm

terraspace wrote:
Mon Aug 05, 2019 4:42 pm
So I assume then once that is available I would be able to do something like:

IF (SourceStream IS H264) {
... USE IL Client
}
ELSE IF (SourceStream IS H265) {
... USE V4L2 Client
}

?

I specifically do not want to rely on FFMPEG and would rather be able to go directly to the OS/Driver/API level.
Start reading the V4L2 stateless decoder API spec then - https://hverkuil.home.xs4all.nl/codec-a ... coder.html
It is not a trivial feed bitstream in, get frames out. The state is being held in the client (ie FFmpeg or your app).

I've been involved in the discussions for a while, and it's still not clear to me. Offloading the problem onto the FFmpeg or GStreamer folks has a lot of benefits.
The only realy client implementation that I'm aware of (running against Allwinner hardware) is https://github.com/bootlin/libva-v4l2-request
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

terraspace
Posts: 76
Joined: Mon Dec 03, 2018 3:56 pm

Re: HEVC/H265 Decoding

Mon Aug 05, 2019 5:07 pm

I will have a look at that,

It just seems to me that at the level of gstreamer/ffmpeg we're talking about a product and putting the ability to use h/w there makes it considerably less appealing or more difficult for someone who wanted to write an alternative to their existing product. It's a bit like an "Internet Explorer" type lock-in, although they are open source and can be utilised it feels like the wrong place to be offloading this effort to imho.

terraspace
Posts: 76
Joined: Mon Dec 03, 2018 3:56 pm

Re: HEVC/H265 Decoding

Mon Aug 05, 2019 5:11 pm

The V4L2 interface doesn't look "too" complicated. I guess with at least one comprehensive example built against it, it should be fairly easy to make use of.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7135
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: HEVC/H265 Decoding

Mon Aug 05, 2019 5:16 pm

If you can get your head around the V4L2 request API and exactly how data is required to be passed in and out of the codec, then you're probably in the top 0.001%.
The older stateful codec API, where you did just pass chunks of bitstream in and you got frames out, was many orders of complexity easier to understand.
The newer API seems to shift a load of bitstream parsing and reference frame control onto the client. This is not trivial stuff, and FFmpeg and the likes will generally implement it sensibly. IL and MMAL have shielded you from that lot up until now.

There is a MMAL wrapper around libavcodec (the libav fork annoyingly), so it may be possible to pull FFmpeg into MMAL pipelines. That's a way down the line though.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

terraspace
Posts: 76
Joined: Mon Dec 03, 2018 3:56 pm

Re: HEVC/H265 Decoding

Mon Aug 05, 2019 6:20 pm

Agreed, The state management and knowing exactly how to use the Request API is the "fun" part. As usual the documentation is found sorely wanting too (something I find a common annoyance about Linux and derivatives).

IL deceased or on it's way definitely has simplicity as a merit.


6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7135
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: HEVC/H265 Decoding

Mon Aug 05, 2019 7:08 pm

No. The first is the stateful codec api. The second is Qualcomm specific stuff.

The stateless api is implemented for Allwinner chips and that is about all at present. I can't recall what Amlogic have done with their chips at present.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

terraspace
Posts: 76
Joined: Mon Dec 03, 2018 3:56 pm

Re: HEVC/H265 Decoding

Mon Aug 05, 2019 7:25 pm

Oh dear.. I thought that code looked quite simple.. guess that explains why :)

Well hopefully we land up with a simple / clean API for this.. it would be nice if there were some proper standards in the Linux world that were actually well thought out and fully supported..

I don't need H.265 yet anyway, but it would be nice to offer it in the next year or so.

tvjon
Posts: 710
Joined: Mon Jan 07, 2013 9:11 am

Re: HEVC/H265 Decoding

Mon Aug 05, 2019 7:27 pm

terraspace wrote:
Mon Aug 05, 2019 6:20 pm
...
IL deceased or on it's way definitely has simplicity as a merit.

Agreed, after all the effort to try to understand it, seems such a waste...

6by9 wrote:
Mon Aug 05, 2019 5:16 pm
...
There is a MMAL wrapper around libavcodec (the libav fork annoyingly), so it may be possible to pull FFmpeg into MMAL pipelines. ...
Apparently, in ffmpeg, threading & hardware acceleration are mutually exclusive, so would libav get around that?

pik33
Posts: 173
Joined: Thu Sep 10, 2015 4:26 pm

Re: HEVC/H265 Decoding

Mon Aug 05, 2019 8:02 pm

Now I don't understand this.

I understand the hardware codec is a set of registers at defined addresses. You put commands and parameters there, you get results from there. Why has it to be wrapped in a complex library/api which only 0.001% of people can understand?

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23338
Joined: Sat Jul 30, 2011 7:41 pm

Re: HEVC/H265 Decoding

Mon Aug 05, 2019 8:29 pm

pik33 wrote:
Mon Aug 05, 2019 8:02 pm
Now I don't understand this.

I understand the hardware codec is a set of registers at defined addresses. You put commands and parameters there, you get results from there. Why has it to be wrapped in a complex library/api which only 0.001% of people can understand?
Because you need a standard API, otherwise every app that wants to use it needs to write its own code to do so.

That doesn't explain why these standard API's are so hard to understand, but that could simply be down to the fact that the job they need to do is complex. Driver writing (which is what this is) has never been easy since OS's became complex.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

terraspace
Posts: 76
Joined: Mon Dec 03, 2018 3:56 pm

Re: HEVC/H265 Decoding

Mon Aug 05, 2019 8:54 pm

I think most of the issue comes down to good documentation. If things are well documented and thus enough people can use the API effectively, improvements and issues can be worked out and the API improved over time from it's initial design. I find the issue with a lot of this stuff is that it becomes so arcane from the start that very few people even try and hence the API doesn't get the visibility or adoption required.

That is one area where M$ have gotten things right, solid APIs, tons of documentation and examples and the odds are that someone has done what you want before and you're likely to find an example, tutorial, S/O post about it etc.. Quite often in the Linux world you venture down a black hole of 1990's esque text-only web sites and repositories, trawling through git repos trying to figure out what you're suppose to do..

ALSA is a prime example of this.. It's not the worst API ever, but the documentation might lead you to believe so.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7135
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: HEVC/H265 Decoding

Mon Aug 05, 2019 9:10 pm

To those complaining, then this is all open source development. Feel free to subscribe to the linux-media mailing list and join in the discussion of the APIs and contribute your expertise.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

terraspace
Posts: 76
Joined: Mon Dec 03, 2018 3:56 pm

Re: HEVC/H265 Decoding

Wed Aug 07, 2019 6:50 pm

I read in another thread that V4L2 was going to be built ontop of MMAL. Looking at the MMAL Encode/Decode examples they seem very similar to the lower IL interface, surely then MMAL would have access to the H265 encoder ?

trejan
Posts: 516
Joined: Tue Jul 02, 2019 2:28 pm

Re: HEVC/H265 Decoding

Wed Aug 07, 2019 6:54 pm

terraspace wrote:
Wed Aug 07, 2019 6:50 pm
I read in another thread that V4L2 was going to be built ontop of MMAL.
No? What thread did you read that said that?
terraspace wrote:
Wed Aug 07, 2019 6:50 pm
Looking at the MMAL Encode/Decode examples they seem very similar to the lower IL interface, surely then MMAL would have access to the H265 encoder ?
MMAL is built on top of IL so it looking similar isn't that surprising.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7135
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: HEVC/H265 Decoding

Wed Aug 07, 2019 7:47 pm

trejan wrote:
Wed Aug 07, 2019 6:54 pm
terraspace wrote:
Wed Aug 07, 2019 6:50 pm
I read in another thread that V4L2 was going to be built ontop of MMAL.
No? What thread did you read that said that?
The H264 legacy codecs stateful V4L2 drivers are built on top of MMAL (as is the V4L2 camera driver).
H265 is an entirely separate block, and requires a different style driver.
trejan wrote:
terraspace wrote:
Wed Aug 07, 2019 6:50 pm
Looking at the MMAL Encode/Decode examples they seem very similar to the lower IL interface, surely then MMAL would have access to the H265 encoder ?
MMAL is built on top of IL so it looking similar isn't that surprising.
Not quite.
IL has a large block of code that would be duplicated. All of that is abstracted out into a core (referred to as ILCS). The components implement a reduced IL API, known as RIL.
MMAL reuses the RIL API to allow reuse of the component code with minimal modification.

MMAL does allow ARM side components, and indeed there is a libavcodec wrapper component. It hasn't been used for a long time, but it may be possible to resurrect it to allow use of H265 from MMAL.
No such option really exists in IL, therefore this is not an option with that API.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

trejan
Posts: 516
Joined: Tue Jul 02, 2019 2:28 pm

Re: HEVC/H265 Decoding

Wed Aug 07, 2019 7:57 pm

6by9 wrote:
Wed Aug 07, 2019 7:47 pm
The H264 legacy codecs stateful V4L2 drivers are built on top of MMAL (as is the V4L2 camera driver).
H265 is an entirely separate block, and requires a different style driver.
Ah. I was only thinking about h.265.
6by9 wrote:
Wed Aug 07, 2019 7:47 pm
Not quite.
IL has a large block of code that would be duplicated. All of that is abstracted out into a core (referred to as ILCS). The components implement a reduced IL API, known as RIL.
MMAL reuses the RIL API to allow reuse of the component code with minimal modification.

MMAL does allow ARM side components, and indeed there is a libavcodec wrapper component. It hasn't been used for a long time, but it may be possible to resurrect it to allow use of H265 from MMAL.
No such option really exists in IL, therefore this is not an option with that API.
Thanks for explaining how it all fits together.

terraspace
Posts: 76
Joined: Mon Dec 03, 2018 3:56 pm

Re: HEVC/H265 Decoding

Wed Aug 07, 2019 10:15 pm

Useful to know the structure of the stack there, thanks 6by9!

One thing that is very useful with IL H264 implementation is the ability to configure it with Dispmanx, so for example a UI on layer 1 and the video render running on layer 2. Would that still be possible with MMAL? (Assuming H265 is at some point implemented via MMAL).

How does the V4L2 driver and Dispmanx work together or do they not ?

User avatar
dividuum
Posts: 163
Joined: Sun Jun 16, 2013 1:18 pm
Location: Germany
Contact: Website

Re: HEVC/H265 Decoding

Thu Aug 08, 2019 11:54 am

6by9 wrote:
Wed Aug 07, 2019 7:47 pm
MMAL does allow ARM side components, and indeed there is a libavcodec wrapper component. It hasn't been used for a long time, but it may be possible to resurrect it to allow use of H265 from MMAL.
No such option really exists in IL, therefore this is not an option with that API.
Isn't the ALSA support in omxplayer implemented with a custom OMX component? Or is that not what's happening there? Or is that considered hacky?

Custom MMAL components seem easier and cleaner to implement though and a H265 component would be superb.
info-beamer hosted - A user and programmer friendly digital signage platform for the Pi: https://info-beamer.com/hosted

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7135
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: HEVC/H265 Decoding

Thu Aug 08, 2019 12:53 pm

dividuum wrote:
Thu Aug 08, 2019 11:54 am
6by9 wrote:
Wed Aug 07, 2019 7:47 pm
MMAL does allow ARM side components, and indeed there is a libavcodec wrapper component. It hasn't been used for a long time, but it may be possible to resurrect it to allow use of H265 from MMAL.
No such option really exists in IL, therefore this is not an option with that API.
Isn't the ALSA support in omxplayer implemented with a custom OMX component? Or is that not what's happening there? Or is that considered hacky?
That would be a hack in my view.
https://github.com/popcornmix/omxplayer ... .cpp#L1433 intercepts component create calls for components starting "OMX.alsa." and sends them to an alternate set of create functions.
If I recall the IL spec correctly then technically you can have multiple component providers, but reality is that trying to get that to work isn't trivial.
dividuum wrote:Custom MMAL components seem easier and cleaner to implement though and a H265 component would be superb.
Yes, MMAL is set up for all component providers to call mmal_component_supplier_register with the appropriate search terms. The core then keeps track of all the providers, and on component create will search for the appropriate provider.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

terraspace
Posts: 76
Joined: Mon Dec 03, 2018 3:56 pm

Re: HEVC/H265 Decoding

Thu Aug 08, 2019 5:01 pm

So will the V4L2 interface inter-op with DISPMANX ? IE: Video Render to a particular Dispmanx layer ?
Or would the idea be that v4l2 would have no ability to render, simply decode the frame and which point you could render it with MMAL or another API?

Return to “Graphics programming”