-
- Posts: 84
- Joined: Mon Dec 03, 2018 3:56 pm
HEVC/H265 Decoding
What are the timelines or proposed approach for h/w accelerated H.265 decoding on the PI, will this still be possible via IL or would I need two use two different decoder APIs ?
Re: HEVC/H265 Decoding
We intend to provide a V4L2 driver, and integration with FFMPEG. We won't be doing an IL/OpenMAX component.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed.
I've been saying "Mucho" to my Spanish friend a lot more lately. It means a lot to him.
Contrary to popular belief, humorous signatures are allowed.
I've been saying "Mucho" to my Spanish friend a lot more lately. It means a lot to him.
-
- Posts: 84
- Joined: Mon Dec 03, 2018 3:56 pm
Re: HEVC/H265 Decoding
So I assume then once that is available I would be able to do something like:
IF (SourceStream IS H264) {
... USE IL Client
}
ELSE IF (SourceStream IS H265) {
... USE V4L2 Client
}
?
I specifically do not want to rely on FFMPEG and would rather be able to go directly to the OS/Driver/API level.
IF (SourceStream IS H264) {
... USE IL Client
}
ELSE IF (SourceStream IS H265) {
... USE V4L2 Client
}
?
I specifically do not want to rely on FFMPEG and would rather be able to go directly to the OS/Driver/API level.
-
- Raspberry Pi Engineer & Forum Moderator
- Posts: 10258
- Joined: Wed Dec 04, 2013 11:27 am
- Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.
Re: HEVC/H265 Decoding
Start reading the V4L2 stateless decoder API spec then - https://hverkuil.home.xs4all.nl/codec-a ... coder.htmlterraspace wrote: ↑Mon Aug 05, 2019 4:42 pmSo I assume then once that is available I would be able to do something like:
IF (SourceStream IS H264) {
... USE IL Client
}
ELSE IF (SourceStream IS H265) {
... USE V4L2 Client
}
?
I specifically do not want to rely on FFMPEG and would rather be able to go directly to the OS/Driver/API level.
It is not a trivial feed bitstream in, get frames out. The state is being held in the client (ie FFmpeg or your app).
I've been involved in the discussions for a while, and it's still not clear to me. Offloading the problem onto the FFmpeg or GStreamer folks has a lot of benefits.
The only realy client implementation that I'm aware of (running against Allwinner hardware) is https://github.com/bootlin/libva-v4l2-request
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.
I'm not interested in doing contracts for bespoke functionality - please don't ask.
-
- Posts: 84
- Joined: Mon Dec 03, 2018 3:56 pm
Re: HEVC/H265 Decoding
I will have a look at that,
It just seems to me that at the level of gstreamer/ffmpeg we're talking about a product and putting the ability to use h/w there makes it considerably less appealing or more difficult for someone who wanted to write an alternative to their existing product. It's a bit like an "Internet Explorer" type lock-in, although they are open source and can be utilised it feels like the wrong place to be offloading this effort to imho.
It just seems to me that at the level of gstreamer/ffmpeg we're talking about a product and putting the ability to use h/w there makes it considerably less appealing or more difficult for someone who wanted to write an alternative to their existing product. It's a bit like an "Internet Explorer" type lock-in, although they are open source and can be utilised it feels like the wrong place to be offloading this effort to imho.
-
- Posts: 84
- Joined: Mon Dec 03, 2018 3:56 pm
Re: HEVC/H265 Decoding
The V4L2 interface doesn't look "too" complicated. I guess with at least one comprehensive example built against it, it should be fairly easy to make use of.
-
- Raspberry Pi Engineer & Forum Moderator
- Posts: 10258
- Joined: Wed Dec 04, 2013 11:27 am
- Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.
Re: HEVC/H265 Decoding
If you can get your head around the V4L2 request API and exactly how data is required to be passed in and out of the codec, then you're probably in the top 0.001%.
The older stateful codec API, where you did just pass chunks of bitstream in and you got frames out, was many orders of complexity easier to understand.
The newer API seems to shift a load of bitstream parsing and reference frame control onto the client. This is not trivial stuff, and FFmpeg and the likes will generally implement it sensibly. IL and MMAL have shielded you from that lot up until now.
There is a MMAL wrapper around libavcodec (the libav fork annoyingly), so it may be possible to pull FFmpeg into MMAL pipelines. That's a way down the line though.
The older stateful codec API, where you did just pass chunks of bitstream in and you got frames out, was many orders of complexity easier to understand.
The newer API seems to shift a load of bitstream parsing and reference frame control onto the client. This is not trivial stuff, and FFmpeg and the likes will generally implement it sensibly. IL and MMAL have shielded you from that lot up until now.
There is a MMAL wrapper around libavcodec (the libav fork annoyingly), so it may be possible to pull FFmpeg into MMAL pipelines. That's a way down the line though.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.
I'm not interested in doing contracts for bespoke functionality - please don't ask.
-
- Posts: 84
- Joined: Mon Dec 03, 2018 3:56 pm
Re: HEVC/H265 Decoding
Agreed, The state management and knowing exactly how to use the Request API is the "fun" part. As usual the documentation is found sorely wanting too (something I find a common annoyance about Linux and derivatives).
IL deceased or on it's way definitely has simplicity as a merit.
IL deceased or on it's way definitely has simplicity as a merit.
-
- Posts: 84
- Joined: Mon Dec 03, 2018 3:56 pm
-
- Raspberry Pi Engineer & Forum Moderator
- Posts: 10258
- Joined: Wed Dec 04, 2013 11:27 am
- Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.
Re: HEVC/H265 Decoding
No. The first is the stateful codec api. The second is Qualcomm specific stuff.terraspace wrote: ↑Mon Aug 05, 2019 6:43 pmSome more useful info
https://events.static.linuxfound.org/im ... debski.pdf
https://github.com/rawoul/v4l2-decode
The stateless api is implemented for Allwinner chips and that is about all at present. I can't recall what Amlogic have done with their chips at present.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.
I'm not interested in doing contracts for bespoke functionality - please don't ask.
-
- Posts: 84
- Joined: Mon Dec 03, 2018 3:56 pm
Re: HEVC/H265 Decoding
Oh dear.. I thought that code looked quite simple.. guess that explains why 
Well hopefully we land up with a simple / clean API for this.. it would be nice if there were some proper standards in the Linux world that were actually well thought out and fully supported..
I don't need H.265 yet anyway, but it would be nice to offer it in the next year or so.

Well hopefully we land up with a simple / clean API for this.. it would be nice if there were some proper standards in the Linux world that were actually well thought out and fully supported..
I don't need H.265 yet anyway, but it would be nice to offer it in the next year or so.
Re: HEVC/H265 Decoding
terraspace wrote: ↑Mon Aug 05, 2019 6:20 pm...
IL deceased or on it's way definitely has simplicity as a merit.
Agreed, after all the effort to try to understand it, seems such a waste...
Apparently, in ffmpeg, threading & hardware acceleration are mutually exclusive, so would libav get around that?
Re: HEVC/H265 Decoding
Now I don't understand this.
I understand the hardware codec is a set of registers at defined addresses. You put commands and parameters there, you get results from there. Why has it to be wrapped in a complex library/api which only 0.001% of people can understand?
I understand the hardware codec is a set of registers at defined addresses. You put commands and parameters there, you get results from there. Why has it to be wrapped in a complex library/api which only 0.001% of people can understand?
Re: HEVC/H265 Decoding
Because you need a standard API, otherwise every app that wants to use it needs to write its own code to do so.pik33 wrote: ↑Mon Aug 05, 2019 8:02 pmNow I don't understand this.
I understand the hardware codec is a set of registers at defined addresses. You put commands and parameters there, you get results from there. Why has it to be wrapped in a complex library/api which only 0.001% of people can understand?
That doesn't explain why these standard API's are so hard to understand, but that could simply be down to the fact that the job they need to do is complex. Driver writing (which is what this is) has never been easy since OS's became complex.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed.
I've been saying "Mucho" to my Spanish friend a lot more lately. It means a lot to him.
Contrary to popular belief, humorous signatures are allowed.
I've been saying "Mucho" to my Spanish friend a lot more lately. It means a lot to him.
-
- Posts: 84
- Joined: Mon Dec 03, 2018 3:56 pm
Re: HEVC/H265 Decoding
I think most of the issue comes down to good documentation. If things are well documented and thus enough people can use the API effectively, improvements and issues can be worked out and the API improved over time from it's initial design. I find the issue with a lot of this stuff is that it becomes so arcane from the start that very few people even try and hence the API doesn't get the visibility or adoption required.
That is one area where M$ have gotten things right, solid APIs, tons of documentation and examples and the odds are that someone has done what you want before and you're likely to find an example, tutorial, S/O post about it etc.. Quite often in the Linux world you venture down a black hole of 1990's esque text-only web sites and repositories, trawling through git repos trying to figure out what you're suppose to do..
ALSA is a prime example of this.. It's not the worst API ever, but the documentation might lead you to believe so.
That is one area where M$ have gotten things right, solid APIs, tons of documentation and examples and the odds are that someone has done what you want before and you're likely to find an example, tutorial, S/O post about it etc.. Quite often in the Linux world you venture down a black hole of 1990's esque text-only web sites and repositories, trawling through git repos trying to figure out what you're suppose to do..
ALSA is a prime example of this.. It's not the worst API ever, but the documentation might lead you to believe so.
-
- Raspberry Pi Engineer & Forum Moderator
- Posts: 10258
- Joined: Wed Dec 04, 2013 11:27 am
- Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.
Re: HEVC/H265 Decoding
To those complaining, then this is all open source development. Feel free to subscribe to the linux-media mailing list and join in the discussion of the APIs and contribute your expertise.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.
I'm not interested in doing contracts for bespoke functionality - please don't ask.
-
- Posts: 84
- Joined: Mon Dec 03, 2018 3:56 pm
Re: HEVC/H265 Decoding
I read in another thread that V4L2 was going to be built ontop of MMAL. Looking at the MMAL Encode/Decode examples they seem very similar to the lower IL interface, surely then MMAL would have access to the H265 encoder ?
Re: HEVC/H265 Decoding
No? What thread did you read that said that?terraspace wrote: ↑Wed Aug 07, 2019 6:50 pmI read in another thread that V4L2 was going to be built ontop of MMAL.
MMAL is built on top of IL so it looking similar isn't that surprising.terraspace wrote: ↑Wed Aug 07, 2019 6:50 pmLooking at the MMAL Encode/Decode examples they seem very similar to the lower IL interface, surely then MMAL would have access to the H265 encoder ?
-
- Raspberry Pi Engineer & Forum Moderator
- Posts: 10258
- Joined: Wed Dec 04, 2013 11:27 am
- Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.
Re: HEVC/H265 Decoding
The H264 legacy codecs stateful V4L2 drivers are built on top of MMAL (as is the V4L2 camera driver).trejan wrote: ↑Wed Aug 07, 2019 6:54 pmNo? What thread did you read that said that?terraspace wrote: ↑Wed Aug 07, 2019 6:50 pmI read in another thread that V4L2 was going to be built ontop of MMAL.
H265 is an entirely separate block, and requires a different style driver.
Not quite.trejan wrote:MMAL is built on top of IL so it looking similar isn't that surprising.terraspace wrote: ↑Wed Aug 07, 2019 6:50 pmLooking at the MMAL Encode/Decode examples they seem very similar to the lower IL interface, surely then MMAL would have access to the H265 encoder ?
IL has a large block of code that would be duplicated. All of that is abstracted out into a core (referred to as ILCS). The components implement a reduced IL API, known as RIL.
MMAL reuses the RIL API to allow reuse of the component code with minimal modification.
MMAL does allow ARM side components, and indeed there is a libavcodec wrapper component. It hasn't been used for a long time, but it may be possible to resurrect it to allow use of H265 from MMAL.
No such option really exists in IL, therefore this is not an option with that API.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.
I'm not interested in doing contracts for bespoke functionality - please don't ask.
Re: HEVC/H265 Decoding
Ah. I was only thinking about h.265.
Thanks for explaining how it all fits together.6by9 wrote: ↑Wed Aug 07, 2019 7:47 pmNot quite.
IL has a large block of code that would be duplicated. All of that is abstracted out into a core (referred to as ILCS). The components implement a reduced IL API, known as RIL.
MMAL reuses the RIL API to allow reuse of the component code with minimal modification.
MMAL does allow ARM side components, and indeed there is a libavcodec wrapper component. It hasn't been used for a long time, but it may be possible to resurrect it to allow use of H265 from MMAL.
No such option really exists in IL, therefore this is not an option with that API.
-
- Posts: 84
- Joined: Mon Dec 03, 2018 3:56 pm
Re: HEVC/H265 Decoding
Useful to know the structure of the stack there, thanks 6by9!
One thing that is very useful with IL H264 implementation is the ability to configure it with Dispmanx, so for example a UI on layer 1 and the video render running on layer 2. Would that still be possible with MMAL? (Assuming H265 is at some point implemented via MMAL).
How does the V4L2 driver and Dispmanx work together or do they not ?
One thing that is very useful with IL H264 implementation is the ability to configure it with Dispmanx, so for example a UI on layer 1 and the video render running on layer 2. Would that still be possible with MMAL? (Assuming H265 is at some point implemented via MMAL).
How does the V4L2 driver and Dispmanx work together or do they not ?
Re: HEVC/H265 Decoding
Isn't the ALSA support in omxplayer implemented with a custom OMX component? Or is that not what's happening there? Or is that considered hacky?6by9 wrote: ↑Wed Aug 07, 2019 7:47 pmMMAL does allow ARM side components, and indeed there is a libavcodec wrapper component. It hasn't been used for a long time, but it may be possible to resurrect it to allow use of H265 from MMAL.
No such option really exists in IL, therefore this is not an option with that API.
Custom MMAL components seem easier and cleaner to implement though and a H265 component would be superb.
info-beamer hosted - A user and programmer friendly digital signage platform for the Pi: https://info-beamer.com/hosted
-
- Raspberry Pi Engineer & Forum Moderator
- Posts: 10258
- Joined: Wed Dec 04, 2013 11:27 am
- Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.
Re: HEVC/H265 Decoding
That would be a hack in my view.dividuum wrote: ↑Thu Aug 08, 2019 11:54 amIsn't the ALSA support in omxplayer implemented with a custom OMX component? Or is that not what's happening there? Or is that considered hacky?6by9 wrote: ↑Wed Aug 07, 2019 7:47 pmMMAL does allow ARM side components, and indeed there is a libavcodec wrapper component. It hasn't been used for a long time, but it may be possible to resurrect it to allow use of H265 from MMAL.
No such option really exists in IL, therefore this is not an option with that API.
https://github.com/popcornmix/omxplayer ... .cpp#L1433 intercepts component create calls for components starting "OMX.alsa." and sends them to an alternate set of create functions.
If I recall the IL spec correctly then technically you can have multiple component providers, but reality is that trying to get that to work isn't trivial.
Yes, MMAL is set up for all component providers to call mmal_component_supplier_register with the appropriate search terms. The core then keeps track of all the providers, and on component create will search for the appropriate provider.dividuum wrote:Custom MMAL components seem easier and cleaner to implement though and a H265 component would be superb.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.
I'm not interested in doing contracts for bespoke functionality - please don't ask.
-
- Posts: 84
- Joined: Mon Dec 03, 2018 3:56 pm
Re: HEVC/H265 Decoding
So will the V4L2 interface inter-op with DISPMANX ? IE: Video Render to a particular Dispmanx layer ?
Or would the idea be that v4l2 would have no ability to render, simply decode the frame and which point you could render it with MMAL or another API?
Or would the idea be that v4l2 would have no ability to render, simply decode the frame and which point you could render it with MMAL or another API?