Hardware-assisted H.264 video encoding


52 posts   Page 1 of 3   1, 2, 3
by pyke369 » Sat Feb 04, 2012 9:06 pm
Hi everyone,

According to http://www.broadcom.com/products/BCM2835, the BCM2835 has a VideoCore IV onboard, capable of Full HD (1080p) High Profile H.264 encoding and decoding. Will these capabilities be exposed as a specific API (or through some V4L2 structures)?

Also, the VC IV is supposed to be reconfigurable at will (soft silicon) to enable other codecs support: will this be exposed to the developer too?

Thanks for your answer.

Cheers,

Pierre-Yves
Posts: 15
Joined: Sat Feb 04, 2012 8:54 pm
by jamesh » Sat Feb 04, 2012 10:02 pm
pyke369 said:


Hi everyone,

According to http://www.broadcom.com/products/BCM2835, the BCM2835 has a VideoCore IV onboard, capable of Full HD (1080p) High Profile H.264 encoding and decoding. Will these capabilities be exposed as a specific API (or through some V4L2 structures)?

Also, the VC IV is supposed to be reconfigurable at will (soft silicon) to enable other codecs support: will this be exposed to the developer too?

Thanks for your answer.

Cheers,

Pierre-Yves


I think you first point of call should be the blog post on the home page which talks about codecs.

Decode is definitely available via OpenMAX and a custom player to be provided. I think encode is as well, although OpenMAX can be a struggle to work with.

Codec support is limited to that provided by the SoC supplier, and whatever licences the Foundations provide. You cannot write you own codecs to run on the GPU.
Unemployed software engineer currently specialising in camera drivers and frameworks, but can put mind to most embedded tasks. Got a job in N.Cambridge or surroundings? I'm interested!
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 11728
Joined: Sat Jul 30, 2011 7:41 pm
by pyke369 » Sat Feb 04, 2012 10:24 pm
Thanks for your reply James, really appreciate it. I'll have a look at OpenMAX.

I'm interested in building a low-cost mobile HD encoder that would hook an HD camcorder output to a bunch of network-bundled 3G/WiFi USB adapters in order to stream HD H.264/AAC streams live from any location.

I know these systems already exist, but they are _really_ expensive and the R.Pi could provide the same service for a fraction of the price. I've already experimented with other embedded platforms (TI) for hw video encoding but was not really pleased by the software support.

I'm putting high hopes in this project for this ;-)
Posts: 15
Joined: Sat Feb 04, 2012 8:54 pm
by jamesh » Sun Feb 05, 2012 9:36 am
It certainly very possible, the SoC supplier launched something very similar at CES last month, using the same GPU core.
Unemployed software engineer currently specialising in camera drivers and frameworks, but can put mind to most embedded tasks. Got a job in N.Cambridge or surroundings? I'm interested!
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 11728
Joined: Sat Jul 30, 2011 7:41 pm
by pyke369 » Sun Feb 05, 2012 11:40 am
Thanks James, I'll have a look at what Broadcom announced at CES.

In the meantime, I found the following in the comments from http://www.raspberrypi.org/arc.....2#comments:

JamesH on January 31, 2012 at 3:45 pm said:

Don’t quote me on this – I think if you can write the appropriate OpenMAX code (and its not easy), you may be able to do encode of a file stream. Depends if its enabled on the GPU.

I hope this is enabled by Broadcom for the R.Pi then!
Posts: 15
Joined: Sat Feb 04, 2012 8:54 pm
by drgeoff » Sun Feb 05, 2012 11:41 am
pyke369 said:


I'm interested in building a low-cost mobile HD encoder that would hook an HD camcorder output to a bunch of network-bundled 3G/WiFi USB adapters in order to stream HD H.264/AAC streams live from any location.


How are you intending getting that HD camcorder output (HDMI?) into the RP (even if it could do the encoding)?
Posts: 2862
Joined: Wed Jan 25, 2012 6:39 pm
by pyke369 » Sun Feb 05, 2012 11:58 am
@drgeoff

Semi-professional HD camcorder (like the ones from Sony) generally have either analogHD or SDI outputs. I would use some USB capture card to get the raw frames inside the R.Pi and then use the GPU to offload H.264 encoding. Dunno about USB bandwidth on R.Pi (USB 2.0 / 480Mbps theorical), but there should be enought horsepower to do the trick.

Another interesting application would be to build some H.264 real-time live transcoding arrays, to re-encode live streams to different qualities on the fly, for a very low TCO.

Cheers,

Pierre-Yves
Posts: 15
Joined: Sat Feb 04, 2012 8:54 pm
by drgeoff » Sun Feb 05, 2012 12:11 pm
pyke369 said:


@drgeoff

Semi-professional HD camcorder (like the ones from Sony) generally have either analogHD or SDI outputs. I would use some USB capture card to get the raw frames inside the R.Pi and then use the GPU to offload H.264 encoding. Dunno about USB bandwidth on R.Pi (USB 2.0 / 480Mbps theorical), but there should be enought horsepower to do the trick.

Another interesting application would be to build some H.264 real-time live transcoding arrays, to re-encode live streams to different qualities on the fly, for a very low TCO.

Cheers,

Pierre-Yves



HD video is over 1 Gbit/s.  USB 2.0 even running at theoretical max is not fast enough.  Unless there is some way to get the uncompressed HD straight into the GPU, I think a 700 MHz CPU is going to be pretty busy trying to handle a 125 Mbyte/s input.
Posts: 2862
Joined: Wed Jan 25, 2012 6:39 pm
by Gert van Loo » Sun Feb 05, 2012 12:48 pm
drgeoff said:

......
HD video is over 1 Gbit/s.  USB 2.0 even running at theoretical max is not fast enough.  Unless there is some way to get the uncompressed HD straight into the GPU, I think a 700 MHz CPU is going to be pretty busy trying to handle a 125 Mbyte/s input.


The following is all theoretical without the GPU SW support but it is interesting anyway:

You can use the CSI port which goes up to ~4Gbits/sec.So that should go a long way of getting data into the board.

There is a limit to the speed at which the GPU H264 core can encode (or decode). But there is nothing which prevents you from splitting up the picture and let multiple Pi's each handle a section.
User avatar
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 2051
Joined: Tue Aug 02, 2011 7:27 am
by pyke369 » Sun Feb 05, 2012 2:03 pm
Gert said:


drgeoff said:


......
HD video is over 1 Gbit/s.  USB 2.0 even running at theoretical max is not fast enough.  Unless there is some way to get the uncompressed HD straight into the GPU, I think a 700 MHz CPU is going to be pretty busy trying to handle a 125 Mbyte/s input.


The following is all theoretical without the GPU SW support but it is interesting anyway:

You can use the CSI port which goes up to ~4Gbits/sec.So that should go a long way of getting data into the board.

There is a limit to the speed at which the GPU H264 core can encode (or decode). But there is nothing which prevents you from splitting up the picture and let multiple Pi's each handle a section.


Thanks for the insights Gert. When you say "CSI port", do you refer to http://elinux.org/Rpi_Low-leve.....IPI_CSI-2? Also is this through FD1, FD6 or P2 on http://elinux.org/File:RpiFront.jpg (maybe I didn't RTFM enough ...)?

Also when you say: "There is a limit to the speed at which the GPU H264 core can encode", what do you mean exactly? I thought putting something like "1080p30 Full HD HP H.264 Video Encode/Decode" in the BCM2835 brochure would mean it could actually encode such a stream in realtime? Maybe I'm misleaded? Would you care to elaborate a bit? Thx.

Last question: since you are "on the inside" at Broadcom, do you know if H.264 encoding is/will be enabled through OpenMAX on the R.Pi?

Thanks for you kind answers,

Pierre-Yves
Posts: 15
Joined: Sat Feb 04, 2012 8:54 pm
by drgeoff » Sun Feb 05, 2012 2:24 pm
Gert said:


drgeoff said:


......
HD video is over 1 Gbit/s.  USB 2.0 even running at theoretical max is not fast enough.  Unless there is some way to get the uncompressed HD straight into the GPU, I think a 700 MHz CPU is going to be pretty busy trying to handle a 125 Mbyte/s input.


The following is all theoretical without the GPU SW support but it is interesting anyway:

You can use the CSI port which goes up to ~4Gbits/sec.So that should go a long way of getting data into the board.

There is a limit to the speed at which the GPU H264 core can encode (or decode). But there is nothing which prevents you from splitting up the picture and let multiple Pi's each handle a section.


Splitting up reduces compression efficiency as it makes it difficult to impossible to use motion compensated prediction from the other parts.  There are also problems "stitching" the compressed bit streams together.  Not impossible, but its certainly more complex than  might first appear from the "there is nothing which prevents you from splitting up the picture and let multiple Pi's each handle a section." sentence.

For the OPs original application I think that using the H.264 encoder already in many HD camcorders would be a more fruitful approach.
Posts: 2862
Joined: Wed Jan 25, 2012 6:39 pm
by Gert van Loo » Sun Feb 05, 2012 2:26 pm
Yes I mean that CSI port. Last thing I read from Liz is that the connectors are not on the production batch. I hope that is not true as post-soldering a trough hole GPIO connector is no issue. But post mounting a CSI SM connector is a whole different can of worms. I think it was FD1. Not sure must check that.

First there is the question of encode/decode. I would rephrase it as "1080p30 Full HD HP H.264 Video Encode or Decode". I don't think it can do both at the same time. I mean that if you want to encode let's say 1080p60 you could use more then one board each doing half a screen.

As to OpenMAX: I am 90% "on the inside" of the hardware. I have no idea what the SW plans are.
User avatar
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 2051
Joined: Tue Aug 02, 2011 7:27 am
by drgeoff » Sun Feb 05, 2012 2:35 pm
Gert said:


As to OpenMAX: I am 90% "on the inside" of the hardware.


Can you reveal any details on the motion estimator in the GPU?  What displacement range?  What search technique - eg full search, tree search, logarithmic etc?  What sub-pixel resolution?  How may reference frames can it search?
Posts: 2862
Joined: Wed Jan 25, 2012 6:39 pm
by jamesh » Sun Feb 05, 2012 4:21 pm
I think most of that stuff will be confidential, but I can try to find out.
Unemployed software engineer currently specialising in camera drivers and frameworks, but can put mind to most embedded tasks. Got a job in N.Cambridge or surroundings? I'm interested!
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 11728
Joined: Sat Jul 30, 2011 7:41 pm
by drgeoff » Sun Feb 05, 2012 4:34 pm
JamesH said:


I think most of that stuff will be confidential, but I can try to find out.



I expect it is confidential but if you don't ask, you don't get :-) .
Posts: 2862
Joined: Wed Jan 25, 2012 6:39 pm
by pyke369 » Sun Feb 05, 2012 5:38 pm
drgeoff said:


Gert said:


drgeoff said:


......
HD video is over 1 Gbit/s.  USB 2.0 even running at theoretical max is not fast enough.  Unless there is some way to get the uncompressed HD straight into the GPU, I think a 700 MHz CPU is going to be pretty busy trying to handle a 125 Mbyte/s input.


The following is all theoretical without the GPU SW support but it is interesting anyway:

You can use the CSI port which goes up to ~4Gbits/sec.So that should go a long way of getting data into the board.

There is a limit to the speed at which the GPU H264 core can encode (or decode). But there is nothing which prevents you from splitting up the picture and let multiple Pi's each handle a section.


Splitting up reduces compression efficiency as it makes it difficult to impossible to use motion compensated prediction from the other parts.  There are also problems "stitching" the compressed bit streams together.  Not impossible, but its certainly more complex than  might first appear from the "there is nothing which prevents you from splitting up the picture and let multiple Pi's each handle a section." sentence.

For the OPs original application I think that using the H.264 encoder already in many HD camcorders would be a more fruitful approach.



The problem is that you wouldn't have hardware/software access to these encoders "already in many HD camcorders". Let's say I'd like to take the raw H.264 NALs and remux/republish them to a live streaming system using HTTP or RTMP, that may prove problematic. Some camcorders are RTSP-enabled and we could transmux the outgoing stream, but it's not the vast majority.

Of course there are other HD H.264 compression chips out there, but not so many: MG2850/3500 from Mobilygen (now Maxim), DM368/6xxx from TI (the DaVinci platform), some media processors from Samsung, and that's almost the end of it.

The problem is that they are not sourcable for the "common people" (like hobbyists and enthusiasts), and that software support is often scarce, sometimes scary and most of the time outdated.

Take the LeopardBoard 368 for instance (DaVinci DM368, https://www.leopardimaging.com/Leopardboard_368.html): it's not that expensive, but still 5 times the price of the R.Pi, and you "only" get a 460Mhz JZ ARM core, and a media processor not anywhere near the VC IV in terms of OpenGL-iness. Plus the TI SDK is kinda outdated (old Linux kernel) and the compressor API is... well, complicated (to say the least) ;-)

The same apply for the Maxim chips: you may try to reverse-engineer the Elgato Turbo 264HD USB stick (http://www.elgato.com/elgato/n.....t1.en.html), like the crusher264 project started to, but it would imply many man/days just to understand the communication protocol and the intrinsics of the H.264 compression settings, without any documentation from the manufacturer of course.

I think the R.Pi is a really interesting product because of its very low TCO (initially targetted at education, but also a real bargain for experimenters/hobbyists). If Broadcom was willing to give access to all those GPU bells and whistles in a quite simple and standardized way (say OpenMAX), that would just be terrific!

Fingers crossed and keep the good work!
Posts: 15
Joined: Sat Feb 04, 2012 8:54 pm
by pyke369 » Sun Feb 05, 2012 5:44 pm
Gert said:


Yes I mean that CSI port. Last thing I read from Liz is that the connectors are not on the production batch. I hope that is not true as post-soldering a trough hole GPIO connector is no issue. But post mounting a CSI SM connector is a whole different can of worms. I think it was FD1. Not sure must check that.

First there is the question of encode/decode. I would rephrase it as "1080p30 Full HD HP H.264 Video Encode or Decode". I don't think it can do both at the same time. I mean that if you want to encode let's say 1080p60 you could use more then one board each doing half a screen.

As to OpenMAX: I am 90% "on the inside" of the hardware. I have no idea what the SW plans are.



Ok, thanks again for the clarification Gert. The application I had in mind wasn't going to do both encoding and decoding simultaneously anyway. As for the "insider" thingy, I was just wondering if you could "poke around" among your colleagues at Broadcom and just get a yes/no kind of answer on that (H.264 GPU encoding supported through OpenMAX IL API in the R.Pi context), as long as you're not breaking any confidentiality rule of course.

Or maybe Liz/Eben/James would have some insights they could share on this?

Thanks again, I can't wait to get my hands on a R.Pi in the coming weeks!

Cheers,

Pierre-Yves
Posts: 15
Joined: Sat Feb 04, 2012 8:54 pm
by jamesh » Sun Feb 05, 2012 5:57 pm
I'm more in a position to ask the encode question. I try and remember and find time on Monday.
Unemployed software engineer currently specialising in camera drivers and frameworks, but can put mind to most embedded tasks. Got a job in N.Cambridge or surroundings? I'm interested!
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 11728
Joined: Sat Jul 30, 2011 7:41 pm
by drgeoff » Sun Feb 05, 2012 6:35 pm
@pyke369

I was suggesting to get inside the camcorder and try to get at the compressed bit stream.  In the past I have successfully purchased a Sony camcorder manual from a Sony Service Centre.  That and many other Sony manuals I have contain very complete schematics and board layout diagrams.
Posts: 2862
Joined: Wed Jan 25, 2012 6:39 pm
by pyke369 » Mon Feb 06, 2012 3:38 pm
JamesH said:


I'm more in a position to ask the encode question. I try and remember and find time on Monday.


Thanks James!
Posts: 15
Joined: Sat Feb 04, 2012 8:54 pm
by jamesh » Tue Feb 07, 2012 5:05 pm
pyke369 said:


JamesH said:


I'm more in a position to ask the encode question. I try and remember and find time on Monday.


Thanks James!


Bad news I'm afraid. The licence only covers H264/MPEG4 decode. To get encode as well would mean more fees. Which is pretty mean of MPEG-LA to be honest.

And the GPU doesn't currently have encode software for the free formats.
Unemployed software engineer currently specialising in camera drivers and frameworks, but can put mind to most embedded tasks. Got a job in N.Cambridge or surroundings? I'm interested!
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 11728
Joined: Sat Jul 30, 2011 7:41 pm
by pyke369 » Tue Feb 07, 2012 5:08 pm
JamesH said:


pyke369 said:


JamesH said:


I'm more in a position to ask the encode question. I try and remember and find time on Monday.


Thanks James!


Bad news I'm afraid. The licence only covers H264/MPEG4 decode. To get encode as well would mean more fees. Which is pretty mean of MPEG-LA to be honest.

And the GPU doesn't currently have encode software for the free formats.


Too bad. But the R.Pi is still an amazing piece of hardware anyway.

Thanks for asking around James.

Cheers,

Pierre-Yves
Posts: 15
Joined: Sat Feb 04, 2012 8:54 pm
by waveform » Thu Feb 09, 2012 12:09 am
Imminent noob question on licensing. Ignore at will.



JamesH said:

Bad news I"m afraid. The licence only covers H264/MPEG4 decode. To get encode as well would mean more fees. Which is pretty mean of MPEG-LA to be honest.



Is there not some kind of charity license that exists that a valiant cause like the Raspberry Pi could avail currently?



Or is it tied to the Broadcom chip somehow? I honestly have no clue on how licensing works in this regard.

Maybe it"s more complicated that I can imagine.
But maybe it shouldn"t be more complicated and some sort of license for charities should exist in the world of video compression and software in general.
Posts: 28
Joined: Wed Nov 30, 2011 6:10 pm
by RaTTuS » Thu Feb 09, 2012 9:22 am
it's more complicated than you can imagine  :-P

licensing always is

it's not Broadcom but MEPEG-LA that are the bad guys here [for various levels of bad]
1QC43qbL5FySu2Pi51vGqKqxy3UiJgukSX - Prosliver FTW
"That's not right, the badgers have moved the goalposts."
User avatar
Posts: 5167
Joined: Tue Nov 29, 2011 11:12 am
Location: North West UK
by drgeoff » Thu Feb 09, 2012 10:48 am
The Foundation may be a charity but all the purchasers of RPs wil not be.
Posts: 2862
Joined: Wed Jan 25, 2012 6:39 pm