Hardware-assisted H.264 video encoding

[mccp]

First of all I have no desire to distract anyone from the task of releasing the boards that are about to be made so please ignore the following if it is distracting .

We report sales of MPEG-2 and H.264 encoders/decoders to MPEG-LA (who may or may not be evil, but they are certainly persistent). I believe that we are obliged to pay $2.00 for an MPEG-2 encoder and the same for a decoder (i.e. $4.00 for a transcoder).

I can't find the agreements at the moment, but I believe that we only report the quantities of H.264 encoder/decoders as we are below the annual sales threshold that triggers royalty payments.

I have just looked at the sample agreement on the MPEG-LA site and it seems that they consider that a unit is: "(a decoder, encoder, or product consisting of one decoder and one encoder = “unit”)".

License fees are 0-100,000 units per annum $0.00, 100,000 to 5,000,000 per annum $0.20 and >5m per annum is $0.10 per unit.

So it may be that MPEG-LA patent portfolio license is not the culprit for the lack of H.264 support.

We found that the fees for AAC were much more painful than H.264 - this is managed by Via Licensing. They charge $0.98 for 1-500,000 units for hardware encoders/decoders or $0.48 for SW decoders & $0.98 for SW encoders. Plus there's a $15K up front payment required.

I would very much like to see an encoding variant of the Raspberry Pi as I can see many useful applications, but I can easily understand that this is not going to be at the forefront of the organisation's plans if it adds significant costs or administration. I hope at least that it will be added to a wishlist for future releases to be considered when there is time.

I also hope that the bleating about evil licensors and charity status, etc. will stop as it must be clear that the Raspberry Pi organisation is selling products to end-users that are required to be licensed by the laws of various countries. Unless you've had the pleasure of dealing with the lawyers employed by organisations such as MPEG-LA/VIALicensing/Fraunhofer your opinion on the subject may not be terribly relevant .

9+9 (personal best!)

[waveform]

Hey mccp!

Thanks a lot for sharing your post and giving some insight.

It would still be great though if there was some discounted rate offered for the educational Pi units.

[pyke369]

mccp said:

I have just looked at the sample agreement on the MPEG-LA site and it seems that they consider that a unit is: "(a decoder, encoder, or product consisting of one decoder and one encoder = “unit”)".
License fees are 0-100,000 units per annum $0.00, 100,000 to 5,000,000 per annum $0.20 and >5m per annum is $0.10 per unit.


According to this statement, it seems the licence paid for the decoder also covers the encoding portion of the "unit". So why wouldn't Broadcom unlock the feature as part of the initial release for the R.Pi? Or am I reading this wrong?

Cheers,

Pierre-Yves

[jamesh]

No, I checked this - we only pay for a decode licence. Encode will add more cost.

I also agree with the posters assessment of AAC- that is really very expensive for such a crappy little codec - much more than H264. Which is why it is not include on the GPU and users will need to install Arm side AAC decode.

[Gert van Loo]

As to a discount for the Pi: I don't think that would be feasible even if they wanted as the boards do not all go to charity/education. It has been clearly stated that anybody can buy the boards and sell them on. So giving a discount or free licenses would cause false competition.

[pyke369]

JamesH said:


No, I checked this - we only pay for a decode licence. Encode will add more cost.



Bummer

According to this http://www.mpegla.com/main/pro.....ummary.pdf (probably what mccp was referring to), there is nothing to pay under 100k units / year. Above, it's an extra $0.20 but it opens interesting options for video-conferencing and video streaming.

Anyway, when you say "we only pay", who's "we"? The R.Pi foundation? Or Broadcom?

Is the decode licensing price already included by Broadcom when it sells the BCM2835 to the R.Pi foundation?

(sorry for being insistent, I just want to know the gory details )

[jamesh]

The foundation pays the licence fees. Broadcom just provides the GPU software to match the licences purchased.

[Minthos]

So I assume the added cost is for the software from Broadcom, not the licence from MPEG-LA?

Would it be possible to add hardware encoding in a C version of the device? Perhaps with other additional features as well?

[jamesh]

No, there is an addition licence cost that needs to be paid to MPEG-LA. If that cost is paid, Broadcom will provide the appropriate encoder software on the GPU blob.

I'm not sure what you mean by C version.

[mole125]

Minthos said:


So I assume the added cost is for the software from Broadcom, not the licence from MPEG-LA?

Would it be possible to add hardware encoding in a C version of the device? Perhaps with other additional features as well?


No, from what has been said in numerous other posts, the added cost is for the licences from MPEG-LA. If the the foundation can prove to Broadcom they have got the necessary patent license then Broadcom can provide the keys to allow the other hardware decoders/encoders to be used.  The challenge for the foundation is

a) the cost and effort to negotiate and administer the license

b) convincing the MPEG-LA that they have sufficient technical measures to stop them from being abused - ie people not taking the model C bootcode.bin and using it on a model B without paying the license.

[pyke369]

This is so depressing and sad

It would have been a lovely feature on the Pi (I was dreaming of a Pi H.264 encoding cluster, but I'm biased toward video encoding ... ^_^)

[jbeale]

How about Ogg Theora? http://en.wikipedia.org/wiki/Theora

Three magic words: "no licensing fees" (encode or decode).

Theora has seen limited use due to a number of reasons, probably much due to H.264 being already widely adopted. But if the RasPi supported Theora encode and decode, things might change.

"…the differences in quality, bitrate and file size between a YouTube H.264 video and a transcoded Ogg video file are negligible."
from http://en.wikipedia.org/wiki/Theora which is referencing http://www.freesoftwaremagazin.....la_firefox

[Max]

jbeale said:


How about Ogg Theora? http://en.wikipedia.org/wiki/Theora


Or Google's WebM? http://en.wikipedia.org/wiki/Webm

Advantage being that the encoded files could be played back by Firefox, Chrome and Opera without additional software. (See browser list at Youtube's HTML 5 beta page: http://www.youtube.com/html5 )

[jamesh]

Max said:


jbeale said:


How about Ogg Theora? http://en.wikipedia.org/wiki/Theora


Or Google's WebM? http://en.wikipedia.org/wiki/Webm

Advantage being that the encoded files could be played back by Firefox, Chrome and Opera without additional software. (See browser list at Youtube's HTML 5 beta page: http://www.youtube.com/html5 )


There is an implementation of WebM on the GPU available, but it's software only i.e. just uses the vector core rather than using the HW acceleration blocks which are tuned to H264. So its only up to about 20fps or something. Can't remember the exact figure.

Not sure about Ogg Theora though.

[mr_sunshine]

mole125 said:


a) the cost and effort to negotiate and administer the license

b) convincing the MPEG-LA that they have sufficient technical measures to stop them from being abused - ie people not taking the model C bootcode.bin and using it on a model B without paying the license.


The Pi clearly needs an H.264 encoder since it opens up so many new uses - video conferencing, video editing and so on - and also because the HW is already capable of it.

But not all users would need it - perhaps it would be possible to offer it as an add-on SW component including the license?

[mole125]

mr_sunshine said:


mole125 said:


a) the cost and effort to negotiate and administer the license

b) convincing the MPEG-LA that they have sufficient technical measures to stop them from being abused - ie people not taking the model C bootcode.bin and using it on a model B without paying the license.


The Pi clearly needs an H.264 encoder since it opens up so many new uses - video conferencing, video editing and so on - and also because the HW is already capable of it.

But not all users would need it - perhaps it would be possible to offer it as an add-on SW component including the license?


There's a difference between need and want unfortunately. For the Foundation to meet there primary aims with the RPi they don't need H264, but I agree we all want it and it opens up so many opportunities. The foundation have however said they plan to provide a hi-res camera and that will need an encoder so I think there is a very good chance that the foundation will find a way to solve the challenges above, hopefully that way will also allow none video encoding use (which probably isn't guaranteed depending how much of a pain MPEG LA are).

[errikos]

So, now that the hardware H.264 encoder is feasible, where should we search to make something?!

[kalehrl]

Hardware encoding is very bad compared to software.
If you expect to be able to make good rips using your raspi, I believe you'll be disappointed.

[ghans]

We'll see.
First somebody has to wrestle himself through OpenMAX for anything to happen.

ghans

[virtualas]

Maybe some updates on this topic related to video streaming? I am looking for same too! And founding just open topics... :/

[ghans]

Nobody seems to be willing to donate his/her time to this ...
The feature is freely available for C programmers now , they don't
even have to unlock it ... but nobody seems to like the OpenMAX API.
VLC says "the implementation is only 99.999 % conforming to the standard , that's not enough" ,
ffmpeg enthusiasts chickened out when they started on OpenMAX (i guess here , because i've not
heard anything since the request for docs ) and gstreamer devs seem very wary of optimization work.

Sad , I wish i already knew how to program properly.

ghans

[jamesh]

OpenMAX is quite unpleasant to work with. Its a great idea, that really hasn't worked out in practice. Probably explains the lack of progress.

What are people actually looking for from the encode ability? Once the camera module is released we plan to have encode working from that, but what else is requried?

[dom]

OpenMAX is quite unpleasant to work with. Its a great idea, that really hasn't worked out in practice. Probably explains the lack of progress.

It is not *that* hard.
kulve was first to get h264 encode running through openMAX and it's only a couple of pages of C code. See hello_encode demo based off his code:
https://github.com/raspberrypi/firmware ... e/encode.c

[jamesh]

OpenMAX is quite unpleasant to work with. Its a great idea, that really hasn't worked out in practice. Probably explains the lack of progress.

It is not *that* hard.
kulve was first to get h264 encode running through openMAX and it's only a couple of pages of C code. See hello_encode demo based off his code:
https://github.com/raspberrypi/firmware ... e/encode.c

Ah, but you are smarter than I am! Main problem I've found is you know what needs to be done, and there doesn't appear to be an order required, but there is...and it doesn't tell you want that order needs to be...and the error messages are 'interesting'!

That said, there is example code as linked! So I don't have to think about it!

[Santa77]

If someone is interested in, small patch is here:

https://github.com/raspberrypi/firmware/issues/114