jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 18035
Joined: Sat Jul 30, 2011 7:41 pm

Re: GPU Processing API

Tue May 29, 2012 5:27 pm

And yet, by keeping these things secret, the Broadcom Videocore is still the lowest power per performance rating GPU you can get. So there is, indeed, magic sauce.
I'm not saying you can infer the complete technology from the instructions sets, but what you can infer is how the chip is partitioned, what the various cores do and how. So you get a concept, and concepts are a good place to start.

I'm not going to get any further persuading you otherwise, and you won't persuade me either (because I understand the commercial interests, and have them in the company), so I guess it's time to call it a day.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Please direct all questions to the forum, I do not do support via PM.

Pickle
Posts: 65
Joined: Tue Sep 20, 2011 5:09 pm

Re: GPU Processing API

Tue May 29, 2012 8:50 pm

The real problem is when the hardware goes into EOL and all support from broadcom will vanish and users are stuck with binaries that will break with other open source software (i.e kernel and X) that continues to advance.

User avatar
Morgaine
Posts: 141
Joined: Mon Mar 12, 2012 1:13 am

Re: GPU Processing API

Fri Jun 01, 2012 2:53 am

Pickle wrote:The real problem is when the hardware goes into EOL and all support from broadcom will vanish and users are stuck with binaries that will break with other open source software (i.e kernel and X) that continues to advance.
Very good point there, Pickle, which I had not brought up.
Intolerance is a failure of education. Education is predicated on tolerance of the uneducated.

AlArenal
Posts: 141
Joined: Sun Feb 26, 2012 6:58 pm
Location: Germany
Contact: Website

Re: GPU Processing API

Mon Jun 04, 2012 10:25 am

Pickle wrote:The real problem is when the hardware goes into EOL and all support from broadcom will vanish and users are stuck with binaries that will break with other open source software (i.e kernel and X) that continues to advance.
So it all comes down to the real problem being a problem of many ifs that lie an unknown number of years in the future.

Time to sell my Pi stocks then, eh?

User avatar
Morgaine
Posts: 141
Joined: Mon Mar 12, 2012 1:13 am

Re: GPU Processing API

Mon Jun 04, 2012 8:33 pm

AlArenal wrote:So it all comes down to the real problem being a problem of many ifs that lie an unknown number of years in the future.
There's no ifs about it.

Commercially-funded activity always ceases when the commercial benefits from it end. Community activity does not.
Intolerance is a failure of education. Education is predicated on tolerance of the uneducated.

AlArenal
Posts: 141
Joined: Sun Feb 26, 2012 6:58 pm
Location: Germany
Contact: Website

Re: GPU Processing API

Tue Jun 05, 2012 8:30 pm

Who says nothing will be opened up once there is no secret worth keeping anymore and no contracts handcuffing Broadcom? Just be honest and say that you were not the first to know the answer to life, the universe and anything..

I'm more concerned on what I can do right now with the Pi and not what I may not be able to do on top of that in 10 years. Heaven knows what we'll play with then..

james
Posts: 11
Joined: Tue Aug 02, 2011 5:46 am

Re: GPU Processing API

Wed Jun 13, 2012 3:07 am

jamesh wrote:And yet, by keeping these things secret, the Broadcom Videocore is still the lowest power per performance rating GPU you can get. So there is, indeed, magic sauce.
I'm not saying you can infer the complete technology from the instructions sets, but what you can infer is how the chip is partitioned, what the various cores do and how. So you get a concept, and concepts are a good place to start.
http://www.faqs.org/patents/imgfull/20110242113_05

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 18035
Joined: Sat Jul 30, 2011 7:41 pm

Re: GPU Processing API

Wed Jun 13, 2012 10:19 am

Good find. Should give people some idea of how complex these devices are, and how difficult to program unless you work with them every day. Note, this is just a part of a 3D pipeline, which is a subset of the whole GPU.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Please direct all questions to the forum, I do not do support via PM.

User avatar
cheery
Posts: 219
Joined: Wed Jan 25, 2012 9:39 pm

Re: GPU Processing API

Wed Jun 13, 2012 5:37 pm

You call that complex? That graph has very few edges.

james
Posts: 11
Joined: Tue Aug 02, 2011 5:46 am

Re: GPU Processing API

Wed Jun 13, 2012 8:41 pm

cheery wrote:You call that complex? That graph has very few edges.
Heh. He called the device complex, not that high level abstraction was complex.
Eg. a graph with a single node that says "Life" is not complex - but says nothing about the subject matter it models :)

User avatar
Morgaine
Posts: 141
Joined: Mon Mar 12, 2012 1:13 am

Re: GPU Processing API

Sat Jun 23, 2012 11:50 pm

Good news from a competitor: Intel Releases Ivy Bridge Programming Docs Under Creative Commons License.

In addition to helping bring the relevant Intel drivers into mainline kernel and allowing the community to keep them up to date with each new kernel release, this also has the very salutary effect of creating prior art which reduces what closed source companies can patent in the area.
Intolerance is a failure of education. Education is predicated on tolerance of the uneducated.

User avatar
Paethon
Posts: 3
Joined: Sun Jun 24, 2012 4:50 pm

Re: GPU Processing API

Sun Jun 24, 2012 5:25 pm

Hmm. I haven't really looked into the whole issue but as long as you can write shaders you can use the GPU for other things. Encoding your data as textures/vertex data, running shaders on them and reading back the textures/vertex data.

People did that for PC-graphics cards before things like CUDA, Stream and OpenCL existed. Stanford for example created BrookGPU which is basically a GPGPU-compiler and framework that uses OpenGL as the computational backend.

So: It can all be done. It just takes a bit of hacking :)

ajay_m
Posts: 2
Joined: Sun Jul 22, 2012 9:31 pm

Re: GPU Processing API

Tue Aug 07, 2012 6:41 pm

I think the truly regrettable thing here is this

1. The Pi is supposedly an homage to the BBC Micro. This was a machine whose every secret was 'knowable' given enough time and patience. This is what truly motivates gifted hackers (and I mean that term in a positive sense).

2. The Foundation's aim is that the Pi will encourage a new generation of developers.

3. But the Pi has a secret 'no go' area. The GPU is off-limits. So this 'new generation of developers' learns their first lesson - be curious but not TOO curious. Indeed, one enthusiastic group have already been told in these very forums to back off for fear of inciting IP litigation from Broadcom. (they've taken the discussion offline, somewhat surprised at the response, understandably).

Now, I know Broadcom aren't going to release proprietary information. I think it would have been a MUCH better idea to have chosen an SoC *without* a GPU, rather than have this closed-off corner of the system. After all, the BBC Micro worked perfectly well without one.

Telling a young and gifted would-be developer to keep their mitts off stuff because they might get sued is just plain ludicrous. The Foundation should have seen this coming. And it's sad. Who knows what brilliant idea some 15 year old genius might have come up with, given access to the GPU internals. We could have had revolutionary new algorithms for parallel processing, image recognition, audio processing, who knows.... but we won't, because, sadly, all the secrets are under lock and key.

So *that's* the educational lesson, boys and girls. You'd be better off training as a lawyer than a developer. After all, everyone's suing everyone else in the tech world these days.....

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 18035
Joined: Sat Jul 30, 2011 7:41 pm

Re: GPU Processing API

Tue Aug 07, 2012 7:08 pm

ajay_m wrote:I think the truly regrettable thing here is this

1. The Pi is supposedly an homage to the BBC Micro. This was a machine whose every secret was 'knowable' given enough time and patience. This is what truly motivates gifted hackers (and I mean that term in a positive sense).

2. The Foundation's aim is that the Pi will encourage a new generation of developers.

3. But the Pi has a secret 'no go' area. The GPU is off-limits. So this 'new generation of developers' learns their first lesson - be curious but not TOO curious. Indeed, one enthusiastic group have already been told in these very forums to back off for fear of inciting IP litigation from Broadcom. (they've taken the discussion offline, somewhat surprised at the response, understandably).

Now, I know Broadcom aren't going to release proprietary information. I think it would have been a MUCH better idea to have chosen an SoC *without* a GPU, rather than have this closed-off corner of the system. After all, the BBC Micro worked perfectly well without one.

Telling a young and gifted would-be developer to keep their mitts off stuff because they might get sued is just plain ludicrous. The Foundation should have seen this coming. And it's sad. Who knows what brilliant idea some 15 year old genius might have come up with, given access to the GPU internals. We could have had revolutionary new algorithms for parallel processing, image recognition, audio processing, who knows.... but we won't, because, sadly, all the secrets are under lock and key.

So *that's* the educational lesson, boys and girls. You'd be better off training as a lawyer than a developer. After all, everyone's suing everyone else in the tech world these days.....
This has been DONE TO DEATH in previous posts, but...

Can you name a SoC without a GPU that can be got at the price point of the SoC on a Raspi? Or one with an open GPU spec at the same price? And with the same level of technical support? I doubt it, because otherwise the Foundation would have chosen it, and you would be complaining about that one instead.

How many people out there learnt their programming trade without going anywhere near a GPU's internals. Most of them, and by most, I mean 99.9999% of them. Not having any access to the GPU internals doesn't harm the Foundations aims in the slightest, because as a programmable device, it's as good as any other. Also, in my opinion (and I'm sure people will disagree), the GPU is way WAY too complicated for any sort of teaching purposes whatsoever.

And I'm afraid that those 'revolutionary algorithms ' that you talk about could just as easily be developed on a single core desktop as on a GPU, in fact, more easily given the complexity of debugging the GPU (You need customer compilers, debuggers etc and JTAG access). You don't do development like that on the GPU itself, you simulate on a more powerful box, then port over as necessary. And in some cases straight to RTL so your algorithms run on HW not software anyway. I've seen good arguments as to making the GPU code OSS, but this one I afraid doesn't hold much water.

And who said anything about suing people?

The BBC btw, didn't encode/ decode 1080P video, or have a camera ISP, or have real time 3D support. If you don't want them on the raspi, don't use them, and pretend there isn't a GPU. That way, (almost) everything you use is OSS.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Please direct all questions to the forum, I do not do support via PM.

Daverj
Posts: 28
Joined: Tue Mar 06, 2012 2:23 am
Contact: Website

Re: GPU Processing API

Tue Aug 07, 2012 8:01 pm

I think the solution is fairly simple. Raise the price of the Pi by $2.

This would raise an additional $2-$3 million dollars per year. That money could be given to Broadcom to hire a small team of 10-15 programmers at $100k-$150k/year each, plus leave several hundred thousand dollars for administrative costs.

That team could then work on creating GPU libraries that interest the Pi community instead of only creating libraries that make commercial sense to Broadcom.

Everybody wins. Broadcom keeps their information proprietary, but the community gets a set of new libraries that go beyond the OpenGL one. The community doesn't need access to the internals. It needs libraries for specific purposes, such as fast video format decoding, high speed math, etc...

User avatar
jackokring
Posts: 815
Joined: Tue Jul 31, 2012 8:27 am
Location: London, UK
Contact: ICQ

Re: GPU Processing API

Tue Aug 07, 2012 8:09 pm

Just my 2p....

Nice device, OpenGL ES is unlikely to go out of fashion. Even if it does (unlikely as 3D matrix math is not changing, just the way it is done sometimes), there should be some abstract utility API. To this end I propose a single instruction co-processor API.

Pass in an instruction stream of "a=b*(c-d*a*a)" 4 operand instructions, with under and overflow vectoring (allows sgn if tests), and with a, b, c, d and the coproc PC locations having addable delta, with abcd being memory locations, for looping arrays. And any float/shader op should be possible. I'm not sure if this is most efficient, but it does allow all computing to be possible. Not with the greatest of efficiency, but this is for provision of definite future utility.
Pi[NFA]=B256R0USB CL4SD8GB Raspbian Stock.
Pi[Work]=A+256 CL4SD8GB Raspbian Stock.
My favourite constant 1.65056745028

hermanhermitage
Posts: 65
Joined: Sat Jul 07, 2012 11:21 pm
Location: Zero Page

Re: GPU Processing API

Tue Aug 07, 2012 11:58 pm

jackokring wrote:Just my 2p....

Nice device, OpenGL ES is unlikely to go out of fashion. Even if it does (unlikely as 3D matrix math is not changing, just the way it is done sometimes), there should be some abstract utility API. To this end I propose a single instruction co-processor API.
I believe there are two units that would be meaningful to expose - VPU Vector (16 way integer horizontally/vertically addressable multimedia processor) and the vertex/fragment shader QPU units.

For both the obvious choice is either a byte or source representation that gets translated to the target. The "high level" library approach will create a non standard orphan that BRCM are unlikely to leverage on other products.

Given both ARM and Imagination are pushing on with OpenCL, I dont think from a business strategy BRCM will have any choice but to be developing OpenCL to exploit the QPUs - otherwise it further cuts them out of the segment. Whether the 2835 is a suitable target for OpenCL or not only they can answer. It may be there is no valuable them supporting their older product range.

On the VPU Vector unit side of things - to be honest multimedia accelerators are so sensitive to memory access patterns - hiding access behind anything but the raw instruction set is often a waste of time. Their implementation is enough of an outlier I cant think of an existing infrastructure that would target it well.

hermanhermitage
Posts: 65
Joined: Sat Jul 07, 2012 11:21 pm
Location: Zero Page

Re: GPU Processing API

Wed Aug 08, 2012 12:51 am

ajay_m wrote:I think the truly regrettable thing here is this
1. The Pi is supposedly an homage to the BBC Micro. This was a machine whose every secret was 'knowable' given enough time and patience. This is what truly motivates gifted hackers (and I mean that term in a positive sense).
Ajay, the positive thing is - this is still true of the Silicon in the RaspPi. The bonus on the Pi is there is a sense some of it is verboten :). Let us consider the positive things the foundation, Broadcom and the community have achieved so far:

1) Broadcom have provided silicon at a volume + price point that enabled the whole thing to happen. Broadcom have their eyes on the prize of the exploding mobile/multimedia device segment. The fact they have spent effort assisting the foundation for no direct commercial gain is a revelation. Remember Broadcom have even provided some pretty detailed documentation!
2) The foundation has hit their price point, and more importantly the community and the realization that the next generation need to be empowered to make and think is growing. Hopefully we are passing out of the darkness of the consumption mindset that has choked education and the masses since the late 90s.
3) The Raspbian & other distro guys have done a brilliant job. There is movement on the X11 front - yes in an ideal world you would ship a new 'computer' with an optimized desktop - but remember there wasnt the $100-1000k floating around to make this happen from day 1, so it falls to the community. Also remember the momentum behind Linux/X is actually centred around multi GHz cpus with many gigs of memory, so it takes alot of tuning to work well on chipsets with different capabilities.
4) The raw silicon is getting out there into peoples hands. The knowledge to make it sing shall come :). Its quirky and interesting hardware in the same way designs like the C64/NES/SNES/Amiga were with their customized silicon.

Now I think that a Multicore ARM Cortex solution with optional AVR might be a better starting point. But the foundation didnt have their strings on that puppet. So it is what it is, and as our knowledge of the Silicon grows so will the results :)

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 18035
Joined: Sat Jul 30, 2011 7:41 pm

Re: GPU Processing API

Wed Aug 08, 2012 8:40 am

Daverj wrote:I think the solution is fairly simple. Raise the price of the Pi by $2.

This would raise an additional $2-$3 million dollars per year. That money could be given to Broadcom to hire a small team of 10-15 programmers at $100k-$150k/year each, plus leave several hundred thousand dollars for administrative costs.

That team could then work on creating GPU libraries that interest the Pi community instead of only creating libraries that make commercial sense to Broadcom.

Everybody wins. Broadcom keeps their information proprietary, but the community gets a set of new libraries that go beyond the OpenGL one. The community doesn't need access to the internals. It needs libraries for specific purposes, such as fast video format decoding, high speed math, etc...
It's an interesting idea. Main problem would be finding 10-15 engineers with experience in the right areas. There basically aren't any. If there were, Broadcom would already have employed them for other more commercial applications (we are desperate for decent staff and have a lot of work on).

I cannot comment on whether Broadcom would be willing to do it however! They would make MUCH* more money employing those engineers to work on aforementioned commercial projects.

*MUCH = more than most people imagine.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Please direct all questions to the forum, I do not do support via PM.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 18035
Joined: Sat Jul 30, 2011 7:41 pm

Re: GPU Processing API

Wed Aug 08, 2012 8:43 am

jackokring wrote:Just my 2p....

Nice device, OpenGL ES is unlikely to go out of fashion. Even if it does (unlikely as 3D matrix math is not changing, just the way it is done sometimes), there should be some abstract utility API. To this end I propose a single instruction co-processor API.

Pass in an instruction stream of "a=b*(c-d*a*a)" 4 operand instructions, with under and overflow vectoring (allows sgn if tests), and with a, b, c, d and the coproc PC locations having addable delta, with abcd being memory locations, for looping arrays. And any float/shader op should be possible. I'm not sure if this is most efficient, but it does allow all computing to be possible. Not with the greatest of efficiency, but this is for provision of definite future utility.
I did start work on something like this, but work and family meant I had to stop. One of the main problems is passing the data to and from the GPU - that's quite expensive (Arm memory managed address space to Videocore flat address space means big copies).
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Please direct all questions to the forum, I do not do support via PM.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 18035
Joined: Sat Jul 30, 2011 7:41 pm

Re: GPU Processing API

Wed Aug 08, 2012 8:45 am

hermanhermitage wrote: Given both ARM and Imagination are pushing on with OpenCL, I dont think from a business strategy BRCM will have any choice but to be developing OpenCL to exploit the QPUs - otherwise it further cuts them out of the segment. Whether the 2835 is a suitable target for OpenCL or not only they can answer. It may be there is no valuable them supporting their older product range.
It not the 2835 that would be the target, but the Videocore section itself - and since that architecture is on a load of other chips, the work is easily moveable from one device to another as long as the VC remains the same. Same as has happened with Android acceleration - Broadcom are implementing it on higher end devices, but the work is also applicable on the 2835, so we get accelerated Android 'for free'
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Please direct all questions to the forum, I do not do support via PM.

secretagent
Posts: 36
Joined: Wed Mar 07, 2012 10:09 am

Re: GPU Processing API

Wed Aug 08, 2012 10:35 pm

jamesh wrote: And who said anything about suing people?
This is probably the thread in question: http://www.raspberrypi.org/phpBB3//view ... 72&t=13457

And I think the point is not so much about litigation (although anything is possible these days) but about possibly causing deterioration in the relationship between Broadcom and the Raspberry Pi foundation.

I'm a bit confused about this because up until anybody actually started work reverse engineering the GPU code the message seemed to be (paraphrasing and correct me if this is wrong...) "It is not worth the effort to reverse engineer it but if you really want to spend your time on that then go ahead". Then immediately after the work actually began it turned into "let's just hope Broadcom doesn't get mad and that it doesn't stop supporting the foundation". And that is for simply disassembling the code.

So, can you please give a clear and specific answer to the question of whether attempting to reverse engineer the GPU code that is provided for the Pi is something that will negatively impact the Pi foundation?

User avatar
jackokring
Posts: 815
Joined: Tue Jul 31, 2012 8:27 am
Location: London, UK
Contact: ICQ

Re: GPU Processing API

Wed Aug 08, 2012 11:38 pm

jamesh wrote:
jackokring wrote:Just my 2p....

Nice device, OpenGL ES is unlikely to go out of fashion. Even if it does (unlikely as 3D matrix math is not changing, just the way it is done sometimes), there should be some abstract utility API. To this end I propose a single instruction co-processor API.

Pass in an instruction stream of "a=b*(c-d*a*a)" 4 operand instructions, with under and overflow vectoring (allows sgn if tests), and with a, b, c, d and the coproc PC locations having addable delta, with abcd being memory locations, for looping arrays. And any float/shader op should be possible. I'm not sure if this is most efficient, but it does allow all computing to be possible. Not with the greatest of efficiency, but this is for provision of definite future utility.
I did start work on something like this, but work and family meant I had to stop. One of the main problems is passing the data to and from the GPU - that's quite expensive (Arm memory managed address space to Videocore flat address space means big copies).
The thing is as long as overflow and underflow to zero, and a value of one result are vectored via 3 memory address, with the PC as a fourth special address, the one instruction is all that is needed if abcd are memory locations encoding the instruction. At each address containing 32 bits, this addresses a 1KB processing block. So a Harvard extension implies a 2KB block. Or maybe a stall thread pair at 4KB page. This page table lookup shouldn't be too hard. It's all to be done in 32 bit floats.

Cheers Jacko
Pi[NFA]=B256R0USB CL4SD8GB Raspbian Stock.
Pi[Work]=A+256 CL4SD8GB Raspbian Stock.
My favourite constant 1.65056745028

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 18035
Joined: Sat Jul 30, 2011 7:41 pm

Re: GPU Processing API

Thu Aug 09, 2012 9:04 am

secretagent wrote:
jamesh wrote: And who said anything about suing people?
This is probably the thread in question: http://www.raspberrypi.org/phpBB3//view ... 72&t=13457

And I think the point is not so much about litigation (although anything is possible these days) but about possibly causing deterioration in the relationship between Broadcom and the Raspberry Pi foundation.

I'm a bit confused about this because up until anybody actually started work reverse engineering the GPU code the message seemed to be (paraphrasing and correct me if this is wrong...) "It is not worth the effort to reverse engineer it but if you really want to spend your time on that then go ahead". Then immediately after the work actually began it turned into "let's just hope Broadcom doesn't get mad and that it doesn't stop supporting the foundation". And that is for simply disassembling the code.

So, can you please give a clear and specific answer to the question of whether attempting to reverse engineer the GPU code that is provided for the Pi is something that will negatively impact the Pi foundation?
I doubt that a third party attempting to disassemble the GPU code (ie not the Foundation who have access anyway) would affect the Foundation/Broadcom relationship at all. However, if done in the USA anyone doing so would have to be very careful about the DMCA implications. Although that is my opinion. It will be almost impossible to get a clear specific answer unless lawyers get involved. And that would be a bad thing.

My opinion is it's not worth the effort. It's millions of lines of very complex code running on a number of different processor with over 15k registers, so many man years of work with very little to show at the end of it (since the Broadcom code will always be in advance of anything reverse engineered).
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Please direct all questions to the forum, I do not do support via PM.

secretagent
Posts: 36
Joined: Wed Mar 07, 2012 10:09 am

Re: GPU Processing API

Thu Aug 09, 2012 10:50 am

jamesh wrote: I doubt that a third party attempting to disassemble the GPU code (ie not the Foundation who have access anyway) would affect the Foundation/Broadcom relationship at all. However, if done in the USA anyone doing so would have to be very careful about the DMCA implications. Although that is my opinion. It will be almost impossible to get a clear specific answer unless lawyers get involved. And that would be a bad thing.
Yes, I think we can all agree that lawyers getting involved would be a bad thing.

What you say, about not affecting the relationship between Broadcom and the Foundation makes sense but there is some contradiction with what gsh said in the other thread and this is where some of the confusion is coming from. I think that even if it is not possible to get an official Broadcom stance it would be a good idea to have a consensus from the Broadcom members here, as well as the Foundation members. That way we can at least know whether it is ok to discuss this type of work in the forums and also we can reasonably expect that the foundation is not being hurt by this (assuming this is the consensus).

As for the DMCA and other local legislation, I think it is up to the individuals being involved to determine whether it is legal for them to do this.

Return to “C/C++”

Who is online

Users browsing this forum: No registered users and 7 guests