Supercomputing-RPi
Posts: 52
Joined: Tue Jul 29, 2014 5:01 pm
Location: Inside the Foundation's Servers :)

100 RPi supercomputer w/ 20 GPU boards, appreciate for help

Tue Jul 29, 2014 5:38 pm

Hello this is a newbie typing,

I am planning on to bulk order 100 RPis and 20 Nvidia Jetson TK1 Developer Boards.
Armed with the power of Nvidia Kepler GPU SOC on the Jetson TK1 running on Linux,
I should be able to link the TK1s to RPis with DHCP via GigE hubs or routers, right?

I am planning to use this cluster in many data and graphics Intensive Applications.

One example is, fetching data from NASA's small body database (via the internet) and embedding that asteriod or comet info into a 3D realtime solar system visualizer showing in a HDMI ready 4K display.

Another example (except it is data intensive), recording athospheric data such as (pressure, tempreature, and etc.) and displaying that info realtime on a graphical interface.

Could this be possible?
I've heard about hacking the OpenGL ES 2.0 on the Broadcom Videocore IV....
Is it possible for the daemon processes of one RPi be handled by the worker nodes?
Is supercomputing feasible with DHCP?
Has anyone tried out RPi compatibility for Jetson TK1 yet?

Jetson TK1 Link
https://developer.nvidia.com/jetson-tk1#source=pr

User avatar
Jim JKla
Posts: 2218
Joined: Sun Jan 29, 2012 11:15 pm
Location: Newcastle upon Tyne UK

Re: 100 RPi supercomputer w/ 20 GPU boards, appreciate for h

Tue Jul 29, 2014 9:35 pm

There's been some supercomputer research (strictly speaking Multicomputer) notably The Southampton Uni experiment this was the one where they used a nest of the original model "B" RaspberryPi's There's a couple of YouTubes of it they Used Lego to create a framework to hold the RPi's.

There will be a wodge of people who will be happy to tell you this project won't get anywhere but please don't let that stop you trying. ;)
Noob is not derogatory the noob is just the lower end of the noob--geek spectrum being a noob is just your first step towards being an uber-geek ;)

If you find a solution please post it in the wiki the forum dies too quick

User avatar
joan
Posts: 14097
Joined: Thu Jul 05, 2012 5:09 pm
Location: UK

Re: 100 RPi supercomputer w/ 20 GPU boards, appreciate for h

Tue Jul 29, 2014 9:44 pm

Jim JKla wrote:There's been some supercomputer research (strictly speaking Multicomputer) notably The Southampton Uni experiment this was the one where they used a nest of the original model "B" RaspberryPi's There's a couple of YouTubes of it they Used Lego to create a framework to hold the RPi's.

There will be a wodge of people who will be happy to tell you this project won't get anywhere but please don't let that stop you trying. ;)
I'm all for people trying. Progress can arrive that way. But this chap seems to be digging a hole without realising he's lost his shovel.

User avatar
Jim JKla
Posts: 2218
Joined: Sun Jan 29, 2012 11:15 pm
Location: Newcastle upon Tyne UK

Re: 100 RPi supercomputer w/ 20 GPU boards, appreciate for h

Tue Jul 29, 2014 10:05 pm

He's looking to buy 100 RPi shovels I say good luck with the dig. :D
Noob is not derogatory the noob is just the lower end of the noob--geek spectrum being a noob is just your first step towards being an uber-geek ;)

If you find a solution please post it in the wiki the forum dies too quick

Supercomputing-RPi
Posts: 52
Joined: Tue Jul 29, 2014 5:01 pm
Location: Inside the Foundation's Servers :)

Re: 100 RPi supercomputer w/ 20 GPU boards, appreciate for h

Tue Jul 29, 2014 10:07 pm

I wanted to use it for visualizing astronomy, creating modeled paths for asteriod or comets, and use it to create graphical pictures of hydrogen-line emissions from space (I'd better get started on this, it takes 10 years to create one 3D picture).

I would divide my 100 node cluster into 5 parts (labs) for my High School.

Astronomy/Physics
Microbiology
Ecology
Artificial Intelligence
Open Networked Computing

This place where I'm going to put it is going to be Khanaku STEM Observatory.

Khanaku comes from the old South Indian language meaning, calculation.

Wondering what STEM means? It is an acronym (Science-Technology-Engineering-Mathematics) for a new curriculum based in the U.S..

With my plan Khanaku STEM Observatory would be able to push out about 8.4 tflops.
Therefore, beating the ASCI White system by a full tflop.
Last edited by Supercomputing-RPi on Tue Jul 29, 2014 10:50 pm, edited 2 times in total.

Supercomputing-RPi
Posts: 52
Joined: Tue Jul 29, 2014 5:01 pm
Location: Inside the Foundation's Servers :)

Re: 100 RPi supercomputer w/ 20 GPU boards, appreciate for h

Tue Jul 29, 2014 10:44 pm

joan wrote: But this chap seems to be digging a hole without realising he's lost his shovel.
What exactly do you mean by saying thw word "shovel"???
Have I faltered in my thoughts?
Do you think I am deliberately falling towards failure my building my machine?

As a 15 yr old HS or (in UK - Higher Sec. School) student, I am trying to improve my school with easy to use hardware.

User avatar
Jim JKla
Posts: 2218
Joined: Sun Jan 29, 2012 11:15 pm
Location: Newcastle upon Tyne UK

Re: 100 RPi supercomputer w/ 20 GPU boards, appreciate for h

Tue Jul 29, 2014 10:56 pm

@Supercomputing-RPi
Forgive joan there was a bit of a mixed metaphore there.

The line usually is "If you are in a hole stop digging"

or

"If you want a big hole go find a big shovel"

Either way do not let it put you off ;)

@joan Are you crazy let him buy a 100 RPi's If his "Supercomputer" works it will be brilliant if it does not they will have a 100 RPi's out in the world ready to do good.
Noob is not derogatory the noob is just the lower end of the noob--geek spectrum being a noob is just your first step towards being an uber-geek ;)

If you find a solution please post it in the wiki the forum dies too quick

Supercomputing-RPi
Posts: 52
Joined: Tue Jul 29, 2014 5:01 pm
Location: Inside the Foundation's Servers :)

Re: 100 RPi supercomputer w/ 20 GPU boards, appreciate for h

Tue Jul 29, 2014 11:02 pm

While one would agree or disagree with me.
We all know that this can't be done single handedly.
I need your support for this. Not asking money.
I am asking online teamwork, even in the condition that we don't know each other.

User avatar
Jim JKla
Posts: 2218
Joined: Sun Jan 29, 2012 11:15 pm
Location: Newcastle upon Tyne UK

Re: 100 RPi supercomputer w/ 20 GPU boards, appreciate for h

Tue Jul 29, 2014 11:10 pm

Yep there's a big team here. All I am saying is check out the Southampton stuff first, you are walking down a path where they have stommped down some of the long grass ahead of you. ;)
Noob is not derogatory the noob is just the lower end of the noob--geek spectrum being a noob is just your first step towards being an uber-geek ;)

If you find a solution please post it in the wiki the forum dies too quick

User avatar
DougieLawson
Posts: 35602
Joined: Sun Jun 16, 2013 11:19 pm
Location: Basingstoke, UK
Contact: Website Twitter

Re: 100 RPi supercomputer w/ 20 GPU boards, appreciate for h

Tue Jul 29, 2014 11:20 pm

What makes you think that a collection of small, slow 700MHz machines is going to make a powerful cluster? How much processing power are you going to lose with data transfer and processor signalling (you don't get a loosely coupled cluster for free)?

If the RPi were an ideal machine for clustered computing do you not think that two years after its introduction we'd be seeing lots of folks running hundreds of RPis in loosely coupled clusters? Have you seen anyone who's successfully run a project with a hundred machines running in parallel? The workload has to be carefully engineered to run in small parallel chunks on a cluster.

The only things in favour of using Raspberry Pis in this way is the extremely low power requirement. But the things against it are the SMSC9514 combined LAN/USB controller and the lack of any high speed interface to your data. You're going to push all of your data through USB 2.0 or ethernet so that SMSC9514 will be the bottleneck no matter where you place your data. Keeping it on the local SDCard won't work when the result set generated on each clustered machine need to be aggregated to form the complete results set.

The thing that marks the high speed tightly coupled mainframes (and supercomputers) out from the normal range of general purpose computers is the high speed fibre optic links between processors (for multi-processing signalling), parallel fibre optic links to hard disks (30K I/O operations (moving 48K of data per operation) per second is normal) and high speed shared memory systems so you don't have to write data to disk on one processor to read it on another.
Note: Having anything humorous in your signature is completely banned on this forum. Wear a tin-foil hat and you'll get a ban.

Any DMs sent on Twitter will be answered next month.

This is a doctor free zone.

Supercomputing-RPi
Posts: 52
Joined: Tue Jul 29, 2014 5:01 pm
Location: Inside the Foundation's Servers :)

Re: 100 RPi supercomputer w/ 20 GPU boards, appreciate for h

Tue Jul 29, 2014 11:25 pm

I have seen the University of Southampton stuff...
But I came here to first verify if the idea is even feasible.

I have lots of questions regarding the hardware and software capabilities.
I am an active participant in the codecademy.com and I am learning Python for uses in RPi terminals.

Supercomputing-RPi
Posts: 52
Joined: Tue Jul 29, 2014 5:01 pm
Location: Inside the Foundation's Servers :)

Re: 100 RPi supercomputer w/ 20 GPU boards, appreciate for h

Tue Jul 29, 2014 11:41 pm

@DougieLawson
The main piece of hardware used for GPU intensive things is going to be the Nvidia Jetson TK1.
It has a SOC GPU which is considerably faster than RPi. As I mentioned above, I am going use 20 of them.
20 TK1s each TK1 has 300 Gflops (that is about 13 RPis rounded [half RPis don't exist anyways]).
20 TK1s equate to the GPU computing power of 250 RPis.
I am also going to use SSDs for all the TK1s and the main node, making it's speed very fast.

User avatar
DougieLawson
Posts: 35602
Joined: Sun Jun 16, 2013 11:19 pm
Location: Basingstoke, UK
Contact: Website Twitter

Re: 100 RPi supercomputer w/ 20 GPU boards, appreciate for h

Wed Jul 30, 2014 12:11 am

How are you going to connect your SSDs to your GPUs? You ARE going to get a 5MB/s bottleneck as soon as you hit the SMSC9514 on a Raspberry Pi.
Note: Having anything humorous in your signature is completely banned on this forum. Wear a tin-foil hat and you'll get a ban.

Any DMs sent on Twitter will be answered next month.

This is a doctor free zone.

Supercomputing-RPi
Posts: 52
Joined: Tue Jul 29, 2014 5:01 pm
Location: Inside the Foundation's Servers :)

Re: 100 RPi supercomputer w/ 20 GPU boards, appreciate for h

Wed Jul 30, 2014 12:31 am

I am going to use the fairywren mini itx board which has a 2.5mm hard drive expansion.

http://www.eetimes.com/document.asp?doc ... e_number=5

The Jetson TK1 has it's own Harddrive slot.

WebPi
Posts: 262
Joined: Wed Apr 10, 2013 6:47 pm
Location: Birmingham, UK
Contact: Website

Re: 100 RPi supercomputer w/ 20 GPU boards, appreciate for h

Wed Jul 30, 2014 2:25 am

I think Dougie's right. This is a very ambitious project, and it would be a shame to buy 100 Pis and then find that it's too complicated or won't work.

You would probably need to write some complex code to send processing tasks from the RPi nodes to the GPUs, and it wouldn't be very efficient because of the ethernet limitations that Dougie mentioned. Professional engineers would find this challenging.

Instead of trying to build an image processing super computer, you could just build a simple cluster for educational/prototyping purposes. If you can build a cluster with RPi nodes, then you can do it with a bunch of PCs with gigabit ethernet. Then you'll have a cluster that has some real power.
raspberrywebserver.com - Raspberry Pi tutorials
LinuxWebServers.net - Linux Web Server tutorials and examples
pyplate.com - Python web publishing tool

Supercomputing-RPi
Posts: 52
Joined: Tue Jul 29, 2014 5:01 pm
Location: Inside the Foundation's Servers :)

Re: 100 RPi supercomputer w/ 20 GPU boards, appreciate for h

Wed Jul 30, 2014 2:38 am

Let's attack this problem strategically, shall we?

So ethernet restrictions.... Preferred Solution: GigE
Memory Restrictions... Preferred Solution: 1 Gig Ram or more.. (Anonymously suggested)
Latency Restrictions... Preferred Solution: Faster SOC chip?
GPU command restriction... Preferred Solution: What if the GPU has compatibility with Linux or Debian? (Jetson TK1 runs on Linux)

Would the above preferred solutions work out??

WebPi
Posts: 262
Joined: Wed Apr 10, 2013 6:47 pm
Location: Birmingham, UK
Contact: Website

Re: 100 RPi supercomputer w/ 20 GPU boards, appreciate for h

Wed Jul 30, 2014 3:38 am

There are a lot of different technical problems to solve in your opening post. Instead of trying to build something so complicated all in one go, it might be best to build a prototype cluster first. Interfacing with the GPUs is another problem entirely, and probably one that requires quite a lot of experience in image processing.

For a prototype, you can use RPis. For a working super computer, you need to use much more powerful (and more expensive) hardware.
raspberrywebserver.com - Raspberry Pi tutorials
LinuxWebServers.net - Linux Web Server tutorials and examples
pyplate.com - Python web publishing tool

ghans
Posts: 7868
Joined: Mon Dec 12, 2011 8:30 pm
Location: Germany

Re: 100 RPi supercomputer w/ 20 GPU boards, appreciate for h

Wed Jul 30, 2014 4:14 am

Let's attack this problem strategically, shall we?
So ethernet restrictions.... Preferred Solution: GigE
Memory Restrictions... Preferred Solution: 1 Gig Ram or more.. (Anonymously suggested)
Latency Restrictions... Preferred Solution: Faster SOC chip?
GPU command restriction... Preferred Solution: What if the GPU has compatibility with Linux or Debian? (Jetson TK1 runs on Linux)

So in short , you should not use a Raspberry Pi , but PCs
instead , because the Pi has fixed specs and is not
upgradeable in any way.

A setup based on Desktop PCs + Desktop GPUs will propably beat
a "Pi Supercomputer" on the FLOPs/Watt and FLOPs/$ ratio at
the same time.

You're not getting Gigabit Ethernet with a board based on the
BCM2835 anyway , because it's fastest open interface is USB 2.0
with a very complex and heavyweight Linux driver.

ghans
• Don't like the board ? Missing features ? Change to the prosilver theme ! You can find it in your settings.
• Don't like to search the forum BEFORE posting 'cos it's useless ? Try googling : yoursearchtermshere site:raspberrypi.org

Supercomputing-RPi
Posts: 52
Joined: Tue Jul 29, 2014 5:01 pm
Location: Inside the Foundation's Servers :)

Re: 100 RPi supercomputer w/ 20 GPU boards, appreciate for h

Wed Jul 30, 2014 4:37 am

How about the HummingBoard i2ex with Raspbian as the OS.
It has an mSATA for the SSD.
SOC GPU onboard sounds good in terms with technicality.
It has 1 GigEthernet Port...
It has 1 Gig RAM onboard.

Does the above qualify as perfect alternative for supercomputing?

Sorry anonymous user, I can't use the bannana PI due to it widespread sales on aliexpress.com, amazon.com, and ebay.com

Supercomputing-RPi
Posts: 52
Joined: Tue Jul 29, 2014 5:01 pm
Location: Inside the Foundation's Servers :)

Re: 100 RPi supercomputer w/ 20 GPU boards, appreciate for h

Wed Jul 30, 2014 4:45 am

Would they ship from the OEM factory direct?

ghans
Posts: 7868
Joined: Mon Dec 12, 2011 8:30 pm
Location: Germany

Re: 100 RPi supercomputer w/ 20 GPU boards, appreciate for h

Wed Jul 30, 2014 4:46 am

Do you already have any idea on how to tackle the software
aspect of this project ? Lastly that is going to decide which
paltform you need.

ghans
• Don't like the board ? Missing features ? Change to the prosilver theme ! You can find it in your settings.
• Don't like to search the forum BEFORE posting 'cos it's useless ? Try googling : yoursearchtermshere site:raspberrypi.org

Supercomputing-RPi
Posts: 52
Joined: Tue Jul 29, 2014 5:01 pm
Location: Inside the Foundation's Servers :)

Re: 100 RPi supercomputer w/ 20 GPU boards, appreciate for h

Wed Jul 30, 2014 5:01 am

Rough Ideas / Plans
would provide if needed for helping me.

User avatar
Jim JKla
Posts: 2218
Joined: Sun Jan 29, 2012 11:15 pm
Location: Newcastle upon Tyne UK

Re: 100 RPi supercomputer w/ 20 GPU boards, appreciate for h

Wed Jul 30, 2014 7:15 am

For all your suggestions you seem blind to the fact that the OP wants desperately to build a "Supercomputer" with RaspberryPi units and to his (I am assuming it's a he forgive me if I am wrong) credit is choosing to hammer out the problems with using the RPi as a desired approach in the cauldron that is this forum.

I don't want to see this project fail but I would really like to see it atempted. ;)

Look at the OP's chosen forum name, here I see a desperate desire to be able to say I tried this suceed or fail.

@Supercomputing-RPi (Please correct me if I am wrong in this view or any of my other asumptions)
Noob is not derogatory the noob is just the lower end of the noob--geek spectrum being a noob is just your first step towards being an uber-geek ;)

If you find a solution please post it in the wiki the forum dies too quick

Supercomputing-RPi
Posts: 52
Joined: Tue Jul 29, 2014 5:01 pm
Location: Inside the Foundation's Servers :)

Re: 100 RPi supercomputer w/ 20 GPU boards, appreciate for h

Wed Jul 30, 2014 7:31 am

I would like to see this project being accomplished by the hardware or software help of Raspberry Pi.
Raspberry Pi seems to be the easiest platform to work with.
Even if the performance degrades...
I am still choosing to not completely eradicate RPi out of my focus.
I will still experiment with the RPi on my cluster.
For actual proven performance, I would have to choose an alternative.
I am planning to on to make a flowchart for schematics.

Banana PI QA's
How long does it take for it arrive from Shenzen, CN?
How many GFLOPS does one unit output?

Has people seen Nvidia's Jetson TK1 platform?
I am planning on to use that board as an external GPU.
Check it out on the below link....

https://developer.nvidia.com/jetson-tk1#source=pr

Please keep the conversation going!!!

User avatar
RaTTuS
Posts: 10381
Joined: Tue Nov 29, 2011 11:12 am
Location: North West UK

Re: 100 RPi supercomputer w/ 20 GPU boards, appreciate for h

Wed Jul 30, 2014 7:32 am

start off with 2 RPI's
get that working in how you want to do things
then add more RPI's
do some calculations
add a couple of PC's
do some calculations
add a couple of jetsons [or whatever is current at the time]
do some more calculations
see how it runs
start small aim high
How To ask Questions :- http://www.catb.org/esr/faqs/smart-questions.html
WARNING - some parts of this post may be erroneous YMMV

1QC43qbL5FySu2Pi51vGqKqxy3UiJgukSX
Covfefe

Return to “Off topic discussion”