Clustering

[abishur]

Okay first off, if anyone does use the R-Pi in a cluster I think it either needs to be called a druplet (druplets are the tiny bumps that make up a raspberry) or a bramble (which is actually a cluster of raspberries is called). Just a thought that makes me laugh.

The second (and actual point of my post) is how effective would a bramble be? I admit that the power requirements (electricity cost) and footprint of a traditional cluster has kept me from getting to play with them, but there's a couple things I think about when it comes to a R-Pi cluster.

1) CPU/Memory Power - The R-Pi isn't exactly built to be a powerhouse. I get the idea of a cluster is basically to harness a group of computers, distribute the load of processing but when you start off with a low resource device, it takes a lot of them to see the benefits you would see by clustering "normal" PCs

2) Power/Networking - Once you get enough of these devices together to get a really good cluster (bramble ) going you're going to need a significant amount of networking ports and power outlets. Once you get all that up and going do you think the cost (especially of the hub) will outweigh the benefits of low cost R-Pi. A lot of that might change if PoE is provided, then you'd just need a single PoE capable hub and you'd have both fronts covered.

3) OS support - I know we definitely have support for a regular Ubuntu install on the ARM11 but do you think there's a cluster ready OS compatible with the R-Pi

Finally, if we stepped back from the uber computer concept, do you think you could cluster say 4 or 5 of them together and get better performance then a cheap (under $300) desktop? Or am I just completely misunderstanding that aspect of a cluster?

[ukscone]

Quote from abishur on August 3, 2011, 00:00
Okay first off, if anyone does use the R-Pi in a cluster I think it either needs to be called a druplet (druplets are the tiny bumps that make up a raspberry) or a bramble (which is actually a cluster of raspberries is called). Just a thought that makes me laugh.

The second (and actual point of my post) is how effective would a bramble be? I admit that the power requirements (electricity cost) and footprint of a traditional cluster has kept me from getting to play with them, but there's a couple things I think about when it comes to a R-Pi cluster.

1) CPU/Memory Power - The R-Pi isn't exactly built to be a powerhouse. I get the idea of a cluster is basically to harness a group of computers, distribute the load of processing but when you start off with a low resource device, it takes a lot of them to see the benefits you would see by clustering "normal" PCs

2) Power/Networking - Once you get enough of these devices together to get a really good cluster (bramble ) going you're going to need a significant amount of networking ports and power outlets. Once you get all that up and going do you think the cost (especially of the hub) will outweigh the benefits of low cost R-Pi. A lot of that might change if PoE is provided, then you'd just need a single PoE capable hub and you'd have both fronts covered.

3) OS support - I know we definitely have support for a regular Ubuntu install on the ARM11 but do you think there's a cluster ready OS compatible with the R-Pi

Finally, if we stepped back from the uber computer concept, do you think you could cluster say 4 or 5 of them together and get better performance then a cheap (under $300) desktop? Or am I just completely misunderstanding that aspect of a cluster?
Hunter Davis ( http://hunterdavis.com ) has written a nice series of articles on using pogoplug's as a distributed compile cluster for cross compiling x86-64. I think the same principles apply http://www.discursivelabs.com/?p=610
--
I don't want to advertise for him on this site but if you check any of your favourite ebook stores he has turned all his articles on this into an ebook for a few bucks and if you are a poor pennyless student he'll send you a copy for free if you ask nicely

[amiga65]

Just to throw My totally uneducated opinion in the pool here.
My amd 965 ocd to 3.8 ghz with four cores round down to 3.5ghz = 14 ghz, that would take 20 R-pi's
to =14 ghz now I'm sure there going to take a speed hit due to the interconnects, and running a 32 bit os vs. a 64 bit. So may need to add a few more to make up for that. Thhen comes the server, I don't think the Rpi can handle that many nodes effectively, so gonna need a $300 desktop for the server. Now don't quote Me on this as I know nothing about clusters the above just sounded kinda sorta logical in an unintelligent way. but think I'm gonna try building a small one anyways.

[ShiftPlusOne]

I think anyone who is buying these to 'cluster' them in order to get anything useful out of it is going to be disappointed. You're better off using the money to upgrade your desktop.

If they're doing it just because they can... well that's different.

[MDC]

Bramble or Druplet, if I had to choose it would be a bramble.

Though I have a couple of names of my own for a cluster, I still like Rasporgy Pi, even though I thought it up as a joke initially.

The other name for the R-Pi I was thinking of was a stomach ache, though I think if you went to your friends and said I made a stomach ache upstairs they may get the wrong impression and think you have either vomited upstairs or crapped yourself 8O .

[Lob0426]

A cluster is a series of processing units that work toward one end as a task, wouldn't a BRAMBLE be a set of computing units that are assembled together as a network unit to carry out dissimilar tasks forwarded to the centralized unit? Hardware mutlitasking. Actually using the Ras.Pi units in my description of a bramble would be more profitable to performance than using the Ras.Pi as a cluster. Another consideration is the cluster master computer needs to be a step above the clustered units to handle the overhead or it will be the bottleneck in said cluster. And the Switch to handle the cluster will probably cost more than the whole cluster! I am not knocking down the cluster idea it just should have a purpose in mind! A definite goal would be to learn how to construct a linux cluster!

[duese]

Im absolutley new to this very interesting Topic. As far as I understood there could be two different ways to combine the R-PIs. First you have a Master System which delivers Tasks to Hosts. But i dont really get the second method. is it possible to combine them and they behave as if it would be one System? How should that work an is there something similar out there?

[Lob0426]

A cluster is a series of computing devices controlled by a master computer, they usually act in parallel to perform a computing task. Bramble was a play on the post by Abishur's reference to bramble. A network of computers can be used to connect a set of computing devices that are running separate tasks that are then forwarded to a computer that acts as a user interface. Example you have one raspi acting as a GPS, another controlling a camera for streaming and another using I2C sensor for environmental awareness. The user has a fourth device to display the information from the other devices. The MIT wearables site used a "body network" comprised of an Ethernet hub, USB network hubs and a I2C hubs to carry out these functions.

[dogboi]

I was seriously considering the idea of building a cluster with a bunch of home built AMD machines, but this would certainly be a cheaper option for me to play with clustering. I'm more interested in the "how to program for clusters" rather than the "let's get lots of power from clustering", so this might suit my needs just fine. If nothing else, I can buy a few and test it out, and if it doesn't work out too well, I'm sure I can find other uses for them (XMBC box? Box to experiment with low memory programming? etc, etc).

Edit: And I do have a machine lying around I could use for the "head" machine. So no cost there. I'd need to buy a reliable switch is all.

[Lob0426]

Figuring out clustering is definitely a good use for the RasPi. They were designed to be learning tools. You would need a reliable switch no matter which way you set it up. Try to get one that has at least one Gigabit port to hook to your central machine if it supports it.

[robert.keizer]

A while ago I did a really hacked together presentation on distributed operating systems for a local unix users group.

Because of the ARM architecture rather than x86, some modifications to kerrighed or openssi would need to be made. That being said, they aren't impossible changes, as the source is very much available and only a handful of files are architecture specific.

http://muug.mb.ca/blog/meeting.....g-systems/

I plan to purchase a few R.Pi devices and shove them into a laptop form factor. Most likely converting an old thinkpad into a modular system where I can add more R.Pi's at will. Given the size, my initial dimensions come out that I could fit 7-8 devices into an existing Z60m case.

[Lob0426]

Laptop cluster! Now that is an idea I can hold with. lol

[iAreNewb]

Not sure if you can fit them into a laptop case safely, since they'd need something to hold them in place. They'd either bang around inside or be squished, depending on how much space there is. Plus, you'd have to fit a switch in there.

As for a cluster, they are *not* for general-purpose computing like playing games. This is because most software is poorly optimized for multiple processor usage; they can barely take advantage of multiple cores on a single die, let alone multiple distinct CPUs over Ethernet.

However, if this is geared toward developers looking for parallel/distributed computing, I can't see why it's not possible. Even if you can't get them to run a cluster OS (i.e. in which all processing resources are utilized by the OS at a native level [although I'm not sure if this allows unoptimized software to utilize the multi-CPUs]), you might fool around with a software framework. It's possible to make one yourself, or use an open source one, that allows you to send threads or thread info over Ethernet.

To use a Java (7) example, assuming that you're running the same program across your 'bramble', you could send a ForkJoinTask object (or whatever objects necessary for its construction) through a network stream and then grab the return value of its invocation through the same stream.

[Lob0426]

Personally I like the idea of each seperate Device handling a specific task. One device handling Locations services such as GPS/Inertial Guidance. Another dealing with the camera for video/pictures. Another to handle external requests such as network and environmental awareness sensors etc. And yet another to handle storage like the HDD and flash drives. One of the reasons that todays systems have to be so powerful is that they have to handle all of these tasks themselves. This requires more software to be ran resident in memory in the form of services, drivers etc. By reducing the tasks each device has to deal with you reduce the load on resources each device needs. Each device can be "tweaked" to do its job efficiently. The User interface on the primary device can devote itself to just showing you what the other devices have going on. Remember this is only an example as I do not foresee you needing a RasPi for each of those tasks. One or two handling mundane routine plus One as the interface would probably do it. There is also the power for these devices to be considered into the equation.

In the MIT wearables design they actually had three different cores. They had a Computing core board, Video core board and in earlier versions their wireless was another core. All of these were connected through a body bus that contained Ethernet, USB and an I2C bus.

If only there was a good inexpensive HUD available out there I might give it a try. MuVee went out of business which was a possibility as one was used to make a display for a Beagle Board. Everything else is thousands of dollars.

[frafra]

Kerrighed running over Raspberry pc? Sound cool

[Andre_P]

Hmmm, incredibly out of touch but a couple of things come to mind, the first is Beowulf and the second is what happened to the Linux that was aimed at the PS3, A VERY NAIVE thought I recognise but thrown in to disturb someones mind to work out why it can't be done and possibly work out that it can .

[Svartalf]

The CPU muscle would be a bit of a limitation (Though if you could leverage GPGPU, you'd be on to a bit of something...). The other thing would be the scale-out possible due to the networking that IS available to the R-Pi.

As a research project, I'd say go for it. As a serious compute farm...heh...don't get your hopes up until someone cobbles one of these together for the educational aspects of the cluster and reports good results first.

[vadsamoht]

Quote from Svartalf on August 29, 2011, 13:31
As a research project, I'd say go for it. As a serious compute farm...heh...don't get your hopes up until someone cobbles one of these together for the educational aspects of the cluster and reports good results first.


While I agree that the main benefit in regards to clustering would be as an easy way to learn how to do it, there are still ways of making it do something worthwhile. Perhaps the most common-sense (at least to me) way of using the r-pi would be to use is as the head of the cluster, while one or more 'real' computers do the grunt work as nodes. This way, the light but not insignificant work done in rendering the user interface, etc. at the top level would be handled by the weakest processor, which is what you would want.

[Lob0426]

Andre_P:
http://www.tomsguide.com/us/ps.....-5297.html
The Airforce PS3 cluster was completed in 2009.

42 core ARM cluster:
http://m.geek.com/geekdotcom/#.....154a00059c
Notes on how to build a Linux cluster:
http://www.gdargaud.net/Hack/C.....Notes.html

[hajj_3]

if this was a dual core version then maybe it may be worth clustering but you would need alot of them to beat a pentium g620 skt 1155 £51 apu. Also the software would probably only work for very few applications.

[anndra]

I built a Ameoba system out of a load of spark classics back in the day. (Didn't manage to get it to do anything useful, other than heating the house...)

While alas there isn't an ARM port for Ameoba. Plan 9 is available for Arm and Inferno is too. Alas the Plan 9 wedsite seems to be down today so I have no idea which cores it supports.

All the same fun to dream about.

[Svartalf]

Quote from hajj_3 on August 29, 2011, 16:19
if this was a dual core version then maybe it may be worth clustering but you would need alot of them to beat a pentium g620 skt 1155 £51 apu. Also the software would probably only work for very few applications.

1) Cluster... Software's going to be needing to be adjusted at least slightly even if you managed to get a Kerriged/Mosix shared system image cluster going. It just doesn't work transparently out of box unless you're doing a NUMA SMP type cluster (To which, this ARM's not really designed to support that...).

2) Don't forget that you've got a pretty hot GPU there (enough that the CPU hit still doesn't choke down Q3:A when compared to a BeagleBoard...)- GPGPU is possible (with a bit of effort...) and you could do a Beowulf MPI type cluster that you divide and conquer with and accomplish close to the Intel kit without GPU help could do.

The only really crippling aspect of this is the 10/100 Ethernet you're going to be stuck with on these boards. Even if you had Gig-E, you're going to be crippled by the CPU not being able to get it out to the wire fast enough. If you could offset a bit of the CPU disadvantage and found a way to get about gigabit speeds between legs of the cluster- a stack of about 32 of these would be an intriguing low-power cluster that could actually do some usable experimentation on the cheap.

[Svartalf]

Quote from abishur on August 3, 2011, 00:00
Okay first off, if anyone does use the R-Pi in a cluster I think it either needs to be called a druplet (druplets are the tiny bumps that make up a raspberry) or a bramble (which is actually a cluster of raspberries is called). Just a thought that makes me laugh.

Heh... It's a valid thought. We probably ought to move this over to the Collab section- it's liable to form up to be a proper project here once we get our hands on things (Thinking it through, I wouldn't mind making it one of the first things I reach for (which means buying a bunch more than I'd initially planned for...heh...) even though it's not assured to be much more than an educational curiosity to educate them on aspects of current computing directions (A cluster of 32 in a SSI and a Beowulf type configuration would be an interesting thing for students to try their hands on this stuff- and they will need to do it at some point real soon.)

[EWH]

In terms of processing power I don't think it would be cost-effective except for educational purposes. Even if GPGPU is available (and I hope it will be) the effective processing power will be a bit over 15 GFLOPS. (60% of nominal 24 GFLOPS). The SD cards, power supply, cables and ethernet switch ports would work out to $12-$30 per node, depending on how you did it. A 64 node ~1TFLOPS (effective) bramble would cost about $3000 - $5000. That's at least 3x what a conventional GPGPU system would cost (32 bit precision), though the RP would likely use 1/10th the power.

One use where the Raspberry Pi could be effective, though is in display front ends. Each has an HDMI port, so for under $200 per node with 20" 1080p screen, one could make a video wall. For less than $3200 one could make a 4 x 4, 80"+, 33 megapixel, 60Hz display. Other I/O bound applications could also benefit such as sensor arrays.

[Svartalf]

Quote from EWH on September 1, 2011, 18:30
In terms of processing power I don't think it would be cost-effective except for educational purposes. Even if GPGPU is available (and I hope it will be) the effective processing power will be a bit over 15 GFLOPS. (60% of nominal 24 GFLOPS). The SD cards, power supply, cables and ethernet switch ports would work out to $12-$30 per node, depending on how you did it. A 64 node ~1TFLOPS (effective) bramble would cost about $3000 - $5000. That's at least 3x what a conventional GPGPU system would cost (32 bit precision), though the RP would likely use 1/10th the power.

The purpose IS for educational purposes. It's an inexpensive way of building up and understanding basically what the big-boys are doing with heftier hardware. And a 64-node setup's a bit far-fetched. I was envisioning a $500-1000 setup that would be effective for the purposes of teaching someone these things with something cheap that if you mess it up, it's cheap to replace piece parts easily enough.


One use where the Raspberry Pi could be effective, though is in display front ends. Each has an HDMI port, so for under $200 per node with 20" 1080p screen, one could make a video wall. For less than $3200 one could make a 4 x 4, 80"+, 33 megapixel, 60Hz display. Other I/O bound applications could also benefit such as sensor arrays.

Don't think that I've not contemplated that as a Chromium/WireGL type affair

But you've got to come up with most of the same thinking as the HPC cluster for that idea...