Cluster (Bramble...) Design Discussion (Advanced)


169 posts   Page 1 of 7   1, 2, 3, 4, 5 ... 7
by Svartalf » Mon Aug 29, 2011 5:22 pm
Okay, abishur started this discussion in general discussion area. And while it's an okay discussion, all things considered... We should start having a collaboration discussion on how one would actually get to a proper cluster design with a stack of these boards.

"Bramble" is a cute naming for this design effort. So I think it probably will stick. :D

From there...there's considerations on how to interconnect, etc. So, we should probably start considering those design tradeoffs and start thinking on what sorts of work we'd need to accomplish the two common cluster types (and if we could even MANAGE this with the first gen boards because of RAM limitations...).
Posts: 596
Joined: Fri Jul 29, 2011 6:50 pm
by Lob0426 » Mon Aug 29, 2011 8:46 pm
First off you will need a switch that has ports for every device. I would reccomend an 8 port or 16 port switch. Make sure it is stackable in case you decide to have more devices in your "Bramble". Keep all cabling as short as possible for the best performance. You will prbably want to use a regular electronics power supply and so you will need a distribution board for power. One device is going to be your interface to the Bramble. You will have to decide how you are going to mount them. Since all of the boards have mounting holes at, or near, each corner you could use threaded rod with aluminum or plastic spacers for a vertical stack. You could also mount them individually to a project board with screws and spacers, don't crush the SD card socket. Whichever mounting option you finally choose make sure you can plug video and input devices into each device for troubleshooting. your primary device will most likely have an HDD attached so make sure to have a mount for it to.
512MB version 2.0 as WordPress Server
Motorola Lapdock with 512MB
Modded Rev 1.0 with pin headers at USB

http://rich1.dyndns.tv/
(RS)Allied ships old stock to reward its Customers for long wait!
User avatar
Posts: 1898
Joined: Fri Aug 05, 2011 4:30 pm
Location: Susanville CA.
by liz » Mon Aug 29, 2011 8:48 pm
I love the idea of calling a cluster a Bramble. Consider the term as having official Raspberry Pi blessing!
--
Head of Comms, Raspberry Pi Foundation
User avatar
Raspberry Pi Foundation Employee & Forum Moderator
Raspberry Pi Foundation Employee & Forum Moderator
Posts: 4022
Joined: Thu Jul 28, 2011 7:22 pm
by Svartalf » Mon Aug 29, 2011 9:33 pm
Quote from Lob0426 on August 29, 2011, 21:46
First off you will need a switch that has ports for every device. I would reccomend an 8 port or 16 port switch. Make sure it is stackable in case you decide to have more devices in your "Bramble".


This is a good suggestion- and one of the ones I'd make.

One of the other suggestions is that not all switches are made alike- and not all of the name-brand ones are something you'd want for the spine of a Bramble. "Stackable" is going to be a bit of an issue. Most of the stackable ones are "enterprise" level hardware and just took the price advantage of this for experimentation for people and chucked it out the window. If you've got one of the switches worth messing with, while there'll be a slight penalty for daisy-chaining the switch, you can reliably get away with it- so long as you don't try for a high-availability configuration (most consumer hardware does not have spanning-tree support and you'll just cause a packet storm on your cluster with the loops it can't resolve...).


Keep all cabling as short as possible for the best performance.


Heh... Standard 9' cabling will work as well as 3' stuff. Main reason for the short cabling would be to make it easy to keep "tidy". ;)


You will prbably want to use a regular electronics power supply and so you will need a distribution board for power.


While this isn't as crucial as it sounds, it helps. It's just not something that you have to have as we experiment with bringing up a Bramble for the first time.


One device is going to be your interface to the Bramble.


Yep.


You will have to decide how you are going to mount them. Since all of the boards have mounting holes at, or near, each corner you could use threaded rod with aluminum or plastic spacers for a vertical stack. You could also mount them individually to a project board with screws and spacers, don't crush the SD card socket. Whichever mounting option you finally choose make sure you can plug video and input devices into each device for troubleshooting.


For the initial work, you could just stack them one on top of each other 4-8 of them to keep it tidy. For a larger design once we get moving, we could mount them spacered onto a thin piece of plywood or onto metal/plastic railings, four at a time or 8 at a time onto the carrier board. From there, you'd stack those groupings, again for tidiness.


your primary device will most likely have an HDD attached so make sure to have a mount for it to.


Considering that you can get up to 32Gb of SDHC, you might not need anything in the head-node other than that. You could do with the smallest realistic SD card size (2Gb?) and then a huge one on the main device. It's not like this is going to, at least initially with things, be doing stuff that would rate more than that sort of storage.
Posts: 596
Joined: Fri Jul 29, 2011 6:50 pm
by Lob0426 » Tue Aug 30, 2011 12:09 am
All good points. I was just trying to get a starting point here. It all depends on what you do with it.

HDD probably not needed as you say.

Starting with any 16 port switch will work. It would be best if it had a gigabit link port. but the idea is to keep the price down. I don't see any reason to by a switch that costs more than all of the RasPi together. I saw a linksys 16 port Gigabit switch for around $48

I have always had better luck with shorter cabling. I believe there is less possibility of radio interference with shorter lengths. And it is neater. Just one of my quirks I guess.

As to the power supply if you use 16 RasPI you probably would come out cheaper on an electronics supply than 16 adapters and the powerstrips to plug them in. lol

As to mounting threaded rod with spacers rules. Easy access from all directions and easy to setup. vertical stack would be easiest to assemble.
512MB version 2.0 as WordPress Server
Motorola Lapdock with 512MB
Modded Rev 1.0 with pin headers at USB

http://rich1.dyndns.tv/
(RS)Allied ships old stock to reward its Customers for long wait!
User avatar
Posts: 1898
Joined: Fri Aug 05, 2011 4:30 pm
Location: Susanville CA.
by iAreNewb » Tue Aug 30, 2011 12:35 am
The R-Pi doesn't support 1Gbps Ethernet, if I recall correctly.

As for the setup of the R-Pi's in a stack, you could make it horizontal. That is, you could line them up across the table, tilted upwards, with the line going in the same direction as the ports on the switch; this could help reduce wiring clutter, perhaps even compact all the R-Pi's into a modded case of some sort (excluding the one being used as an interface, of course).
Posts: 59
Joined: Sun Aug 28, 2011 4:27 pm
by amiga65 » Tue Aug 30, 2011 1:42 am
I'm glad this thread was started, kudos to You all.
Since first reading about the R-pi months ago I have thought about teaching Myself how to build a cluster / bramble. Now not being an expert at all on brambleing these are just thoughts I've had.
1, Even though the R-pi has only 100 Mb Ethernet I would use a Gb switch as more than a couple R-pi's may saturate the switch, plus most Everyone here has a pc and I'm sure They will at least plug that in as a head node at least once , just because.
2, I've thought about a dozen wall warts hanging all over My desk, 1 large power supply one out there.
3, The case, after seeing the demo video with Ebon and the R-pi with the blinking leds I imagined a dozen R-pi in a shadowbox / picture frame circling around a gutted switch hanging above My desk.
That's just My 2 cents worth.
Posts: 70
Joined: Wed Jul 27, 2011 10:59 pm
by Svartalf » Tue Aug 30, 2011 2:42 am
Quote from iAreNewb on August 30, 2011, 01:35
The R-Pi doesn't support 1Gbps Ethernet, if I recall correctly.


You recall correctly. Biggest problem will be in how much data you can exchange- 100Mbits is pretty limiting for how much you can scale the Bramble out and gain boosts in performance by adding more R-Pi's to it.


As for the setup of the R-Pi's in a stack, you could make it horizontal. That is, you could line them up across the table, tilted upwards, with the line going in the same direction as the ports on the switch; this could help reduce wiring clutter, perhaps even compact all the R-Pi's into a modded case of some sort (excluding the one being used as an interface, of course).


It's a possibility- I'll admit I'd not thought of it, mainly because of the fabrication effort to have them angled. But if it can be simplified, it'd look real cool to do it.
Posts: 596
Joined: Fri Jul 29, 2011 6:50 pm
by Svartalf » Tue Aug 30, 2011 2:53 am
Quote from amiga65 on August 30, 2011, 02:42
I'm glad this thread was started, kudos to You all.


You're welcome. Having worked with much more aggressive hardware than this in the Telecom industry, I'd initially thought that it'd not be worth the effort. After thinking about it a bit and thinking about the base purpose of the device...I rethought that. I'm beginning to think I was right on the re-think... :D


1, Even though the R-pi has only 100 Mb Ethernet I would use a Gb switch as more than a couple R-pi's may saturate the switch, plus most Everyone here has a pc and I'm sure They will at least plug that in as a head node at least once , just because.


The only way you're going to saturate a switch is if it's a poor, really, really cheap switch. Aggregate switch bandwidth can be as high as the data rate on all the ports on the switch- even with inexpensive ones. Now, having said this, the odds are good that the gigabit switch devices, even the craptastic ones, will handle 4Gbits per second aggregate on a 5-port device, and something like 6-7 on an 8 or 16 port cheapie.


2, I've thought about a dozen wall warts hanging all over My desk, 1 large power supply one out there.


Oh, most definitely. At the minimum, we'll want to figure out the maximum power consumption of the Bramble and match it to a power supply that is rated to half again more amps than that. Switching supplies, when you exceed the 2/3rds 3/4ths of maximum rating, tend to have shorter lifespans and produce more heat than one would want them to.


3, The case, after seeing the demo video with Ebon and the R-pi with the blinking leds I imagined a dozen R-pi in a shadowbox / picture frame circling around a gutted switch hanging above My desk.


Hm... Intriguing mix of computer science and art. I Like that. :D
Posts: 596
Joined: Fri Jul 29, 2011 6:50 pm
by Lob0426 » Tue Aug 30, 2011 5:20 am
iAreNewb: Nope RasPi does not support Gigabit but that is still a good price on a 16 port router. It also would have the overhead to uplink ; ).
512MB version 2.0 as WordPress Server
Motorola Lapdock with 512MB
Modded Rev 1.0 with pin headers at USB

http://rich1.dyndns.tv/
(RS)Allied ships old stock to reward its Customers for long wait!
User avatar
Posts: 1898
Joined: Fri Aug 05, 2011 4:30 pm
Location: Susanville CA.
by markstinson » Thu Sep 01, 2011 1:43 am
amiga65 just pointed me over here from my nearly identical post in the Educational applications section (i.e. for classes on HPC concepts & programming OR on cluster administration).

Something not mentioned here so far is it probably would be a good idea to have a female 4 pin serial port with the other, similar to the kind used with JTAG connectors, so folks can use serial2usb cable for true console support on their laptop/desktop. You must always have real serial port for a console connection. USB2Serial adapters on the host requires a kernel module to be loaded & configured. Students will need to learn to set that as kernel params in the boot loader before they run into problems. Without it, USB2serial dongle will unlikely be available in single user mode.

As for physically arranging them, I have to agree that stacking ala PC/104 style can be cumbersome. For the "railed" enclosures (like the Radio Shack and Jameco project cases), it wouldn't hurt to leave some border/margin on the circuit board's long edges so the R-pi could be slid into small case's track ala Blade-style.

And, if you going to have them in a Blade-like style, it would be helpful to have all ports & plugs on one end stacked (with a smidge of spacing due to some USB plug's "fatness") like this :

----------------------------------
[s] 4 pin serial hdmi [h]
[u] USB composite [c]
[u] USB left chan [l]
[e] Ethernet right chan [r]
o power jack 35 mm jack o
----------------------------------

This would leave essential connectors for headless operation on one end and human display & sound at the other. So you can slide them in a case or "plant" them vertically like flowers.

Cheers, Mark S.
Posts: 7
Joined: Wed Aug 31, 2011 7:15 pm
by proffalken » Thu Sep 01, 2011 8:01 am
Hi all,

New to this thread (and the forum) however I'd love to get involved in this project.

My skills lie in automated configuration and management of systems, so if these things run Debian (and I'm assuming that they do given the comments on the front page!) then there's no reason you couldn't install puppet onto the head-node and manage the rest of the cluster with that.

As far as hardware is concerned, the following look like viable options for switching:

http://www.ebuyer.com/116620-z.....10-084003b
http://www.ebuyer.com/119379-z.....10-083003b

Especially given the price and the fact that the "VIP" posts sound suspiciously like QoS to me... :)

Cheers,

PF.
Posts: 13
Joined: Wed Aug 31, 2011 3:11 pm
by Svartalf » Thu Sep 01, 2011 12:41 pm
The VIP ports aren't QoS...well, not in the sense that the standards specifies. On something this cheap, you're going to find that they have a couple of ports and they're given priority over the other ports when transacting things over the fabric.
Posts: 596
Joined: Fri Jul 29, 2011 6:50 pm
by rocket-dog » Mon Sep 05, 2011 7:26 pm
Once the RPi is out in the wild perhaps someone could organise a clustering day?

It seems many of us are going to buy at least a few of these little boxes. If we could get a thousand (!) or so in the same place to perform some computational task it would be good publicity.
Posts: 7
Joined: Sun Sep 04, 2011 6:51 pm
by iAreNewb » Mon Sep 05, 2011 11:06 pm
Well, for one thing, we'd have to have LOTS of switches (that are highly stackable) to the Raspberry Bramble Festival. I mean, for a thousand R-Pi's, you'd need a very efficient protocol to organize the Mega-Bramble, and switches that scale very well.

A single switch, to stack with others, would require 2 ports to daisy chain. Since this would be a disgusting wiring mess, one would probably want a nice wireless AP as opposed to a conventional wired switch. This eliminates the Ethernet requirements, but also throttles bandwidth to a fraction of a USB port's (which when running free of other USB devices is maxed at about 30 Mbps, IIRC). Moreover, wireless APs are probably capable of way more clients and bandwidth.

Set up maybe 20 such APs with a capacity of perhaps 50 connected R-Pi's each, and a simple tree to propagate data to the appropriate R-Pi. Run a lightweight controller program on a laptop, and connect it the main AP, and you can show some major parallel power (via GUI/console that sends "tasks" [i.e. computation tasks]) on a projector. Perfect for a press-covered event. (Solve a freakishly gigantic maze in 1080p, crack a bunch of hashes, donate power to a BOINC project... the list goes on)
Posts: 59
Joined: Sun Aug 28, 2011 4:27 pm
by liz » Mon Sep 05, 2011 11:10 pm
I have two words: LAN party!
--
Head of Comms, Raspberry Pi Foundation
User avatar
Raspberry Pi Foundation Employee & Forum Moderator
Raspberry Pi Foundation Employee & Forum Moderator
Posts: 4022
Joined: Thu Jul 28, 2011 7:22 pm
by amiga65 » Mon Sep 05, 2011 11:22 pm
Wonder how many nodes the worlds biggest cluster has!
Posts: 70
Joined: Wed Jul 27, 2011 10:59 pm
by bnolsen » Tue Sep 06, 2011 1:34 am
Quote from amiga65 on September 6, 2011, 00:22
Wonder how many nodes the worlds biggest cluster has!


biggest or fastest? they aren't the same thing.
Posts: 75
Joined: Sat Aug 13, 2011 5:37 pm
by iAreNewb » Tue Sep 06, 2011 1:53 am
Biggest AND fastest are both debatable, thanks to the ambiguously defined, Internet-based, distributed computing "clusters." Now, if you were to replace the word cluster with "privately owned cluster," then you could probably look up the fastest (maybe biggest too?) one on Top500.
Posts: 59
Joined: Sun Aug 28, 2011 4:27 pm
by jacklang » Tue Sep 06, 2011 8:23 am
Posts: 166
Joined: Thu Aug 04, 2011 10:59 am
by jamesh » Tue Sep 06, 2011 8:48 am
Quote from jacklang on September 6, 2011, 09:23
http://www.zdnet.co.uk/news/em.....-40093356/


So, just 15M required to build if made out of Raspi's...

And if the GPU could be harnessed (OpenCL - not available but possible I suppose), you could really get some ooomph out of it.
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 11471
Joined: Sat Jul 30, 2011 7:41 pm
by liz » Tue Sep 06, 2011 8:48 am
Perhaps we could simulate a lizard brain instead. Think small, folks.
--
Head of Comms, Raspberry Pi Foundation
User avatar
Raspberry Pi Foundation Employee & Forum Moderator
Raspberry Pi Foundation Employee & Forum Moderator
Posts: 4022
Joined: Thu Jul 28, 2011 7:22 pm
by Lakes » Tue Sep 06, 2011 9:45 am
Ant Brain?
Posts: 267
Joined: Wed Aug 24, 2011 2:17 pm
by liz » Tue Sep 06, 2011 10:05 am
I have a neurophysicist buddy in Zurich who models the brains of fruit flies. Last year, he showed me a demo where he was controlling the movements of actual, honest-to-god maggots with an actual, honest-to-god WiMote. (No, I have no idea how.) Possibly the most awesome thing ever.

Perhaps we could find some volunteer arthropods and go from there.
--
Head of Comms, Raspberry Pi Foundation
User avatar
Raspberry Pi Foundation Employee & Forum Moderator
Raspberry Pi Foundation Employee & Forum Moderator
Posts: 4022
Joined: Thu Jul 28, 2011 7:22 pm
by jamesh » Tue Sep 06, 2011 10:34 am
First time I've seen the phrase most awesome thing ever, and maggots in the same paragraph.
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 11471
Joined: Sat Jul 30, 2011 7:41 pm