User avatar
kangmo
Posts: 8
Joined: Tue Feb 17, 2015 4:14 pm
Contact: Website

Building 24 Raspberry Pi2 Cluster

Tue Feb 17, 2015 4:35 pm

Hi,

First of all, thank you Respberry Pi org, for creating such a great device. You guys simply rock.

Last week, I found that I had 24 port switching hub from Cisco.
Image

I started thinking what if...
24 Raspberry Pi2s are connected to the hub, and see if Scala+Akka Cluster would work on top of JVM.
Today, I and my boy(;-)) started building the 24 Pi2 cluster.
Image

Wanted to check if somebody did this before and have some wisdom to share with me. I prepared 5V, 2.4A power for each Pi2, and I am going to connect them using 100Mbps LAN.

I have too many things that I don't know. -_-;;;;;;
Would expensive and speedy micro SD cards boost the performance of each Pi2? I am about to use 32G uSD for each Pi2.
Also is there any easy deployment tool that I can use on Pi2? Docker seems not to be a good solution, as it requires x86 linux to run on top of Arm Pi2 ;-).
Which JVM do you recommend for the best performance? (It is ok, if it is compatible w/ Java 7 or 8)
Last edited by kangmo on Tue Feb 17, 2015 11:09 pm, edited 2 times in total.

User avatar
pluggy
Posts: 3635
Joined: Thu May 31, 2012 3:52 pm
Location: Barnoldswick, Lancashire,UK
Contact: Website

Re: Building 24 Raspberry Pi Cluster

Tue Feb 17, 2015 7:27 pm

Pi clusters tend to be built as proof of concepts, educational tools or 'because I can' rather than to use for a specific purpose, because unless you're a specialist programmer with an application that can be programmed to use a cluster, its next door to useless. If you have something you actually need a cluster for, you wouldn't use a Pi as a starting block, you'd use something far more powerful. You need a task that can be slit up into a series of smaller tasks, you give each individual computer one of the smaller tasks, and then you assemble all the results into a big result. Weather forecasting, rendering CGI movies one frame at a time, and analysing SETI reception files are good tasks for clusters.

A cluster of Pi's does not give you a big powerful computer, it gives you a network of small computers, and its up to you to give each of them a workload.
Don't judge Linux by the Pi.......
I must not tread on too many sacred cows......

User avatar
kangmo
Posts: 8
Joined: Tue Feb 17, 2015 4:14 pm
Contact: Website

Re: Building 24 Raspberry Pi Cluster

Tue Feb 17, 2015 11:00 pm

Hi Pluggy, thanks for the advice. I agree with you :-).
Actually I was expecting to build a server that consumes low energy, but looks like I was too ambitious.

I am going to build a columnar database that runs on N machines using Akka Cluster, which provides cluster membership, heartbeats, and so on. And then my plan is to build a search engine on top of it. My customer wants me to build a search engine that runs on N computers, so my assumption is that the Pi2 cluster will help me to understand how the Akka Cluster works when a machine(=Pi2) turns off, disconnects from a network, etc.

Also testing consensus algorithms like PAXOS or RAFT on top of Pi2 cluster would be interesting to investigate how consensus based virtual currencies like Ripple or Stellar work.

I'll post about the performance I get using the Akka Cluster once the 24 Pi2 cluster starts working. ;-)
pluggy wrote:Pi clusters tend to be built as proof of concepts, educational tools or 'because I can' rather than to use for a specific purpose, because unless you're a specialist programmer with an application that can be programmed to use a cluster, its next door to useless. If you have something you actually need a cluster for, you wouldn't use a Pi as a starting block, you'd use something far more powerful. You need a task that can be slit up into a series of smaller tasks, you give each individual computer one of the smaller tasks, and then you assemble all the results into a big result. Weather forecasting, rendering CGI movies one frame at a time, and analysing SETI reception files are good tasks for clusters.

A cluster of Pi's does not give you a big powerful computer, it gives you a network of small computers, and its up to you to give each of them a workload.

User avatar
B.Goode
Posts: 9833
Joined: Mon Sep 01, 2014 4:03 pm
Location: UK

Re: Building 24 Raspberry Pi2 Cluster

Wed Feb 18, 2015 9:31 am

You will probably have seen the writeup of the project at Southampton University, but here is the url for reference: http://www.southampton.ac.uk/~sjc/raspberrypi/

User avatar
RaTTuS
Posts: 10549
Joined: Tue Nov 29, 2011 11:12 am
Location: North West UK
Contact: Twitter YouTube

Re: Building 24 Raspberry Pi2 Cluster

Wed Feb 18, 2015 9:46 am

having one beefy 5V supply 25A should cover it if nothing else is plugged in
[or a couple of 14A ones, or blocks of 4x 10A - up to you]
would be better an more efficient than 24 small PSU's
How To ask Questions :- http://www.catb.org/esr/faqs/smart-questions.html
WARNING - some parts of this post may be erroneous YMMV

1QC43qbL5FySu2Pi51vGqKqxy3UiJgukSX
Covfefe

User avatar
slmaws
Posts: 32
Joined: Wed Nov 13, 2013 6:41 pm
Location: Poland
Contact: Website

Re: Building 24 Raspberry Pi Cluster

Wed Feb 18, 2015 9:50 am

pluggy wrote:A cluster of Pi's does not give you a big powerful computer, it gives you a network of small computers, and its up to you to give each of them a workload.
It was true for old Pi, on Pi2 you have 4 cores and each core is about 3x faster than old CPU.
24 new Raspberry Pis still will be slower than my laptop with i7 8x 2.7GHz CPU but for testing purposes it's quite good
See my Raspberry Pi tips on http://rpitips.com

User avatar
scidata
Posts: 92
Joined: Tue Jan 06, 2015 4:21 pm
Location: Toronto
Contact: Website

Re: Building 24 Raspberry Pi2 Cluster

Wed Feb 18, 2015 7:07 pm

One thing that is useful to work on is a cluster controller. A pi (or good router) can provide DHCP, DNS, NTP, FTP, and other services (even exotic stuff like NAS, Beowulf, ZFS). This knowledge (and maybe even the controller itself) can be moved to a big, more powerful cluster eventually.
"Self-education is, I firmly believe, the only kind of education there is" - Isaac Asimov

User avatar
pluggy
Posts: 3635
Joined: Thu May 31, 2012 3:52 pm
Location: Barnoldswick, Lancashire,UK
Contact: Website

Re: Building 24 Raspberry Pi Cluster

Wed Feb 18, 2015 7:19 pm

slmaws wrote:
pluggy wrote:A cluster of Pi's does not give you a big powerful computer, it gives you a network of small computers, and its up to you to give each of them a workload.
It was true for old Pi, on Pi2 you have 4 cores and each core is about 3x faster than old CPU.
24 new Raspberry Pis still will be slower than my laptop with i7 8x 2.7GHz CPU but for testing purposes it's quite good
OK with the Pi2 you get a network of not quite as small computers. A single PC with a gutsy processor (i7 as you suggest) is a better bet.
Don't judge Linux by the Pi.......
I must not tread on too many sacred cows......

Heater
Posts: 15386
Joined: Tue Jul 17, 2012 3:02 pm

Re: Building 24 Raspberry Pi2 Cluster

Wed Feb 18, 2015 7:23 pm

slmaws.
24 new Raspberry Pis still will be slower than my laptop with i7 8x 2.7GHz CPU
That is no doubt true from a raw compute speed perspective.

There are other reasons for building a Pi cluster though.

It's a good cheap way to explore clustering software, from etcd to distributed databases and whatever.

It's a good cheap way to build a fault tolerant system.

Your laptop cannot:

Drive 96 USB ports. Or collect images from 24 cameras. Or drive a 24 screen video wall. Or heck, provide 24 UARTs.
Memory in C++ is a leaky abstraction .

User avatar
AndyD
Posts: 2334
Joined: Sat Jan 21, 2012 8:13 am
Location: Melbourne, Australia
Contact: Website

Re: Building 24 Raspberry Pi2 Cluster

Wed Feb 18, 2015 10:02 pm

Heater wrote:slmaws.
24 new Raspberry Pis still will be slower than my laptop with i7 8x 2.7GHz CPU
That is no doubt true from a raw compute speed perspective.

There are other reasons for building a Pi cluster though.

It's a good cheap way to explore clustering software, from etcd to distributed databases and whatever.

It's a good cheap way to build a fault tolerant system.

Your laptop cannot:

Drive 96 USB ports. Or collect images from 24 cameras. Or drive a 24 screen video wall. Or heck, provide 24 UARTs.
+1

User avatar
kangmo
Posts: 8
Joined: Tue Feb 17, 2015 4:14 pm
Contact: Website

Re: Building 24 Raspberry Pi2 Cluster

Thu Feb 19, 2015 4:47 am

My brief test shows that Pi2 has 15MB/s throughput for sequential write on a micro SD card (32GB, speed:x300).
Ideally, the cluster should have 15MB/s x 24 = 360MB/s write throughput. I may be able to achieve it by attaching four SD cards that supports 90MB/s to my desktop, but it means I have only four concurrent writes. If I use the write throughput for serving writing log records of a distributed database system, I assume that 24 concurrent writes will distribute contention on the end of log files, but needs to test if this actually improves something. Any idea about this?

I am about to write a prototype that implements Key/Value store that distributes data across 24 Pi2s to verify the assumption. (Will have to use a consensus algorithm such as RAFT to consistently maintain meta data to answer questions such as "Among 24 Pi2s, which one serves key 'abc'?)

Hopefully, the 24 Pi2 cluster can support huge I/O bandwidth with 4 USB drives on each of them but needs to be tested if Pi2 shows increased I/O bandwidth with additional USB drives still keeping the write latency low on each USB drive.

I am assuming that CPU intensive jobs are not for the 24 Pi2 cluster, but I/O intensive jobs probably could be done better with the cluster instead of a server that I can afford with the same cost that I spent to build the 24 Pi2 cluster.

An update : Scala & Akka Cluster works fine on a single Pi2 device. will verify if it works on the 24 Pi2 cluster.
Last edited by kangmo on Thu Feb 19, 2015 5:09 am, edited 4 times in total.

User avatar
kangmo
Posts: 8
Joined: Tue Feb 17, 2015 4:14 pm
Contact: Website

Re: Building 24 Raspberry Pi2 Cluster

Thu Feb 19, 2015 4:53 am

B.Goode, thanks for the great info. Actually I have seen a photo of the cluster on google image. But seeing video of the cluster was something I wanted. :-)
B.Goode wrote:You will probably have seen the writeup of the project at Southampton University, but here is the url for reference: http://www.southampton.ac.uk/~sjc/raspberrypi/
Last edited by kangmo on Thu Feb 19, 2015 5:06 am, edited 1 time in total.

User avatar
kangmo
Posts: 8
Joined: Tue Feb 17, 2015 4:14 pm
Contact: Website

Re: Building 24 Raspberry Pi2 Cluster

Thu Feb 19, 2015 4:53 am

+1
Heater wrote:slmaws.
24 new Raspberry Pis still will be slower than my laptop with i7 8x 2.7GHz CPU
That is no doubt true from a raw compute speed perspective.

There are other reasons for building a Pi cluster though.

It's a good cheap way to explore clustering software, from etcd to distributed databases and whatever.

It's a good cheap way to build a fault tolerant system.

Your laptop cannot:

Drive 96 USB ports. Or collect images from 24 cameras. Or drive a 24 screen video wall. Or heck, provide 24 UARTs.

User avatar
kangmo
Posts: 8
Joined: Tue Feb 17, 2015 4:14 pm
Contact: Website

Re: Building 24 Raspberry Pi Cluster

Thu Feb 19, 2015 4:57 am

slmaws wrote:
pluggy wrote:A cluster of Pi's does not give you a big powerful computer, it gives you a network of small computers, and its up to you to give each of them a workload.
It was true for old Pi, on Pi2 you have 4 cores and each core is about 3x faster than old CPU.
24 new Raspberry Pis still will be slower than my laptop with i7 8x 2.7GHz CPU but for testing purposes it's quite good
slmaws, I also assume that 24 Pi2s will be slower than your laptop. :-) I think the cluster should be doing better for I/O intensive jobs.

User avatar
kangmo
Posts: 8
Joined: Tue Feb 17, 2015 4:14 pm
Contact: Website

Re: Building 24 Raspberry Pi2 Cluster

Thu Feb 19, 2015 5:00 am

RaTTuS wrote:having one beefy 5V supply 25A should cover it if nothing else is plugged in
[or a couple of 14A ones, or blocks of 4x 10A - up to you]
would be better an more efficient than 24 small PSU's
RaTTuS, thanks for the info. Actually my USB hub has two modes. A. 5V / 2.4A for 5 ports, B. 5V / 1A for 10 ports.
I assume that you agree that 5V / 1A for each Pi2 is enough if nothing is attached to it. :-)

User avatar
RaTTuS
Posts: 10549
Joined: Tue Nov 29, 2011 11:12 am
Location: North West UK
Contact: Twitter YouTube

Re: Building 24 Raspberry Pi2 Cluster

Thu Feb 19, 2015 8:36 am

kangmo wrote:
RaTTuS wrote:having one beefy 5V supply 25A should cover it if nothing else is plugged in
[or a couple of 14A ones, or blocks of 4x 10A - up to you]
would be better an more efficient than 24 small PSU's
RaTTuS, thanks for the info. Actually my USB hub has two modes. A. 5V / 2.4A for 5 ports, B. 5V / 1A for 10 ports.
I assume that you agree that 5V / 1A for each Pi2 is enough if nothing is attached to it. :-)
1A / RPi2 will be fine [YMMV]
a bigger PSU will give more efficiency in general
How To ask Questions :- http://www.catb.org/esr/faqs/smart-questions.html
WARNING - some parts of this post may be erroneous YMMV

1QC43qbL5FySu2Pi51vGqKqxy3UiJgukSX
Covfefe

mimi123
Posts: 583
Joined: Thu Aug 22, 2013 3:32 pm

Re: Building 24 Raspberry Pi Cluster

Thu Feb 19, 2015 9:17 am

kangmo wrote:
slmaws wrote:
pluggy wrote:A cluster of Pi's does not give you a big powerful computer, it gives you a network of small computers, and its up to you to give each of them a workload.
It was true for old Pi, on Pi2 you have 4 cores and each core is about 3x faster than old CPU.
24 new Raspberry Pis still will be slower than my laptop with i7 8x 2.7GHz CPU but for testing purposes it's quite good
slmaws, I also assume that 24 Pi2s will be slower than your laptop. :-) I think the cluster should be doing better for I/O intensive jobs.
I think that using VideoCore-side SHA256 will make the Pi2Bs faster( or even NEON)

User avatar
RichardUK
Posts: 238
Joined: Fri Jun 01, 2012 5:12 pm

Re: Building 24 Raspberry Pi2 Cluster

Thu Feb 19, 2015 9:40 pm

Just a small note, I did some bandwidth tests of a USB drive (USB SATA with SSD Drive) connected to a RPi2 against the same drive connected to a RPi1 B+. I found no difference when transferring data over the network. I concluded that the bottleneck was the network port. So I would not expect that the RPi2 would give you any faster bandwidth over a RPi1 B+. It was not a very scientific test, copied a 2 gig file and looked at the bandwith. Was pinned @ 10MB/sec. My Stora NAS on a gigabit network port gets around 16MB/sec. Connecting it using a 100Mb switch gave the same bandwidth as the RPi's.

I would be interested in how well it scales with 24 of them.

User avatar
slmaws
Posts: 32
Joined: Wed Nov 13, 2013 6:41 pm
Location: Poland
Contact: Website

Re: Building 24 Raspberry Pi2 Cluster

Thu Feb 19, 2015 9:58 pm

Heater wrote: Drive 96 USB ports. Or collect images from 24 cameras. Or drive a 24 screen video wall. Or heck, provide 24 UARTs.
You don't need to create "cluster" to do it. Actually I also have plans to build cluster but just for fun :)
See my Raspberry Pi tips on http://rpitips.com

ajazam
Posts: 6
Joined: Thu Feb 19, 2015 11:03 pm

Re: Building 24 Raspberry Pi2 Cluster

Thu Feb 19, 2015 11:07 pm

Hello,

I'm also building a cluster for fun. It'll be used for akka, cassandra, spark etc. It'll only have 8 nodes of raspberry PI 2's.

Can anybody recommend any standoffs so I can create each racks of four nodes?

marked
Posts: 218
Joined: Fri Jul 29, 2011 4:25 pm

Re: Building 24 Raspberry Pi2 Cluster

Fri Feb 20, 2015 9:44 am

ajazam wrote:Can anybody recommend any standoffs so I can create each racks of four nodes?
ha -ha -ha -ha -ha ... thunk.<--- that is my head bouncing off the table.

Absolutely no pi company sells them. They sell 12quid pieces of acrylic that can mount three pis, and standoffs, but not standoffs/nuts/washers/bolts on their own.

You want M2.5 20mm standoffs at least, for clearance both top and bottom to prevent ethernet/usb solder legs from touching the board beneath. Possibly more if you have camera/display/gpio attached.

ebay 5.00ukp for 4 on for metal, 3.00 for 50 for plastic/nylon shipping 3-4weeks. If you want the expensive option digikey. Expensive by which I mean cheap, but expensive postage and packing.. RS/Farnell probably have something similar.

ajazam
Posts: 6
Joined: Thu Feb 19, 2015 11:03 pm

Re: Building 24 Raspberry Pi2 Cluster

Fri Feb 20, 2015 1:50 pm

Great,

I thought I was going mad. I can't find any either from the rpi suppliers. I don't think 20mm will be enough. I found some on ebay

http://www.ebay.co.uk/itm/290931842545

I'll order some and see what I get. They only cost £4.41, but they're coming from hong kong, so it'll be a while. I've only got one rpi 2 for now, seven will come sometime in March. So I should be ok, if the standoffs are the right size.

psutton
Posts: 178
Joined: Thu Jan 12, 2012 10:45 pm
Location: Devon
Contact: Website

Re: Building 24 Raspberry Pi2 Cluster

Fri Feb 20, 2015 3:50 pm

We (well one of the young people) are doing this at the Torbay Pi jam, we don't have that many Pi's other than our own, to play with so starting off with a few laptops, (this works)

We need more resources for the Pi jam but also have no where to store stuff at the moment.

Work so far is in a rough write up here

http://www.dcglug.org.uk/cluster-mark-2/

We are using GNU Parallel which seems to be doing what we want, nicely,

I just wondered where this would fit in to the National curriculum for year 7, in terms of what skills are being developed that would fit in with what gets taught at year 7.

marked
Posts: 218
Joined: Fri Jul 29, 2011 4:25 pm

Re: Building 24 Raspberry Pi2 Cluster

Fri Feb 20, 2015 7:38 pm

you might not have the right ones - you want M2.5 (two point five) which is between M2 and M3.

User avatar
jbeale
Posts: 3626
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: Building 24 Raspberry Pi2 Cluster

Fri Feb 20, 2015 8:00 pm

I have an aging desktop running Debian 7 on Intel Q9550 @ 2 GHz. On the task I tested (raytracing) the desktop is 10x faster than the RPi2, but it also consumes 100x (!) more power (200 W vs 2 W). So the Pi wins by a factor of 10 for computational power per watt. I assume the newer laptops will do better, but don't assume they all do! I got a new, low-end Dell Inspiron 3542 laptop just last month (Intel 3558 @ 1.7 GHz, Win 7). It draws 16 W with the processor loaded, 10 W idle and running flat out it is only 5x faster than the Pi2. It also cost 10x more than a Pi2 (ex tax & shipping) although that's not quite fair as it includes a screen, PSU, HDD. Anyway the bottom line is the Pi2 cluster might actually be more power-efficient than some other options.

Return to “General discussion”