LemonLicker
Posts: 20
Joined: Wed Sep 26, 2012 3:51 pm

Compute Module Super Computer/Cluster

Fri Apr 11, 2014 11:15 pm

Hey there guys,

I come to ask of your help because, whilst my knowledge of programming and software is fairly sound, my hardware knowledge is most lacking in comparison.

The idea of the compute module greatly excited me, as it appeared to me that, along with the fact that the board would fit into a SODIMM slot and that it now included less redundant ports, it would be easier than ever to create a rack-like server with compute modules.

Now my questions are these. Firstly, do the extra potential ports of the compute module allow for a more seamless interface between the multiple compute modules? Could the compute modules function together, beyond the seamlessness of a cluster, to act more as a sort of multi-core computer? Sorry if these questions seem stupid or foolish, I really do not know too much about hardware.

Secondly, if one was to build a cluster with computer modules as nodes, what would be the most efficient way of linking all of them and secondly of powering all of them?

Any input would be greatly appreciated.

pbgben
Posts: 2
Joined: Sat Apr 12, 2014 12:04 pm

Re: Compute Module Super Computer/Cluster

Sat Apr 12, 2014 12:08 pm

I've also thought about this. They aren't that powerful on there own, however you can cram lots of them in a relitivle small place and they don't need much power ether.

I'm sure there is a market for a "clustered" setup, I made a simple design concept to fit 120 of them in a 1U full length rack mount case.

Image

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 17882
Joined: Sat Jul 30, 2011 7:41 pm

Re: Compute Module Super Computer/Cluster

Sat Apr 12, 2014 2:15 pm

Clusters really only have a benefit to education stuff - the power/watt is somewhat below a desktop PC.

The problem, as with any cluster, is the connection between the boards. That is not a trivial problem to sort out, given the limited I/O speed of the Raspi.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Please direct all questions to the forum, I do not do support via PM.

User avatar
GTR2Fan
Posts: 1601
Joined: Sun Feb 23, 2014 9:20 pm
Location: South East UK

Re: Compute Module Super Computer/Cluster

Sat Apr 12, 2014 2:27 pm

Also, if you were taking full advantage of the available horsepower, you'd be pushing the GPUs fairly hard. Each compute board could easily reach 80°C+ under such circumstances if 'Turbo' overclocked meaning that you'd need to provide active cooling to keep every compute board cool unless you were happy with them throttling back when they reach 85°C. I could see that being a bit of a problem in a 1U case... unless you weren't overclocking them of course, but that would be a terrible waste of available processing power IMO. ;)
Pi2B Mini-PC/Media Centre: ARM=1GHz (+3), Core=500MHz, v3d=500MHz, h264=333MHz, RAM=DDR2-1200 (+6/+4/+4+schmoo). Sandisk Ultra HC-I 32GB microSD card on '50=100' OCed slot (42MB/s read) running Raspbian/KODI16, Seagate 3.5" 1.5TB HDD mass storage.

j0z0r
Posts: 52
Joined: Fri Oct 28, 2011 5:46 pm

Re: Compute Module Super Computer/Cluster

Mon Apr 21, 2014 6:19 am

LemonLicker wrote:Hey there guys,

I come to ask of your help because, whilst my knowledge of programming and software is fairly sound, my hardware knowledge is most lacking in comparison.

The idea of the compute module greatly excited me, as it appeared to me that, along with the fact that the board would fit into a SODIMM slot and that it now included less redundant ports, it would be easier than ever to create a rack-like server with compute modules.

Now my questions are these. Firstly, do the extra potential ports of the compute module allow for a more seamless interface between the multiple compute modules? Could the compute modules function together, beyond the seamlessness of a cluster, to act more as a sort of multi-core computer? Sorry if these questions seem stupid or foolish, I really do not know too much about hardware.

Secondly, if one was to build a cluster with computer modules as nodes, what would be the most efficient way of linking all of them and secondly of powering all of them?

Any input would be greatly appreciated.
The compute module doesn't expose any higher speed ports/interfaces than were previously available on the Model A or B, and that is the bottleneck. That's why quad core machines have the processors all together; they are able to share data at blazing speed, since they are on the same die. It would be useful for tasks that are easy to distribute over separate processors. I could see a device you're imagining mining bitcoins or doing SETI@home or something similar.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 17882
Joined: Sat Jul 30, 2011 7:41 pm

Re: Compute Module Super Computer/Cluster

Mon Apr 21, 2014 8:18 am

GTR2Fan wrote:Also, if you were taking full advantage of the available horsepower, you'd be pushing the GPUs fairly hard. Each compute board could easily reach 80°C+ under such circumstances if 'Turbo' overclocked meaning that you'd need to provide active cooling to keep every compute board cool unless you were happy with them throttling back when they reach 85°C. I could see that being a bit of a problem in a 1U case... unless you weren't overclocking them of course, but that would be a terrible waste of available processing power IMO. ;)
Er. No.

Running the GPU and ARM full tilt will probably hit 50-55 or so, even when overclocked. Nowhere near 80, unless the ambient condition are overly hot.

That said, I would stick a fan on the box, but it won't need much.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Please direct all questions to the forum, I do not do support via PM.

andycewright
Posts: 10
Joined: Thu Mar 06, 2014 5:29 am

Re: Compute Module Super Computer/Cluster

Mon Apr 21, 2014 11:12 am

The only high speed port on any Pi are the HDMI, CSI & DSI so unless you utilize at least one of them you’re likely wasting resources for a cluster... unless you can utilize IRQ with all or most of the GPIO and write some software.

Does anyone know what speed can be achieved by a single GPIO? I know the PITFT uses GIPO to provide 320x240 video with little lag on the OS, not sure the refresh rate or how many GPIO lines are strictly used for video. With the added GPIO of the CM it might be possible in software to de-multiplex a high speed serial port like SATA, USB 3.0 or gigabit Ethernet into a parallel bus of say 32 GPIO for a very short distance and use hardware multiplexing to reconstruct the parallel bus back into the desired high speed signal. You might sacrifice too much processor power for most applications. There also might be IRQ limitations preventing it. I don't know enough about Linux & Python to write anything to test it out, more of a hardware guy.

Any thoughts from the Towers?

tony-allan
Posts: 4
Joined: Sat Aug 04, 2012 3:17 am
Location: Melbourne, Australia

Re: Compute Module Super Computer/Cluster

Fri May 30, 2014 8:33 am

I like the idea of an 8 or 16 node cluster, with an carrier board with sockets for the raspberry pi's, a USB to 100 m-bit ethernet chip for each pi (just as the existing Model B's use, and an ethernet switch chip on board as well and a 1g-bit ethernet for off board communications.

Not exactly a super computer but should be quite handily for lots of applications at home or school or even in many business environments.

Tony.

User avatar
RaTTuS
Posts: 9489
Joined: Tue Nov 29, 2011 11:12 am
Location: North West UK

Re: Compute Module Super Computer/Cluster

Fri May 30, 2014 9:50 am

tony-allan wrote:I like the idea of an 8 or 16 node cluster, with an carrier board with sockets for the raspberry pi's, a USB to 100 m-bit ethernet chip for each pi (just as the existing Model B's use, and an ethernet switch chip on board as well and a 1g-bit ethernet for off board communications.

Not exactly a super computer but should be quite handily for lots of applications at home or school or even in many business environments.

Tony.
well something like a 4 node cluster and a 5 port NIC would work well - personally I's route out USB and HDMI ... YMMV
How To ask Questions :- http://www.catb.org/esr/faqs/smart-questions.html
WARNING - some parts of this post may be erroneous YMMV

1QC43qbL5FySu2Pi51vGqKqxy3UiJgukSX
Covfefe

LemonLicker
Posts: 20
Joined: Wed Sep 26, 2012 3:51 pm

Re: Compute Module Super Computer/Cluster

Sat May 31, 2014 11:31 pm

tony-allan wrote:I like the idea of an 8 or 16 node cluster, with an carrier board with sockets for the raspberry pi's, a USB to 100 m-bit ethernet chip for each pi (just as the existing Model B's use, and an ethernet switch chip on board as well and a 1g-bit ethernet for off board communications.

Not exactly a super computer but should be quite handily for lots of applications at home or school or even in many business environments.

Tony.
Exactly my thinking; it would be very nice, cheap and modular :)

stutteringp0et
Posts: 15
Joined: Sun Oct 20, 2013 2:48 am

Re: Compute Module Super Computer/Cluster

Sat Jun 21, 2014 9:55 am

tony-allan wrote: Not exactly a super computer but should be quite handily for lots of applications at home or school or even in many business environments.
I was thinking this would be a great development platform for high-availability services. Rather than invest in servers to test with, one could invest in a single RPiCM unit with any number of modules to test on. For less money, probably, one could have 2 or 3 times as many (CM) servers in a single box drawing the same (or less) power than a single real server.

Heck, maybe even a good production platform....

I'd like one (with 120 modules please)

One request though (for any hardware manufacturers reading this) - lots of blinking lights on the face of that bad boy!

mfa298
Posts: 972
Joined: Tue Apr 22, 2014 11:18 am

Re: Compute Module Super Computer/Cluster

Mon Jun 23, 2014 9:19 am

stutteringp0et wrote: I was thinking this would be a great development platform for high-availability services. Rather than invest in servers to test with, one could invest in a single RPiCM unit with any number of modules to test on. For less money, probably, one could have 2 or 3 times as many (CM) servers in a single box drawing the same (or less) power than a single real server.
I had started wondering the same thing. But when I can put together a cheap system for around £300 and run a number of VM's on it for testing stuff I think a rack mounted Pi system would become more expensive and less flexible.

That said it could make an interesting option for VPS type providers. Instead of having a virtual system you rent a CM module which is all yours.

technix
Posts: 105
Joined: Sat Jul 13, 2013 4:55 pm

Re: Compute Module Super Computer/Cluster

Tue Jul 15, 2014 8:51 pm

I have a strip+backplane design, and the entire system takes up a whole 42U rack.

All Compute Modules are mounted to module strips, which also provides USB 2.0 Gigabit Ethernet chips to every Pi (running at ~300Mbps) and a 16-port gigabit switch with uplink ports to the backplane. Every strip can contain 12 CMs and uplink with 4 aggregated GbE ports, connects to the backplane with a 40-pin edge connector that passes both power and networking. Strips are provided 12V power and use switch-mode converters to derive voltages used by CM and other chips.

Backplanes are simply 40-port GbE+8-port 40GbE switches. The 40 GbE ports are grouped into 10 4-port aggregations that connects to the strips, and the 8 40GbE ports are aggregated into 2 4-port sets at each end. Backplanes accepts 120/240V power.
Backplanes only have frames - and a whole rack filled with such frames will have a clear path vertically through the entire rack. So the cooling can be done with one big fans set at the top or the bottom of the rack.

This design crams 120 CMs in an 1U unit, and 4800 CMs to fill a whole 42U rack (1U power distribution and 1U cooling fan set considered)

This rack provides 4800 processors with GPGPU capabilities, 2.4TB RAM and 20TB storage in eMMC chips.

pbgben
Posts: 2
Joined: Sat Apr 12, 2014 12:04 pm

Re: Compute Module Super Computer/Cluster

Tue Jul 15, 2014 9:51 pm

technix wrote:I have a strip+backplane design, and the entire system takes up a whole 42U rack.

All Compute Modules are mounted to module strips, which also provides USB 2.0 Gigabit Ethernet chips to every Pi (running at ~300Mbps) and a 16-port gigabit switch with uplink ports to the backplane. Every strip can contain 12 CMs and uplink with 4 aggregated GbE ports, connects to the backplane with a 40-pin edge connector that passes both power and networking. Strips are provided 12V power and use switch-mode converters to derive voltages used by CM and other chips.

Backplanes are simply 40-port GbE+8-port 40GbE switches. The 40 GbE ports are grouped into 10 4-port aggregations that connects to the strips, and the 8 40GbE ports are aggregated into 2 4-port sets at each end. Backplanes accepts 120/240V power.
Backplanes only have frames - and a whole rack filled with such frames will have a clear path vertically through the entire rack. So the cooling can be done with one big fans set at the top or the bottom of the rack.

This design crams 120 CMs in an 1U unit, and 4800 CMs to fill a whole 42U rack (1U power distribution and 1U cooling fan set considered)

This rack provides 4800 processors with GPGPU capabilities, 2.4TB RAM and 20TB storage in eMMC chips.

WOW,
Are you looking to go into production? I'd love to see this setup in a real environment.

tony-allan
Posts: 4
Joined: Sat Aug 04, 2012 3:17 am
Location: Melbourne, Australia

Re: Compute Module Super Computer/Cluster

Wed Jul 16, 2014 5:44 am

technix wrote:I have a strip+backplane design, and the entire system takes up a whole 42U rack.
Sounds awesome. As much as I would love 4,800 Raspberry Pi's, I have nowhere to put so many :)

For my needs, the ability to use a single strip with one ethernet connection, a single 12v power connection and some USB connectors would be perfect.

Briarios
Posts: 118
Joined: Thu Feb 21, 2013 11:35 am
Location: Perth, Western Australia
Contact: Website

Re: Compute Module Super Computer/Cluster

Wed Jul 16, 2014 8:39 am

thats more doable, 4 or 8 or so on a single board should be more than enough for most people i believe
http://rpi.pmfg.me -- Test board Eagle Cad Files.

mfa298
Posts: 972
Joined: Tue Apr 22, 2014 11:18 am

Re: Compute Module Super Computer/Cluster

Fri Jul 18, 2014 10:34 am

technix wrote:I have a strip+backplane design, and the entire system takes up a whole 42U rack.
...
Backplanes only have frames - and a whole rack filled with such frames will have a clear path vertically through the entire rack. So the cooling can be done with one big fans set at the top or the bottom of the rack.
If you're trying to design something for data centre type people two things to consider. There are various sizes of rack, some places prefer 47U racks, older comms racks may be shorter. Others may only have part of a rack free so something that's more modular (say a self contained 4U unit) could be more popular. Also rack equipment tends to be front to back cooling (so you have a cold aisle at the front of the rack and hot air is pumped out the back)

citrixmeta
Posts: 2
Joined: Wed Aug 27, 2014 6:23 pm

Re: Compute Module Super Computer/Cluster

Sun Sep 07, 2014 8:17 pm

hey guys, check out this project here. Connecting boards using a high speed back plane, built in switch and 1 adapter.

the chassis handles their own 8 core boards along with RPI's

https://www.kickstarter.com/projects/88 ... datacenter

der-don
Posts: 10
Joined: Fri Sep 21, 2012 8:58 am

Re: Compute Module Super Computer/Cluster

Tue Sep 09, 2014 10:01 am

If anyone builds a board for the compute module like this, let me know.

All I'd need is a board for 8-16 PIs, one or two external connectors for the power supply and the 8-16 external gigabit connectors.

User avatar
RaTTuS
Posts: 9489
Joined: Tue Nov 29, 2011 11:12 am
Location: North West UK

Re: Compute Module Super Computer/Cluster

Tue Sep 09, 2014 10:21 am

der-don wrote:If anyone builds a board for the compute module like this, let me know.

All I'd need is a board for 8-16 PIs, one or two external connectors for the power supply and the 8-16 external gigabit connectors.
well you would be better off using them connected via a DM8806 [5 port 10/100mb and one 1G port]
so build a module that has 5 CM's and 1 uplink ,
that way it become more modular ;)
How To ask Questions :- http://www.catb.org/esr/faqs/smart-questions.html
WARNING - some parts of this post may be erroneous YMMV

1QC43qbL5FySu2Pi51vGqKqxy3UiJgukSX
Covfefe

der-don
Posts: 10
Joined: Fri Sep 21, 2012 8:58 am

Re: Compute Module Super Computer/Cluster

Mon Jan 05, 2015 11:32 am

To be honest I didn't think about the details, because I don't have a clue what I have to consider when designing such a board.

Nuno Marques
Posts: 2
Joined: Sat Sep 13, 2014 4:53 pm
Location: Portugal

Re: Compute Module Super Computer/Cluster

Sat Jul 18, 2015 3:36 pm

Can we use the GPIO ports to communicate among some processing nodes?
(ex. http://codeandlife.com/2012/07/03/bench ... pio-speed/ and viewtopic.php?t=50538&p=391909 )

In a worst case scenario, I think this setting may use a lot of cpu. However, there are some settings where it may be cost effective: I am thinking of specific applications for non-von neumann architectures (namely cognitive computing, inspiration by the distributed processing, ex. IBM project on neuro-synaptic chips: http://www.research.ibm.com/cognitive-c ... hips.shtml ). MAYBE, a board to connect/cluster several RPi compute modules could be a great test bed for such a software, namely in more academic/educational frameworks? (maybe even a cost effective solution for some small industrial apps?).

Just an idea...

som
Posts: 1
Joined: Fri Dec 18, 2015 1:56 am

Re: Compute Module Super Computer/Cluster

Fri Dec 18, 2015 2:12 am

My 2 cents to this posts, even it is old.
In our DC one of our customer is running multiple pi cluster with each 48 rpi2 inside 3U rack case. Inside are PSU, switches etc, output are 2 LANs, 2x power input. He spent a lot of time on development of this and took him almost 6 months, now his system is stable as i can see every day on zabbix cause we are monitoring it. So my advise is - if u wanna build cluster - not go the way as much as possible into one case, each module/rpi needs place for management, cooling, take out single rpi for examine ( without affecting other rpi's) if something wrong, the lan cables etc...

janaka_a
Posts: 1
Joined: Mon Jul 31, 2017 6:50 am

Re: Compute Module Super Computer/Cluster

Mon Jul 31, 2017 6:53 am

tony-allan wrote:
Fri May 30, 2014 8:33 am
I like the idea of an 8 or 16 node cluster, with an carrier board with sockets for the raspberry pi's, a USB to 100 m-bit ethernet chip for each pi (just as the existing Model B's use, and an ethernet switch chip on board as well and a 1g-bit ethernet for off board communications.

Not exactly a super computer but should be quite handily for lots of applications at home or school or even in many business environments.

Tony.
I'm this kind of setup has potential also. Just came across this post googling for it.

Imre
Posts: 6
Joined: Fri Jul 14, 2017 8:43 pm
Location: Hungary
Contact: Website

Re: Compute Module Super Computer/Cluster

Mon Sep 04, 2017 10:32 pm

Hi Everybody,

Are there anybody how really built a multi-CM based system? I've read quite many plans and 'wishes', but not a physical one... :)
'Good luck, Mr. Gorsky!' - Neil Armstrong, July 21, 1969, Moon

Return to “Compute Module”

Who is online

Users browsing this forum: No registered users and 3 guests