mkelly99
Posts: 22
Joined: Wed Apr 06, 2016 2:17 pm

Compute Module 3 Cluster

Fri Apr 15, 2016 10:22 am

I developed a multiple CM cluster a few months back, but held off fabbing. Once I saw the reference to the CM3 in the Pi 3 announcement I went ahead and sent it to the fab house. Should be back in a few weeks. I am looking to get some CM3's as soon as possible, but we'll debug using CM1. Note that we used the CM1 as the board management controller in a previous x86 cluster design, so we know that works pretty well.

I have been thinking of open sourcing this carrier and the blades that hold the CM. We are not really in the business of servers so we know it ill take some collaboration to bring it to market. Plus we want to make it available to as many folks as possible.
501m10_viewa1.jpg
501m10_viewa1.jpg (49.66 KiB) Viewed 18107 times
The carrier has ten blades and one CM for management. It has two cross coupled 8-Port GIGe switches with four out-facing ethernet ports. Each blade has the CM, an M.2 SSD socket and a GIGe controller. The management CM has an HDMI and USB connector plus can be programmed via the Micro-USB.
blade_capture.JPG
blade_capture.JPG (58.27 KiB) Viewed 18107 times
I also have a 3D pdf of both boards that anyone can have to let you rotate, zoom in, etc.

I am looking for feedback and if anyone is interested in getting involved let me know! We plan to build 5 or 6 alpha units for initial testing. My work email is [email protected].

Note - I originally posted this under the CM3 thread, but I think it should be on it's own.

User avatar
adun
Posts: 102
Joined: Fri Mar 20, 2015 9:25 am
Location: Switzerland

Re: Compute Module 3 Cluster

Sat Apr 16, 2016 8:58 pm

Interesting project!
The communication beetwen the CMs is done over ethernet right. What speed benefit do you get from using GIGe controller? As it goes still over the USB 2 port.
And what does the board management controller CM exactly do?

mkelly99
Posts: 22
Joined: Wed Apr 06, 2016 2:17 pm

Re: Compute Module 3 Cluster

Sat Apr 16, 2016 10:05 pm

The BMC interfaces to each CM over USB for programming. It uses GPIO to control power and reset. It also takes the CM UART and allows remote console access.

The BMC programs the GIGe switches and has a 10/100 link into one of them. The BMC has a USB host port and HDMI for local use.

As for running GIGe over USB 2.0, we measure IPERF at a bit over 300Mbps using the Realtek controller. That is close to 4 times the IPERF throughput of a 100Mbit link.

I will be creating a block diagram in the next day or so and will post it here and on our web site www.pi2design.com under upcoming products.

User avatar
Akane
Posts: 41
Joined: Tue May 27, 2014 1:20 pm
Location: Tsukuba, Japan

Re: Compute Module 3 Cluster

Mon Apr 18, 2016 7:47 am

Can you tell me more about the board? What is the chips which have many pins next to each Raspberry Pi Compute Module?

mkelly99
Posts: 22
Joined: Wed Apr 06, 2016 2:17 pm

Re: Compute Module 3 Cluster

Mon Apr 18, 2016 3:25 pm

To the right of the CM is an M.2 SSD socket. we are showing an example of an M.2 SSD Module loaded into the socket. This was done to give each CM it's own local storage.

Michael

User avatar
Akane
Posts: 41
Joined: Tue May 27, 2014 1:20 pm
Location: Tsukuba, Japan

Re: Compute Module 3 Cluster

Mon Apr 18, 2016 11:50 pm

Thanks! That's very interesting because we are also developing a cluster board which uses Raspberry Pi's: https://twitter.com/9_ties/status/702330841783906304

mkelly99
Posts: 22
Joined: Wed Apr 06, 2016 2:17 pm

Re: Compute Module 3 Cluster

Tue Apr 19, 2016 11:32 am

Yes, we saw your announcement awhile ago. We had looked at doing something similar, since we already had the Compute Module design, plus we have designed some Pi-Zero Hats (http://www.pi2design.com/store/p10/504D ... _Zero.html and http://www.pi2design.com/504ssd-solid-s ... -zero.html), but decided the Compute Module was more like what we had already done with our x86 cluster. Hopefully we can both succeed!

Michael

mkelly99
Posts: 22
Joined: Wed Apr 06, 2016 2:17 pm

Re: Compute Module 3 Cluster

Tue Apr 19, 2016 11:36 am

As mentioned earlier, we have a block diagram of the 501M10 carrier and the 501CMX blade. The link is http://www.pi2design.com/uploads/4/8/5/ ... _block.pdf. The 3D PDF for the 501M10 is here:
http://www.pi2design.com/uploads/4/8/5/ ... _3d_p1.pdf (warning 21MB!) and the501CMX is here:
http://www.pi2design.com/uploads/4/8/5/ ... _3d_p1.pdf

You will need actual Adobe Acrobat reader. Clones won't open 3D PDF's (AFAIK).

Michael

rln
Posts: 172
Joined: Wed Apr 09, 2014 1:43 pm
Location: Sweden
Contact: Website

Re: Compute Module 3 Cluster

Wed Apr 20, 2016 8:27 am

Your designs are clearly industrial :)
You may be interested in my SW project Nard SDK which is focused on HW like this. Raspberry Pi in embedded systems, running day and night for years in industrial systems.
http://www.arbetsmyra.dyndns.org/nard/
Last edited by rln on Wed Apr 20, 2016 3:23 pm, edited 1 time in total.
Author of the robust Nard distro http://www.nard.se

DrML
Posts: 2
Joined: Wed Apr 20, 2016 2:15 pm

Re: Compute Module 3 Cluster

Wed Apr 20, 2016 2:22 pm

Interesting design.

As always, a good question is... How much?

Also, how does the cost of the blade 'carrier' compare to the cost of populating the extra components directly on the motherboard? (Still keeping the CM and SSD mounted vertically but held only by their vertical mount connectors.)

And is there enough clearance to add a heatsink to each CM for the hot RasPi3?

Thanks,
Martin

mkelly99
Posts: 22
Joined: Wed Apr 06, 2016 2:17 pm

Re: Compute Module 3 Cluster

Wed Apr 20, 2016 7:49 pm

Thanks Martin!

Price is still TBD, but we expect the carrier to be under $300 (with a CM1 or CM3) and the blade boards under $70 (w/o CM3). Add in the cost of the CM, but without an SSD yields around $1.4K assuming the CM 3 is still under $40. That's ~$1.4K for forty 1.2Ghz 64-Bit ARM Cortex-A53 cores!

As for direct mounting the compute modules and the SSD, there were a few issues we looked at. The first was that there was insufficient space for routing the 10 sets of USB, control signals and GIGe in a 4-layer PCB. Because we need controlled impedance for the USB and GIGe we would have had to go to an 8-layer PCB. The second issue is that there are no vertical mount SSD standards, neither mSATA or M.2. SATA Disk On Module (DOM) is a very loose standard and there is no mechanical compatibility. Third, the vertical SODIMM connectors are quite pricey (and not in wide supply) at around $5 each, which is about the cost of the blade PCB.

Finally we felt this would give us some flexibility as to what is actual mounted. We could see doing a different blade with a 64-Bit ARM SoC directly on the blade as opposed to a module.

All in all, it felt like the right set of compromises. And hey, isn't that really what engineering is all about! :)

As for clearance, the distance from the top surface of the CM to the backside of the next blade is 9mm. Tight, but a small 6-8mm top hat will fit. There are no components on the backside of the blade. This allows the typical pi heatsink (like this one we plan to use from pimoroni https://shop.pimoroni.com/products/heatsink) to fit.

We will be mounting this in the M350 Mini-ITX case (http://www.mini-box.com/M350-universal- ... -enclosure) with a couple of 40mm x 10mm fans overhead. Add in an external 12V @10A+ supply and it will all be good to go! Just as long as we can get CM3's!

If anyone has any feedback, questions, comments, etc do not hesitate. This is why we made this post. :)

Michael

mkelly99
Posts: 22
Joined: Wed Apr 06, 2016 2:17 pm

Re: Compute Module 3 Cluster

Thu Apr 28, 2016 11:44 am

Update - The blank PCB's for the carrier and the blades have arrived. It will take a few weeks to gather up the parts to build a couple of carriers and 20 or so blades. Then of course, we need to wait on the Compute Module 3. Hopefully when they come out they won't become hard to get. If anyone at the foundation wants to do some barter, we'd be happy to trade a system or two for a bunch of compute modules! :)

Michael

tim_lossen
Posts: 1
Joined: Sat Dec 03, 2016 5:18 pm

Re: Compute Module 3 Cluster

Sat Dec 03, 2016 5:32 pm

hello mike,

so the compute module 3 is (finally) expected to be released "to the general public by the end of the year".

are you still working on the cluster?

cheers
tim

laurent
Posts: 274
Joined: Thu Jul 26, 2012 11:24 am

Re: Compute Module 3 Cluster

Wed Dec 14, 2016 3:51 pm

Hello

Have you considered a cluster of the forthcoming Pi 3 A ? It could be a cheapest solution. The only available USB could be used as an ethernet gadget device, like with the existing A+ and zero. The only weak point is the need of a µSD card (not as reliable as the eMMC).

fruitoftheloom
Posts: 19483
Joined: Tue Mar 25, 2014 12:40 pm
Location: Delightful Dorset

Re: Compute Module 3 Cluster

Wed Dec 14, 2016 4:31 pm

laurent wrote:Hello

Have you considered a cluster of the forthcoming Pi 3 A ? It could be a cheapest solution. The only available USB could be used as an ethernet gadget device, like with the existing A+ and zero. The only weak point is the need of a µSD card (not as reliable as the eMMC).
FYI the RPi 1A+ by default can not be in put Gadget Mode :)
adieu

My other Computer is an Asus CS10 ChromeBit running Chrome Operating System.
HP Envy 4500 Wireless Printer supported by HPLIP software in Raspbian Stretch.

laurent
Posts: 274
Joined: Thu Jul 26, 2012 11:24 am

Re: Compute Module 3 Cluster

Wed Dec 14, 2016 7:13 pm

fruitoftheloom wrote:
laurent wrote:Hello

Have you considered a cluster of the forthcoming Pi 3 A ? It could be a cheapest solution. The only available USB could be used as an ethernet gadget device, like with the existing A+ and zero. The only weak point is the need of a µSD card (not as reliable as the eMMC).
FYI the RPi 1A+ by default can not be in put Gadget Mode :)
My bad, I didn't know that :oops:
So sad, my dream of a cheap cluster is flying out :cry:

fruitoftheloom
Posts: 19483
Joined: Tue Mar 25, 2014 12:40 pm
Location: Delightful Dorset

Re: Compute Module 3 Cluster

Wed Dec 14, 2016 7:23 pm

laurent wrote:
fruitoftheloom wrote: FYI the RPi 1A+ by default can not be in put Gadget Mode :)
My bad, I didn't know that :oops:
http://blog.gbaman.info/?p=791

Whether or not the RPi 3A will have Gadget Mode no one apart from RPT knows !!

But this has no relevance to the Original Poster who is using Compute Modules :shock:
adieu

My other Computer is an Asus CS10 ChromeBit running Chrome Operating System.
HP Envy 4500 Wireless Printer supported by HPLIP software in Raspbian Stretch.

User avatar
rpdom
Posts: 14081
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: Compute Module 3 Cluster

Wed Dec 14, 2016 7:30 pm

fruitoftheloom wrote:Whether or not the RPi 3A will have Gadget Mode no one apart from RPT knows !!
If it sticks to the current form factor, it won't. The standard USB sockets don't have the ID connection to enable it. Switching to a MicroUSB socket would break compatibility with existing products that use the A+ form (that's making the assumption that the 3A will replace the A+, I dunno.).

gregeric
Posts: 1508
Joined: Mon Nov 28, 2011 10:08 am

Re: Compute Module 3 Cluster

Wed Dec 14, 2016 8:07 pm

Gadget mode on Pi A has been possible since this commit thanks to prudy. See https://github.com/raspberrypi/documentation/issues/324

tldr: in the absence of the OTG_sense pin, you can use an override in config.txt to force gadget mode when using a USB A-A cable on the A.

laurent
Posts: 274
Joined: Thu Jul 26, 2012 11:24 am

Re: Compute Module 3 Cluster

Thu Dec 15, 2016 8:13 am

gregeric wrote:Gadget mode on Pi A has been possible since this commit thanks to prudy. See https://github.com/raspberrypi/documentation/issues/324

tldr: in the absence of the OTG_sense pin, you can use an override in config.txt to force gadget mode when using a USB A-A cable on the A.
Many thanks for this information :D
I'll give it a try very soon !

Return to “Compute Module”