Page 1 of 1

Compute Module 3 Cluster

Posted: Fri Apr 15, 2016 10:22 am
by mkelly99
I developed a multiple CM cluster a few months back, but held off fabbing. Once I saw the reference to the CM3 in the Pi 3 announcement I went ahead and sent it to the fab house. Should be back in a few weeks. I am looking to get some CM3's as soon as possible, but we'll debug using CM1. Note that we used the CM1 as the board management controller in a previous x86 cluster design, so we know that works pretty well.

I have been thinking of open sourcing this carrier and the blades that hold the CM. We are not really in the business of servers so we know it ill take some collaboration to bring it to market. Plus we want to make it available to as many folks as possible.
501m10_viewa1.jpg
501m10_viewa1.jpg (49.66 KiB) Viewed 20730 times
The carrier has ten blades and one CM for management. It has two cross coupled 8-Port GIGe switches with four out-facing ethernet ports. Each blade has the CM, an M.2 SSD socket and a GIGe controller. The management CM has an HDMI and USB connector plus can be programmed via the Micro-USB.
blade_capture.JPG
blade_capture.JPG (58.27 KiB) Viewed 20730 times
I also have a 3D pdf of both boards that anyone can have to let you rotate, zoom in, etc.

I am looking for feedback and if anyone is interested in getting involved let me know! We plan to build 5 or 6 alpha units for initial testing. My work email is [email protected].

Note - I originally posted this under the CM3 thread, but I think it should be on it's own.

Re: Compute Module 3 Cluster

Posted: Sat Apr 16, 2016 8:58 pm
by adun
Interesting project!
The communication beetwen the CMs is done over ethernet right. What speed benefit do you get from using GIGe controller? As it goes still over the USB 2 port.
And what does the board management controller CM exactly do?

Re: Compute Module 3 Cluster

Posted: Sat Apr 16, 2016 10:05 pm
by mkelly99
The BMC interfaces to each CM over USB for programming. It uses GPIO to control power and reset. It also takes the CM UART and allows remote console access.

The BMC programs the GIGe switches and has a 10/100 link into one of them. The BMC has a USB host port and HDMI for local use.

As for running GIGe over USB 2.0, we measure IPERF at a bit over 300Mbps using the Realtek controller. That is close to 4 times the IPERF throughput of a 100Mbit link.

I will be creating a block diagram in the next day or so and will post it here and on our web site www.pi2design.com under upcoming products.

Re: Compute Module 3 Cluster

Posted: Mon Apr 18, 2016 7:47 am
by Akane
Can you tell me more about the board? What is the chips which have many pins next to each Raspberry Pi Compute Module?

Re: Compute Module 3 Cluster

Posted: Mon Apr 18, 2016 3:25 pm
by mkelly99
To the right of the CM is an M.2 SSD socket. we are showing an example of an M.2 SSD Module loaded into the socket. This was done to give each CM it's own local storage.

Michael

Re: Compute Module 3 Cluster

Posted: Mon Apr 18, 2016 11:50 pm
by Akane
Thanks! That's very interesting because we are also developing a cluster board which uses Raspberry Pi's: https://twitter.com/9_ties/status/702330841783906304

Re: Compute Module 3 Cluster

Posted: Tue Apr 19, 2016 11:32 am
by mkelly99
Yes, we saw your announcement awhile ago. We had looked at doing something similar, since we already had the Compute Module design, plus we have designed some Pi-Zero Hats (http://www.pi2design.com/store/p10/504D ... _Zero.html and http://www.pi2design.com/504ssd-solid-s ... -zero.html), but decided the Compute Module was more like what we had already done with our x86 cluster. Hopefully we can both succeed!

Michael

Re: Compute Module 3 Cluster

Posted: Tue Apr 19, 2016 11:36 am
by mkelly99
As mentioned earlier, we have a block diagram of the 501M10 carrier and the 501CMX blade. The link is http://www.pi2design.com/uploads/4/8/5/ ... _block.pdf. The 3D PDF for the 501M10 is here:
http://www.pi2design.com/uploads/4/8/5/ ... _3d_p1.pdf (warning 21MB!) and the501CMX is here:
http://www.pi2design.com/uploads/4/8/5/ ... _3d_p1.pdf

You will need actual Adobe Acrobat reader. Clones won't open 3D PDF's (AFAIK).

Michael

Re: Compute Module 3 Cluster

Posted: Wed Apr 20, 2016 8:27 am
by rln
Your designs are clearly industrial :)
You may be interested in my SW project Nard SDK which is focused on HW like this. Raspberry Pi in embedded systems, running day and night for years in industrial systems.
http://www.arbetsmyra.dyndns.org/nard/

Re: Compute Module 3 Cluster

Posted: Wed Apr 20, 2016 2:22 pm
by DrML
Interesting design.

As always, a good question is... How much?

Also, how does the cost of the blade 'carrier' compare to the cost of populating the extra components directly on the motherboard? (Still keeping the CM and SSD mounted vertically but held only by their vertical mount connectors.)

And is there enough clearance to add a heatsink to each CM for the hot RasPi3?

Thanks,
Martin

Re: Compute Module 3 Cluster

Posted: Wed Apr 20, 2016 7:49 pm
by mkelly99
Thanks Martin!

Price is still TBD, but we expect the carrier to be under $300 (with a CM1 or CM3) and the blade boards under $70 (w/o CM3). Add in the cost of the CM, but without an SSD yields around $1.4K assuming the CM 3 is still under $40. That's ~$1.4K for forty 1.2Ghz 64-Bit ARM Cortex-A53 cores!

As for direct mounting the compute modules and the SSD, there were a few issues we looked at. The first was that there was insufficient space for routing the 10 sets of USB, control signals and GIGe in a 4-layer PCB. Because we need controlled impedance for the USB and GIGe we would have had to go to an 8-layer PCB. The second issue is that there are no vertical mount SSD standards, neither mSATA or M.2. SATA Disk On Module (DOM) is a very loose standard and there is no mechanical compatibility. Third, the vertical SODIMM connectors are quite pricey (and not in wide supply) at around $5 each, which is about the cost of the blade PCB.

Finally we felt this would give us some flexibility as to what is actual mounted. We could see doing a different blade with a 64-Bit ARM SoC directly on the blade as opposed to a module.

All in all, it felt like the right set of compromises. And hey, isn't that really what engineering is all about! :)

As for clearance, the distance from the top surface of the CM to the backside of the next blade is 9mm. Tight, but a small 6-8mm top hat will fit. There are no components on the backside of the blade. This allows the typical pi heatsink (like this one we plan to use from pimoroni https://shop.pimoroni.com/products/heatsink) to fit.

We will be mounting this in the M350 Mini-ITX case (http://www.mini-box.com/M350-universal- ... -enclosure) with a couple of 40mm x 10mm fans overhead. Add in an external 12V @10A+ supply and it will all be good to go! Just as long as we can get CM3's!

If anyone has any feedback, questions, comments, etc do not hesitate. This is why we made this post. :)

Michael

Re: Compute Module 3 Cluster

Posted: Thu Apr 28, 2016 11:44 am
by mkelly99
Update - The blank PCB's for the carrier and the blades have arrived. It will take a few weeks to gather up the parts to build a couple of carriers and 20 or so blades. Then of course, we need to wait on the Compute Module 3. Hopefully when they come out they won't become hard to get. If anyone at the foundation wants to do some barter, we'd be happy to trade a system or two for a bunch of compute modules! :)

Michael

Re: Compute Module 3 Cluster

Posted: Sat Dec 03, 2016 5:32 pm
by tim_lossen
hello mike,

so the compute module 3 is (finally) expected to be released "to the general public by the end of the year".

are you still working on the cluster?

cheers
tim

Re: Compute Module 3 Cluster

Posted: Wed Dec 14, 2016 3:51 pm
by laurent
Hello

Have you considered a cluster of the forthcoming Pi 3 A ? It could be a cheapest solution. The only available USB could be used as an ethernet gadget device, like with the existing A+ and zero. The only weak point is the need of a µSD card (not as reliable as the eMMC).

Re: Compute Module 3 Cluster

Posted: Wed Dec 14, 2016 4:31 pm
by fruitoftheloom
laurent wrote:Hello

Have you considered a cluster of the forthcoming Pi 3 A ? It could be a cheapest solution. The only available USB could be used as an ethernet gadget device, like with the existing A+ and zero. The only weak point is the need of a µSD card (not as reliable as the eMMC).
FYI the RPi 1A+ by default can not be in put Gadget Mode :)

Re: Compute Module 3 Cluster

Posted: Wed Dec 14, 2016 7:13 pm
by laurent
fruitoftheloom wrote:
laurent wrote:Hello

Have you considered a cluster of the forthcoming Pi 3 A ? It could be a cheapest solution. The only available USB could be used as an ethernet gadget device, like with the existing A+ and zero. The only weak point is the need of a µSD card (not as reliable as the eMMC).
FYI the RPi 1A+ by default can not be in put Gadget Mode :)
My bad, I didn't know that :oops:
So sad, my dream of a cheap cluster is flying out :cry:

Re: Compute Module 3 Cluster

Posted: Wed Dec 14, 2016 7:23 pm
by fruitoftheloom
laurent wrote:
fruitoftheloom wrote: FYI the RPi 1A+ by default can not be in put Gadget Mode :)
My bad, I didn't know that :oops:
http://blog.gbaman.info/?p=791

Whether or not the RPi 3A will have Gadget Mode no one apart from RPT knows !!

But this has no relevance to the Original Poster who is using Compute Modules :shock:

Re: Compute Module 3 Cluster

Posted: Wed Dec 14, 2016 7:30 pm
by rpdom
fruitoftheloom wrote:Whether or not the RPi 3A will have Gadget Mode no one apart from RPT knows !!
If it sticks to the current form factor, it won't. The standard USB sockets don't have the ID connection to enable it. Switching to a MicroUSB socket would break compatibility with existing products that use the A+ form (that's making the assumption that the 3A will replace the A+, I dunno.).

Re: Compute Module 3 Cluster

Posted: Wed Dec 14, 2016 8:07 pm
by gregeric
Gadget mode on Pi A has been possible since this commit thanks to prudy. See https://github.com/raspberrypi/documentation/issues/324

tldr: in the absence of the OTG_sense pin, you can use an override in config.txt to force gadget mode when using a USB A-A cable on the A.

Re: Compute Module 3 Cluster

Posted: Thu Dec 15, 2016 8:13 am
by laurent
gregeric wrote:Gadget mode on Pi A has been possible since this commit thanks to prudy. See https://github.com/raspberrypi/documentation/issues/324

tldr: in the absence of the OTG_sense pin, you can use an override in config.txt to force gadget mode when using a USB A-A cable on the A.
Many thanks for this information :D
I'll give it a try very soon !