Heater
Posts: 15777
Joined: Tue Jul 17, 2012 3:02 pm

Oracle's New Supercomputer Has 1,060 Raspberry Pis

Sat Sep 21, 2019 7:31 pm

Never thought I'd see Oracle being kool:

Oracle's New Supercomputer Has 1,060 Raspberry Pis https://www.tomshardware.com/news/oracl ... 40412.html

Image
Memory in C++ is a leaky abstraction .

User avatar
bomblord
Posts: 265
Joined: Sun Jul 14, 2019 2:54 am

Re: Oracle's New Supercomputer Has 1,060 Raspberry Pis

Sat Sep 21, 2019 8:48 pm

oracle engineer wrote:ServeTheHome asked Oracle why it chose to create a cluster of Raspberry Pis instead of using a virtualized Arm server and one company rep said simply that "...a big cluster is cool."

User avatar
davidcoton
Posts: 4872
Joined: Mon Sep 01, 2014 2:37 pm
Location: Cambridge, UK
Contact: Website

Re: Oracle's New Supercomputer Has 1,060 Raspberry Pis

Sat Sep 21, 2019 8:50 pm

bomblord wrote:
Sat Sep 21, 2019 8:48 pm
oracle engineer wrote:ServeTheHome asked Oracle why it chose to create a cluster of Raspberry Pis instead of using a virtualized Arm server and one company rep said simply that "...a big cluster is cool."
It won't be so cool when they upgrade it to use Pi4Bs. :o
Signature retired

User avatar
bomblord
Posts: 265
Joined: Sun Jul 14, 2019 2:54 am

Re: Oracle's New Supercomputer Has 1,060 Raspberry Pis

Sat Sep 21, 2019 8:51 pm

davidcoton wrote:
Sat Sep 21, 2019 8:50 pm
bomblord wrote:
Sat Sep 21, 2019 8:48 pm
oracle engineer wrote:ServeTheHome asked Oracle why it chose to create a cluster of Raspberry Pis instead of using a virtualized Arm server and one company rep said simply that "...a big cluster is cool."
It won't be so cool when they upgrade it to use Pi4Bs. :o
It'll be cooler but too hot for you 8-)

User avatar
Imperf3kt
Posts: 3559
Joined: Tue Jun 20, 2017 12:16 am
Location: Australia

Re: Oracle's New Supercomputer Has 1,060 Raspberry Pis

Sat Sep 21, 2019 9:10 pm

We don't expect this product to go commercial, but it is a really neat example of just how much you can do with a $35 computer.
Except it's not, its an example of what you can do with $37,100 worth of computers plus many thousands of dollars in additional equipment.
The money for the Pi's alone is more than my yearly salary.
55:55:44:44:4C
52:4C:52:42:41

Heater
Posts: 15777
Joined: Tue Jul 17, 2012 3:02 pm

Re: Oracle's New Supercomputer Has 1,060 Raspberry Pis

Sat Sep 21, 2019 9:43 pm

I think it's neat.

Given that you want to show off your clustering software over a 1000 nodes is there a cheaper way to do it?

It's peanuts for Oracle anyway. Never seen so many Porches and Ferraris parked outside an office as the Oracle office in Helsinki.
Memory in C++ is a leaky abstraction .

User avatar
Gavinmc42
Posts: 4480
Joined: Wed Aug 28, 2013 3:31 am

Re: Oracle's New Supercomputer Has 1,060 Raspberry Pis

Sat Sep 21, 2019 11:39 pm

Wonder when they finished it?
One week before the Pi4 was announced?
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

User avatar
rin67630
Posts: 1013
Joined: Fri Mar 04, 2016 10:15 am

Re: Oracle's New Supercomputer Has 1,060 Raspberry Pis

Sun Sep 22, 2019 1:56 am

I hope, once they have finished to play with it, that they will donate the Pis to a school, where they belong to.

User avatar
rin67630
Posts: 1013
Joined: Fri Mar 04, 2016 10:15 am

Re: Oracle's New Supercomputer Has 1,060 Raspberry Pis

Sun Sep 22, 2019 2:23 am

I have seen speed tests finding out that USB network adapters were up to five time faster than the built in network socket.
Would the cluster not have been much more efficient if they had used USB communication?

User avatar
rpdom
Posts: 16968
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: Oracle's New Supercomputer Has 1,060 Raspberry Pis

Sun Sep 22, 2019 2:50 am

rin67630 wrote:
Sun Sep 22, 2019 2:23 am
I have seen speed tests finding out that USB network adapters were up to five time faster than the built in network socket.
Would the cluster not have been much more efficient if they had used USB communication?
The built in network socket on the 3B+ IS a USB network adaptor (as were the ones on all B models prior to that).

The speed tests you refer to are probably comparing the 1/2/3B's 100Mb Ethernet port with a 1000Mb USB adaptor, which will give improvements of over three times the throughput.

That doesn't apply to the 3B+ which has a 1000Mb Ethernet port on board, although it is still restricted by the USB2.0 bus speeds.
Unreadable squiggle

ejolson
Posts: 5107
Joined: Tue Mar 18, 2014 11:47 am

Re: Oracle's New Supercomputer Has 1,060 Raspberry Pis

Sun Sep 22, 2019 3:21 am

rin67630 wrote:
Sun Sep 22, 2019 2:23 am
I have seen speed tests finding out that USB network adapters were up to five time faster than the built in network socket.
Would the cluster not have been much more efficient if they had used USB communication?
In a way I'm disappointed Oracle didn't build their cluster using SPARC-based single board computers. Maybe the Raspberry SPARC was hastily cancelled after the Pi 4B released.

It would be interesting if, instead of monetising the SPARC architecture to the point of market extinction, Oracle had commoditised it the way IBM is currently doing with the OpenPower initiative. While a similar statement could be said about the DEC Alpha, Compaq has already learned the lesson of such extreme waste by going out of business and being acquired by HP.

Since the gigabit Ethernet port on the Pi 3B+ is capable of saturating the USB2 bus, it would be difficult to create an interconnect for the 3B+ that was any faster. On the other hand, it may be possible to beat the gigabit Ethernet of the 4B using a custom USB3-based networking fabric. Although I can't imagine any qualified engineering team thinking such a project worthwhile, it's possible the next Oracle Pi cluster will do exactly that.

Heater
Posts: 15777
Joined: Tue Jul 17, 2012 3:02 pm

Re: Oracle's New Supercomputer Has 1,060 Raspberry Pis

Sun Sep 22, 2019 5:26 am

Looks like a super computer to me. I have only ever seen one super computer in the flesh and it looks like it. This one: https://www.computerhistory.org/projects/cray2/

Image

Oh, and I saw a sibling in the museum in Berlin:
Image

I'm sure the Oracle machine (Orac ?) can out perform them.

Anyway, it looks great, love the colors. Reminds me of the Tardis. Amazingly most of the space inside seems to be empty.
Memory in C++ is a leaky abstraction .


jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 26378
Joined: Sat Jul 30, 2011 7:41 pm

Re: Oracle's New Supercomputer Has 1,060 Raspberry Pis

Sun Sep 22, 2019 9:43 am

jcyr wrote:
Sun Sep 22, 2019 3:14 am
Hardly a supercomputer, more like a silly PR stunt!
Completely wrong. Simply because the definition of supercomputer change every year. The Cray 1 was a supercomputer. The Pi4 is faster than a
Cray 1. Does that stop the Cray 1 from being a supercomputer? Does a supercar from the 60's stop being a supercar? No.

And most certainly not a publicity stunt. This is a means of testing software destined for top of the performance league computers wich cost many millions, without having to waste development time on those very devices. Debug your distributed code on a Pi cluster, very cheaply, then move it to the real workhorse. That would pay for this cluster in a matter of days. Modern supercomputer time is expensive.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed.
I've been saying "Mucho" to my Spanish friend a lot more lately. It means a lot to him.

tenochtitlanuk
Posts: 156
Joined: Fri Jul 06, 2012 8:51 pm
Location: Taunton, Somerset, UK
Contact: Website

Re: Oracle's New Supercomputer Has 1,060 Raspberry Pis

Sun Sep 22, 2019 10:14 am

A Cray was the most comfortable computer I ever sat on. Piś are too prickly and need an extra cushion....

User avatar
DougieLawson
Posts: 38773
Joined: Sun Jun 16, 2013 11:19 pm
Location: A small cave in deepest darkest Basingstoke, UK
Contact: Website Twitter

Re: Oracle's New Supercomputer Has 1,060 Raspberry Pis

Sun Sep 22, 2019 10:22 am

jamesh wrote:
Sun Sep 22, 2019 9:43 am
And most certainly not a publicity stunt. This is a means of testing software destined for top of the performance league computers wich cost many millions, without having to waste development time on those very devices. Debug your distributed code on a Pi cluster, very cheaply, then move it to the real workhorse. That would pay for this cluster in a matter of days. Modern supercomputer time is expensive.
The fact that you need to explain this proves that most folks don't have a clue about how supercomputers work.

It's not like you boot Windows on it. You need a specialised workload and some other peripheral computers to support getting the workload split into chunks and distributed to the supercomputer. Even the Cray Twins at the old weather centre in Bracknell had a plain old IBM S/370 mainframe in-front of them as the peripheral computer to handle getting the workload running, storing data on tapes & disks and printing reports on 3211 line printers.

It's a very specialised program that runs on a supercomputer and building a massive RPi cluster to learn how that piece works seems like perfect sense.
Note: Any requirement to use a crystal ball or mind reading will result in me ignoring your question.

Criticising any questions is banned on this forum.

Any DMs sent on Twitter will be answered next month.
All non-medical doctors are on my foes list.

User avatar
davidcoton
Posts: 4872
Joined: Mon Sep 01, 2014 2:37 pm
Location: Cambridge, UK
Contact: Website

Re: Oracle's New Supercomputer Has 1,060 Raspberry Pis

Sun Sep 22, 2019 10:58 am

tenochtitlanuk wrote:
Sun Sep 22, 2019 10:14 am
A Cray was the most comfortable computer I ever sat on. Piś are too prickly and need an extra cushion....
Is that where the term "hot seat" came from? :lol:
Signature retired

ejolson
Posts: 5107
Joined: Tue Mar 18, 2014 11:47 am

Re: Oracle's New Supercomputer Has 1,060 Raspberry Pis

Sun Sep 22, 2019 4:23 pm

jcyr wrote:
Sun Sep 22, 2019 2:16 pm
jamesh wrote:
Sun Sep 22, 2019 9:43 am
Completely wrong. Simply because the definition of supercomputer change every year. The Cray 1 was a supercomputer. The Pi4 is faster than a
Cray 1. Does that stop the Cray 1 from being a supercomputer? Does a supercar from the 60's stop being a supercar? No.
They are not calling a 50 year old design a supercomputer, they are calling a current design a supercomputer. Furthermore, the super slow interconnect matrix using USB 2.0 constrained EThernet I would consider a disqualifier.
Now that the Pi 4B has true [edit] gigabit Ethernet, it makes a much more reasonable machine to build clusters out of. It would be amusing to see a Raspberry Pi 4B entry in the student cluster competition at ISC 2020. In the past the goal has been to run a number of computational codes as fast as possible using less than 3kW. As each Pi uses about 5W, that would be about 600 Raspberry Pi computers.

As this is an educational event promoting STEM disciplines, it makes good sense for the foundation to be an equipment sponsor. The deadline for universities to submit team proposals is November 8. Part of a team proposal includes specifying the desired equipment that will be used during the competition.

If Cambridge has difficulty rounding up a team of students that want to enter the competition with a Pi cluster, it might be possible to find a team here in the liberal frontier. Every year we've thought about it, but never submitted a proposal.
Last edited by ejolson on Sun Sep 22, 2019 4:42 pm, edited 1 time in total.

User avatar
rpdom
Posts: 16968
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: Oracle's New Supercomputer Has 1,060 Raspberry Pis

Sun Sep 22, 2019 4:37 pm

ejolson wrote:
Sun Sep 22, 2019 4:23 pm
Now that the Pi 4B has true Ethernet
I think you missed out a "Gigabit" in there ;)
All of the Pi's have had "true Ethernet" of various speeds.
Unreadable squiggle

ejolson
Posts: 5107
Joined: Tue Mar 18, 2014 11:47 am

Re: Oracle's New Supercomputer Has 1,060 Raspberry Pis

Sun Sep 22, 2019 4:43 pm

rpdom wrote:
Sun Sep 22, 2019 4:37 pm
ejolson wrote:
Sun Sep 22, 2019 4:23 pm
Now that the Pi 4B has true Ethernet
I think you missed out a "Gigabit" in there ;)
All of the Pi's have had "true Ethernet" of various speeds.
Good catch. I've fixed it.

If we write a proposal, would you be willing to proofread it?

User avatar
clicky
Posts: 452
Joined: Thu Oct 25, 2012 7:34 am

Re: Oracle's New Supercomputer Has 1,060 Raspberry Pis

Sun Sep 22, 2019 4:50 pm

It almost makes no difference using USB2 or gigabit ethernet as both demonstrate principle (and problems) such configurations (let's call them 'super computer' in this instance) exhibit: communication between processing cores is always going to be slow (in comparison to CPU <- > memory) when you scale it to thousands of such cores.

This 1000 RPis (super or not) computer thingy is perfect for experimenting and learning how to deal with that problem. Plus gives (someone?) factor of 1000 of CPU power over regular Raspberry Pis (given problem 'solution' is properly applied to such configuration).

Back to pedantic: is it super or not so super computer question.

What are requirements for something to be a general purpose computer? Would one disqualify RPi (let's say zero) for providing only one USB2 port, no networking (not PiZero W), not fast bus to drive storage from, no RTC, no sound, etc... And yet, we still call it computer because it has CPU, memory, I/O, storage and ability for other peripherals to be added. And runs nicely proper Linux kernel (only as a benchmark of good OS no other reason or excluding other 'proper' OSes)... Using similar principles, I would call this thing above 'super computer' because it fits some (intuitive?) definition of what super computer is over general purpose computer. Does it not? ;)

Heater
Posts: 15777
Joined: Tue Jul 17, 2012 3:02 pm

Re: Oracle's New Supercomputer Has 1,060 Raspberry Pis

Sun Sep 22, 2019 4:55 pm

You need to get some students on the case. Your PDEWulf cluster seems to be a bit neglected: http://fractal.math.unr.edu/~ejolson/ap ... owulf.html

I'm wondering what one can do with 25000 GPIOs ? :)
Memory in C++ is a leaky abstraction .

User avatar
rpdom
Posts: 16968
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: Oracle's New Supercomputer Has 1,060 Raspberry Pis

Sun Sep 22, 2019 5:18 pm

ejolson wrote:
Sun Sep 22, 2019 4:43 pm
rpdom wrote:
Sun Sep 22, 2019 4:37 pm
ejolson wrote:
Sun Sep 22, 2019 4:23 pm
Now that the Pi 4B has true Ethernet
I think you missed out a "Gigabit" in there ;)
All of the Pi's have had "true Ethernet" of various speeds.
Good catch. I've fixed it.

If we write a proposal, would you be willing to proofread it?
Sure. I'll also run it past my wife, who knows nothing about Linux or Pi, but will spot a grammatical error at 1000 paces. ;-)
Unreadable squiggle

ejolson
Posts: 5107
Joined: Tue Mar 18, 2014 11:47 am

Re: Oracle's New Supercomputer Has 1,060 Raspberry Pis

Sun Sep 22, 2019 5:22 pm

clicky wrote:
Sun Sep 22, 2019 4:50 pm
I would call this thing above 'super computer' because it fits some (intuitive?) definition of what super computer is over general purpose computer. Does it not? ;)
The intuitive definition of a supercomputer is that at the time of delivery it is among the fastest machines in the world. Not so intuitively, the performance cutoff is to be one of the 500 best performing machines in the world when solving huge systems of linear equations. Everything else is better called a cluster.

From this point of view, clusters come in different styles: Some are designed for high-performance scientific computing, some designed for big-data mining operations, some for cloud, some for learning, some for distributed computing research, some for software development including the creation of cluster management and provisioning software. It seems the Oracle Pi cluster falls into the last category just mentioned.

One of the amusing engineering problems related to cluster computing is that the fault rate of current hardware makes it difficult to successfully boot enough nodes in a supercomputer to reach exascale speeds. This has promoted fault-tolerant cluster provisioning algorithms and software designs that, until recently, weren't needed in high-performance computing. One of the uses for low-performance clusters of Raspberry Pi computers is to serve as real-hardware development environments for fault-tolerant parallel processing.
Last edited by ejolson on Sun Sep 22, 2019 6:09 pm, edited 4 times in total.

ejolson
Posts: 5107
Joined: Tue Mar 18, 2014 11:47 am

Re: Oracle's New Supercomputer Has 1,060 Raspberry Pis

Sun Sep 22, 2019 5:37 pm

Heater wrote:
Sun Sep 22, 2019 4:55 pm
You need to get some students on the case. Your PDEWulf cluster seems to be a bit neglected: http://fractal.math.unr.edu/~ejolson/ap ... owulf.html

I'm wondering what one can do with 25000 GPIOs ? :)
That page is a bit out of date. Fortunately, the department cluster was dismantled to make room for office space and observe fire codes that weren't properly implemented in the new building.

We still maintain a few department servers on campus, but most large-scale computation is performed on the university cluster which is conveniently housed nearby at the largest, most advanced data center in the world.

Things would have been a lot better, however, if they hadn't broken that fancy Intel switch during a system upgrade earlier this year. I'm happy to say that my Pi Zero cluster was unaffected.
Last edited by ejolson on Sun Sep 22, 2019 6:01 pm, edited 2 times in total.

Return to “General discussion”