macbug
Posts: 12
Joined: Thu Dec 04, 2014 8:50 pm

Architechture of a Beowulf cluster.

Wed Jan 23, 2019 6:03 pm

I aim to build a Beowulf cluster out of 4 Raspberry PI's 3 B+ this weekend. I have started to build an image which i will use on all four RPI's. I will from now refer to them as controller and nodes. Each of controller and nodes will have an identical image on them. The only thing that will be different is the hostname, and they will be named controller, node1, node2 and node3. The image i have been building on is an "Debian Stretch lite" distribution. With some additional packages installed and configured to make it as easy as possible to keep an identical configuration on all of them.

The cluster architecture will look like this:
Image

Am i on the right track? Will this be an Beowulf cluster? The aim is to only connect to the controller and the controller will distribute tasks among the nodes. Please feel free to also discuss beowulf clusters in general,

macbug
Posts: 12
Joined: Thu Dec 04, 2014 8:50 pm

Re: Architechture of a Beowulf cluster.

Thu Jan 24, 2019 4:48 pm

Ive updated the architecture. Gonna buy everything i need tomorrow.
Attachments
test.jpeg
test.jpeg (33.55 KiB) Viewed 4897 times

ejolson
Posts: 4931
Joined: Tue Mar 18, 2014 11:47 am

Re: Architechture of a Beowulf cluster.

Fri Jan 25, 2019 9:24 am

macbug wrote:
Thu Jan 24, 2019 4:48 pm
Ive updated the architecture. Gonna buy everything i need tomorrow.
I liked the previous picture better because it didn't have a frame nor so many lines; however, either diagram is nice.

To be an official Beowulf cluster, you may need to be running Scyld Linux with the bproc extensions that support a unified process space. The idea is to create stubs in the process table of the controller which represent and can be used for signalling the processes running on the nodes. Scyld Linux was recently purchased by Penguin Computing and, as far as I know, only runs on x86 architecture systems.

Alternatively, a Beowulf cluster might simply mean a cluster of computers used for parallel processing which run Linux.
The imagery seems to be related to the fact that Linux came from one of the Nordic countries as did Beowulf along with the Scyldings and the Wulfings.

At any rate, since getting bproc to work on a cluster of Raspberry Pi computers is likely difficult, setting up a simpler computational cluster is more practical. You may be interested in this thread, which describes how to configure a shared filesystem, the slurm batch scheduling system and the MPICH message passing library on a cluster of Pi Zero computers. With minor modifications the same procedure should work for a cluster of Pi 3B+ computers. If you run any of the parallel programming examples in that thread, I'd be interested to know how they perform on your cluster.
Last edited by ejolson on Sat Jan 26, 2019 6:33 am, edited 2 times in total.

fruitoftheloom
Posts: 22704
Joined: Tue Mar 25, 2014 12:40 pm
Location: Delightful Dorset

Re: Architechture of a Beowulf cluster.

Fri Jan 25, 2019 9:34 am

ejolson wrote:
Fri Jan 25, 2019 9:24 am
macbug wrote:
Thu Jan 24, 2019 4:48 pm
Ive updated the architecture. Gonna buy everything i need tomorrow.
I liked the previous picture better because it didn't have a frame nor so many lines; however, either diagram is nice.

To be an official Beowulf cluster, you may need to be running Scyld Linux with the bproc extensions that support a unified process space. The idea is to create stubs in the process table of the controller which represent and can be used for signalling the processes running on the nodes. Scyld Linux was recently purchased by Penguin Computing and, as far as I know, only runs on x86 architecture systems.

Alternatively, a Beowulf cluster might simply mean a cluster of computers used for parallel processing which run Linux.
The imagery seems to be related to the fact that Linux came from one of the Nordic countries as did Beowulf along with the Scyldings and the Wulfings.

At any rate, since getting bproc to work on a cluster of Raspberry Pi computers is likely difficult, setting up a simpler computational cluster is likely more practical. You may be interested in this thread, which describes setting up a shared filesystem, the slurm batch scheduling system and the MPICH message passing library on a cluster of Pi Zero computers. With minor modifications the same procedure should work for a cluster of Pi 3B+ computers. If you run any of the parallel programming examples in that thread, I'd be interested to know how they perform on a cluster of Pi 3B+ computers.

Though not directly related the PiDramble is a good read regarding set up etcetera:

https://www.pidramble.com
Rather than negativity think outside the box !

Asus ChromeBox 3 Celeron is my other computer.

macbug
Posts: 12
Joined: Thu Dec 04, 2014 8:50 pm

Re: Architechture of a Beowulf cluster.

Sun Jan 27, 2019 3:15 pm

Thanks for the comments. My system looks like this at the moment.

Image

And when i run it, it looks like this

Code: Select all

head ➜  ~  mpiexec -f machinefile -n 2 /mnt/pi/HelloWorld/hw.py                                          
Hello Beowulf! from head.
Hello Beowulf! from node1.
head ➜  ~  mpiexec -f machinefile -n 4 /mnt/pi/HelloWorld/hw.py                                          
Hello Beowulf! from head.
Hello Beowulf! from head.
Hello Beowulf! from node1.
Hello Beowulf! from node1.
The code looks like this:

Code: Select all

#!/usr/bin/python3
import socket


def main():
    print("Hello Beowulf! from {}.".format(socket.gethostname()))


if __name__ == "__main__":
    main()
I ran out of money so i will upgrade the cluster with two more RPI's next month. The router connected to the cluster is also a RPI3B+. But now i need a good project to start coding on, maybe someone has an idea? So i can utilize the cluster.

ejolson
Posts: 4931
Joined: Tue Mar 18, 2014 11:47 am

Re: Architechture of a Beowulf cluster.

Sun Jan 27, 2019 4:28 pm

macbug wrote:
Sun Jan 27, 2019 3:15 pm
But now i need a good project to start coding on, maybe someone has an idea? So i can utilize the cluster.
That looks like a good start. To make sure everything is working as expected I would suggest running something like the OSU Point-to-Point MPI Benchmarks. I've not tried these particular programs on a Raspberry Pi cluster so I don't know how easy they are to run. However, such synthetic benchmarks could be interesting to try and would help verify your networking hardware is functioning well.

The usual program people run after bringing up a cluster is HPL the high-performance Linpack benchmark. This program solves a more realistic problem of finding the simultaneous solution to a large system of linear equations using Gaussian elimination with partial pivoting. On the Pi this program should be linked with a custom compiled installation of OpenBLAS. If things go well, I would expect around 10 GFLOPS from a cluster of two Pi 3B+ computers.

There are also a number of MPI programs I wrote in this thread. It would be amusing to see how much faster your cluster is compared to 5 Pi Zero computers.

macbug
Posts: 12
Joined: Thu Dec 04, 2014 8:50 pm

Re: Architechture of a Beowulf cluster.

Sun Jan 27, 2019 6:26 pm

Thanks for input, i will go trough your links and read up on your forum thread.


ejolson
Posts: 4931
Joined: Tue Mar 18, 2014 11:47 am

Re: Architechture of a Beowulf cluster.

Mon Jan 28, 2019 4:29 am

The first four links look useful and the last historically interesting.

In the super-cheap cluster of Pi Zero computers the head node, router and NFS server were all the same Pi. If you did the likewise, then you would already have two computational nodes without buying additional hardware. This might also simplify maintenance as there would only be two different software configurations: the nodes on one hand and the combined router, master controller, file server on the other.

macbug
Posts: 12
Joined: Thu Dec 04, 2014 8:50 pm

Re: Architechture of a Beowulf cluster.

Mon Jan 28, 2019 5:22 am

ejolson wrote:
Mon Jan 28, 2019 4:29 am

The first four links look useful and the last historically interesting.

In the super-cheap cluster of Pi Zero computers the head node, router and NFS server were all the same Pi. If you did the likewise, then you would already have two computational nodes without buying additional hardware. This might also simplify maintenance as there would only be two different software configurations: the nodes on one hand and the combined router, master controller, file server on the other.
Yeah, i will convert the router with two network cards into head and head into node2. but i will receive more rpi's this week since i re-prioritized my needs for this month. So hopefully i'll have 4 nodes up and running before Friday. This project is way more interesting then anything I've done the last year.

ejolson
Posts: 4931
Joined: Tue Mar 18, 2014 11:47 am

Re: Architechture of a Beowulf cluster.

Tue Jan 29, 2019 4:14 am

macbug wrote:
Mon Jan 28, 2019 5:22 am
i will receive more rpi's this week since i re-prioritized my needs for this month.
Hopefully you didn't skimp on the groceries, because despite the name, Raspberry Pi's are a bit too crunchy to actually eat.

There is a long thread in the off-topic forum entitled Why Avoid BASIC on the Raspberry Pi focused on the challenge problem of computing million-digit Fibonacci numbers. The idea being, if a programming language can't compute the 4784969th Fibonacci number, then it can't be good for much. In the context of this thread, a related challenge might be if a computer cluster can't compute even larger Fibonacci numbers with efficient parallel speedup, then it can't be good for much either.

The Pi was engineered to create a second age of personal computing. As such, it is a real computer designed to provide any individual the opportunity to write programs that satisfy their personal needs. A cluster made out of Raspberry Pi computers further allows an individual to write distributed-memory parallel programs. While most people don't have a need to compute multi-million-digit Fibonacci numbers, it is still an interesting question whether a person who builds and learns how to use a Pi cluster has achieved the kind of digital liberation that allows them to efficiently express and practically implement any distributed-memory parallel algorithm they can think up.

At any rate, one idea for a project would be the parallel computation of very large Fibonacci numbers.

macbug
Posts: 12
Joined: Thu Dec 04, 2014 8:50 pm

Re: Architechture of a Beowulf cluster.

Wed Jan 30, 2019 10:02 pm

I Have started to work on a blog. I will update it while developing my cluster.
http://klonq.me/

ejolson
Posts: 4931
Joined: Tue Mar 18, 2014 11:47 am

Re: Architechture of a Beowulf cluster.

Wed Jan 30, 2019 11:15 pm

macbug wrote:
Wed Jan 30, 2019 10:02 pm
I Have started to work on a blog. I will update it while developing my cluster.
http://klonq.me/
Excellent. Are you a musician?

macbug
Posts: 12
Joined: Thu Dec 04, 2014 8:50 pm

Re: Architechture of a Beowulf cluster.

Thu Jan 31, 2019 4:53 am

ejolson wrote:
Wed Jan 30, 2019 11:15 pm
macbug wrote:
Wed Jan 30, 2019 10:02 pm
I Have started to work on a blog. I will update it while developing my cluster.
http://klonq.me/
Excellent. Are you a musician?
Yeah i have music as an hobby, have a small studio and so on. And you?

ejolson
Posts: 4931
Joined: Tue Mar 18, 2014 11:47 am

Re: Architechture of a Beowulf cluster.

Thu Jan 31, 2019 8:51 pm

macbug wrote:
Thu Jan 31, 2019 4:53 am
ejolson wrote:
Wed Jan 30, 2019 11:15 pm
macbug wrote:
Wed Jan 30, 2019 10:02 pm
I Have started to work on a blog. I will update it while developing my cluster.
http://klonq.me/
Excellent. Are you a musician?
Yeah i have music as an hobby, have a small studio and so on. And you?
I used to have a home recording studio. In recent times my only musical activity has been singing in choirs. What does klonq mean?

ejolson
Posts: 4931
Joined: Tue Mar 18, 2014 11:47 am

Re: Architechture of a Beowulf cluster.

Fri Feb 01, 2019 8:52 pm

macbug wrote:
Mon Jan 28, 2019 5:22 am
So hopefully i'll have 4 nodes up and running before Friday.
Did the new Pi computers arrive?

With four nodes it might be expedient to consider a network-mounted root filesystem that can be centrally updated or alternatively some way of doing automated system maintenance simultaneously on multiple SD cards.

ejolson
Posts: 4931
Joined: Tue Mar 18, 2014 11:47 am

Re: Architechture of a Beowulf cluster.

Sat Feb 09, 2019 7:30 pm

ejolson wrote:
Fri Feb 01, 2019 8:52 pm
macbug wrote:
Mon Jan 28, 2019 5:22 am
So hopefully i'll have 4 nodes up and running before Friday.
Did the new Pi computers arrive?

With four nodes it might be expedient to consider a network-mounted root filesystem that can be centrally updated or alternatively some way of doing automated system maintenance simultaneously on multiple SD cards.
Have you made any progress with your cluster?

I've had good luck using slurm to launch programs on the computational nodes of the super-cheap cluster. As slurm is used on many bigger clusters, using slurm on the Raspberry Pi has the advantage of smoothing out the learning curve for anyone interested in doing large-scale computations in the future. Of course there are other ways to configure a cluster too, such as docker swarm, Ansible, Kubernetics, Hadoop and others.

Any news?

oicfar
Posts: 25
Joined: Tue Sep 17, 2019 7:39 am

Re: Architechture of a Beowulf cluster.

Sun Dec 01, 2019 2:55 pm

ejolson wrote:
Sat Feb 09, 2019 7:30 pm
ejolson wrote:
Fri Feb 01, 2019 8:52 pm
macbug wrote:
Mon Jan 28, 2019 5:22 am
So hopefully i'll have 4 nodes up and running before Friday.
Did the new Pi computers arrive?

With four nodes it might be expedient to consider a network-mounted root filesystem that can be centrally updated or alternatively some way of doing automated system maintenance simultaneously on multiple SD cards.
Have you made any progress with your cluster?

I've had good luck using slurm to launch programs on the computational nodes of the super-cheap cluster. As slurm is used on many bigger clusters, using slurm on the Raspberry Pi has the advantage of smoothing out the learning curve for anyone interested in doing large-scale computations in the future. Of course there are other ways to configure a cluster too, such as docker swarm, Ansible, Kubernetics, Hadoop and others.

Any news?
This is my cluster.

https://www.raspberrypi.org/forums/view ... 1#p1547811

But changed it a bit.

ejolson
Posts: 4931
Joined: Tue Mar 18, 2014 11:47 am

Re: Architechture of a Beowulf cluster.

Sun Dec 01, 2019 7:33 pm

oicfar wrote:
Sun Dec 01, 2019 2:55 pm
ejolson wrote:
Sat Feb 09, 2019 7:30 pm
ejolson wrote:
Fri Feb 01, 2019 8:52 pm
Did the new Pi computers arrive?

With four nodes it might be expedient to consider a network-mounted root filesystem that can be centrally updated or alternatively some way of doing automated system maintenance simultaneously on multiple SD cards.
Have you made any progress with your cluster?

I've had good luck using slurm to launch programs on the computational nodes of the super-cheap cluster. As slurm is used on many bigger clusters, using slurm on the Raspberry Pi has the advantage of smoothing out the learning curve for anyone interested in doing large-scale computations in the future. Of course there are other ways to configure a cluster too, such as docker swarm, Ansible, Kubernetics, Hadoop and others.

Any news?
This is my cluster.

https://www.raspberrypi.org/forums/view ... 1#p1547811

But changed it a bit.
That looks like a nice way to mount the Pi computers. Did you build it yourself?

oicfar
Posts: 25
Joined: Tue Sep 17, 2019 7:39 am

Re: Architechture of a Beowulf cluster.

Sun Dec 01, 2019 8:05 pm

ejolson wrote:
Sun Dec 01, 2019 7:33 pm
oicfar wrote:
Sun Dec 01, 2019 2:55 pm
ejolson wrote:
Sat Feb 09, 2019 7:30 pm
Have you made any progress with your cluster?

I've had good luck using slurm to launch programs on the computational nodes of the super-cheap cluster. As slurm is used on many bigger clusters, using slurm on the Raspberry Pi has the advantage of smoothing out the learning curve for anyone interested in doing large-scale computations in the future. Of course there are other ways to configure a cluster too, such as docker swarm, Ansible, Kubernetics, Hadoop and others.

Any news?
This is my cluster.

https://www.raspberrypi.org/forums/view ... 1#p1547811

But changed it a bit.
That looks like a nice way to mount the Pi computers. Did you build it yourself?
This war a "ready" case for 4 Pi's. I didn't have an 3d printer. Than you can make you own case. But I'm happy with this.

ejolson
Posts: 4931
Joined: Tue Mar 18, 2014 11:47 am

Re: Architechture of a Beowulf cluster.

Mon Dec 02, 2019 4:04 pm

oicfar wrote:
Sun Dec 01, 2019 8:05 pm
ejolson wrote:
Sun Dec 01, 2019 7:33 pm
oicfar wrote:
Sun Dec 01, 2019 2:55 pm


This is my cluster.

https://www.raspberrypi.org/forums/view ... 1#p1547811

But changed it a bit.
That looks like a nice way to mount the Pi computers. Did you build it yourself?
This war a "ready" case for 4 Pi's. I didn't have an 3d printer. Than you can make you own case. But I'm happy with this.
It looks nice. Is there a fan for each Pi? I wonder if you place the cluster enclosure horizontally so the Pi computers are vertical (and elevated for air from below through convection), maybe the fans would not be needed.

Return to “Networking and servers”