hosseinpink
Posts: 79
Joined: Fri Mar 22, 2019 9:53 pm

Python programming on Cluster (MPICH or SLURM)

Fri Apr 12, 2019 12:17 am

Hi Friends,
I wanna make a cluster for 4 Pis.
I read two articles: one MPICH cluster and another one SLURM cluster.
As my end goal is to evaluate my prepared Python code on the cluster , which one is better?
I don't wanna face any challenge or extra coding after building my cluster.

ejolson
Posts: 4842
Joined: Tue Mar 18, 2014 11:47 am

Re: Python programming on Cluster (MPICH or SLURM)

Fri Apr 12, 2019 12:37 am

hosseinpink wrote:
Fri Apr 12, 2019 12:17 am
Hi Friends,
I wanna make a cluster for 4 Pis.
I read two articles: one MPICH cluster and another one SLURM cluster.
As my end goal is to evaluate my prepared Python code on the cluster , which one is better?
I don't wanna face any challenge or extra coding after building my cluster.
I'm not sure what you can do with a cluster that doesn't involve extra coding.

Slurm is a batch scheduling system while MPICH is a message passing library for parallel programming. Every cluster I have ever used had both.

hosseinpink
Posts: 79
Joined: Fri Mar 22, 2019 9:53 pm

Re: Python programming on Cluster (MPICH or SLURM)

Fri Apr 12, 2019 12:39 am

ejolson wrote:
Fri Apr 12, 2019 12:37 am
hosseinpink wrote:
Fri Apr 12, 2019 12:17 am
Hi Friends,
I wanna make a cluster for 4 Pis.
I read two articles: one MPICH cluster and another one SLURM cluster.
As my end goal is to evaluate my prepared Python code on the cluster , which one is better?
I don't wanna face any challenge or extra coding after building my cluster.
I'm not sure what you can do with a cluster that doesn't involve extra coding.

Slurm is a batch scheduling system while MPICH is a message passing library for parallel programming. Every cluster I have ever used had both.
I mean which one can support Python and its packages better?

hosseinpink
Posts: 79
Joined: Fri Mar 22, 2019 9:53 pm

Re: Python programming on Cluster (MPICH or SLURM)

Fri Apr 12, 2019 12:42 am

ejolson wrote:
Fri Apr 12, 2019 12:37 am
hosseinpink wrote:
Fri Apr 12, 2019 12:17 am
Hi Friends,
I wanna make a cluster for 4 Pis.
I read two articles: one MPICH cluster and another one SLURM cluster.
As my end goal is to evaluate my prepared Python code on the cluster , which one is better?
I don't wanna face any challenge or extra coding after building my cluster.
I'm not sure what you can do with a cluster that doesn't involve extra coding.

Slurm is a batch scheduling system while MPICH is a message passing library for parallel programming. Every cluster I have ever used had both.
with which one I will have fewer headaches?

ejolson
Posts: 4842
Joined: Tue Mar 18, 2014 11:47 am

Re: Python programming on Cluster (MPICH or SLURM)

Fri Apr 12, 2019 1:36 am

hosseinpink wrote:
Fri Apr 12, 2019 12:42 am
ejolson wrote:
Fri Apr 12, 2019 12:37 am
hosseinpink wrote:
Fri Apr 12, 2019 12:17 am
Hi Friends,
I wanna make a cluster for 4 Pis.
I read two articles: one MPICH cluster and another one SLURM cluster.
As my end goal is to evaluate my prepared Python code on the cluster , which one is better?
I don't wanna face any challenge or extra coding after building my cluster.
I'm not sure what you can do with a cluster that doesn't involve extra coding.

Slurm is a batch scheduling system while MPICH is a message passing library for parallel programming. Every cluster I have ever used had both.
with which one I will have fewer headaches?
Unless you have already written or plan to write parallel programs using MPICH, it would be more important to have Slurm installed and working.

Note that Python does have MPI subroutine wrappers, so writing a parallel program in Python using MPICH would be possible. In this case you would probably want both Slurm and MPICH along with the Python MPI wrappers to be installed. However, in that case it would be easier to use Fortran or C instead of Python.

hosseinpink
Posts: 79
Joined: Fri Mar 22, 2019 9:53 pm

Re: Python programming on Cluster (MPICH or SLURM)

Fri Apr 12, 2019 2:06 am

ejolson wrote:
Fri Apr 12, 2019 1:36 am
hosseinpink wrote:
Fri Apr 12, 2019 12:42 am
ejolson wrote:
Fri Apr 12, 2019 12:37 am
I'm not sure what you can do with a cluster that doesn't involve extra coding.

Slurm is a batch scheduling system while MPICH is a message passing library for parallel programming. Every cluster I have ever used had both.
with which one I will have fewer headaches?
Unless you have already written or plan to write parallel programs using MPICH, it would be more important to have Slurm installed and working.

Note that Python does have MPI subroutine wrappers, so writing a parallel program in Python using MPICH would be possible. In this case you would probably want both Slurm and MPICH along with the Python MPI wrappers to be installed. However, in that case it would be easier to use Fortran or C instead of Python.
Can I run both MPICH and SLURN simultaneously on same PiS??
As I wanna evaluate some ML algorithms here Python will be so practical.

ejolson
Posts: 4842
Joined: Tue Mar 18, 2014 11:47 am

Re: Python programming on Cluster (MPICH or SLURM)

Sat Apr 13, 2019 12:20 am

hosseinpink wrote:
Fri Apr 12, 2019 2:06 am
ejolson wrote:
Fri Apr 12, 2019 1:36 am
hosseinpink wrote:
Fri Apr 12, 2019 12:42 am

with which one I will have fewer headaches?
Unless you have already written or plan to write parallel programs using MPICH, it would be more important to have Slurm installed and working.

Note that Python does have MPI subroutine wrappers, so writing a parallel program in Python using MPICH would be possible. In this case you would probably want both Slurm and MPICH along with the Python MPI wrappers to be installed. However, in that case it would be easier to use Fortran or C instead of Python.
Can I run both MPICH and SLURN simultaneously on same PiS??
As I wanna evaluate some ML algorithms here Python will be so practical.
Yes, a standard configuration for many clusters is to include both a batch submission system and a message passing library for parallel processing. Slurm is used to schedule and start a computation while MPICH shares data between the parallel ranks during a computation.

It should be emphasized that both Slurm and MPICH were created for high-performance scientific computing. As a result, these softwares are not well suited for starting or running the cloud-like micro services popular with websites, mobile apps and IOT devices. If you are still interested in fog then you might be better off looking at on-the-ground integration between OpenStack, Docker Swarm and things like Kubernetics.

hosseinpink
Posts: 79
Joined: Fri Mar 22, 2019 9:53 pm

Re: Python programming on Cluster (MPICH or SLURM)

Sat Apr 13, 2019 3:35 am

ejolson wrote:
Sat Apr 13, 2019 12:20 am
hosseinpink wrote:
Fri Apr 12, 2019 2:06 am
ejolson wrote:
Fri Apr 12, 2019 1:36 am
Unless you have already written or plan to write parallel programs using MPICH, it would be more important to have Slurm installed and working.

Note that Python does have MPI subroutine wrappers, so writing a parallel program in Python using MPICH would be possible. In this case you would probably want both Slurm and MPICH along with the Python MPI wrappers to be installed. However, in that case it would be easier to use Fortran or C instead of Python.
Can I run both MPICH and SLURN simultaneously on same PiS??
As I wanna evaluate some ML algorithms here Python will be so practical.
Yes, a standard configuration for many clusters is to include both a batch submission system and a message passing library for parallel processing. Slurm is used to schedule and start a computation while MPICH shares data between the parallel ranks during a computation.

It should be emphasized that both Slurm and MPICH were created for high-performance scientific computing. As a result, these softwares are not well suited for starting or running the cloud-like micro services popular with websites, mobile apps and IOT devices. If you are still interested in fog then you might be better off looking at on-the-ground integration between OpenStack, Docker Swarm and things like Kubernetics.
The sicentific programming and learning and training ML algortihms on cheap cluster devices is what i am interested in.
as Pis are analogy of Fog computing I just try to host those kinds of computation on them rather than clouds.
Kubernetes and evaluating them are my next project but I wanted to get a sense of cluster and its evaluation before.
Maybe it can be good idea to cluster far away devices for some processing and release them again.
therefore I also looked for some latencies in clusters ; delaying the message passing to simulate the fog nodes in far away distances; how it sounds??

ejolson
Posts: 4842
Joined: Tue Mar 18, 2014 11:47 am

Re: Python programming on Cluster (MPICH or SLURM)

Sat Apr 13, 2019 4:06 am

hosseinpink wrote:
Sat Apr 13, 2019 3:35 am
how it sounds??
It will be interesting to find out what you discover.

hosseinpink
Posts: 79
Joined: Fri Mar 22, 2019 9:53 pm

Re: Python programming on Cluster (MPICH or SLURM)

Sat Apr 13, 2019 5:51 am

ejolson wrote:
Sat Apr 13, 2019 4:06 am
hosseinpink wrote:
Sat Apr 13, 2019 3:35 am
how it sounds??
It will be interesting to find out what you discover.
Surely I will inform you.

hosseinpink
Posts: 79
Joined: Fri Mar 22, 2019 9:53 pm

Re: Python programming on Cluster (MPICH or SLURM)

Mon Apr 15, 2019 12:43 am

ejolson wrote:
Sat Apr 13, 2019 4:06 am
hosseinpink wrote:
Sat Apr 13, 2019 3:35 am
how it sounds??
It will be interesting to find out what you discover.
Do you know what is the Python version of this cluster?

hosseinpink
Posts: 79
Joined: Fri Mar 22, 2019 9:53 pm

Re: Python programming on Cluster (MPICH or SLURM)

Tue Apr 16, 2019 2:39 am

hosseinpink wrote:
Sat Apr 13, 2019 3:35 am
ejolson wrote:
Sat Apr 13, 2019 12:20 am
hosseinpink wrote:
Fri Apr 12, 2019 2:06 am

Can I run both MPICH and SLURN simultaneously on same PiS??
As I wanna evaluate some ML algorithms here Python will be so practical.
Yes, a standard configuration for many clusters is to include both a batch submission system and a message passing library for parallel processing. Slurm is used to schedule and start a computation while MPICH shares data between the parallel ranks during a computation.

It should be emphasized that both Slurm and MPICH were created for high-performance scientific computing. As a result, these softwares are not well suited for starting or running the cloud-like micro services popular with websites, mobile apps and IOT devices. If you are still interested in fog then you might be better off looking at on-the-ground integration between OpenStack, Docker Swarm and things like Kubernetics.
The sicentific programming and learning and training ML algortihms on cheap cluster devices is what i am interested in.
as Pis are analogy of Fog computing I just try to host those kinds of computation on them rather than clouds.
Kubernetes and evaluating them are my next project but I wanted to get a sense of cluster and its evaluation before.
Maybe it can be good idea to cluster far away devices for some processing and release them again.
therefore I also looked for some latencies in clusters ; delaying the message passing to simulate the fog nodes in far away distances; how it sounds??
I have created my MPICH cluster. now i wanna create my SLURM cluaster.
can i choose the same master node or i need to select another Pi as masater node?
(I wanna have both clusters simultaneously )

ejolson
Posts: 4842
Joined: Tue Mar 18, 2014 11:47 am

Re: Python programming on Cluster (MPICH or SLURM)

Tue Apr 16, 2019 2:49 am

hosseinpink wrote:
Tue Apr 16, 2019 2:39 am
hosseinpink wrote:
Sat Apr 13, 2019 3:35 am
ejolson wrote:
Sat Apr 13, 2019 12:20 am
Yes, a standard configuration for many clusters is to include both a batch submission system and a message passing library for parallel processing. Slurm is used to schedule and start a computation while MPICH shares data between the parallel ranks during a computation.

It should be emphasized that both Slurm and MPICH were created for high-performance scientific computing. As a result, these softwares are not well suited for starting or running the cloud-like micro services popular with websites, mobile apps and IOT devices. If you are still interested in fog then you might be better off looking at on-the-ground integration between OpenStack, Docker Swarm and things like Kubernetics.
The sicentific programming and learning and training ML algortihms on cheap cluster devices is what i am interested in.
as Pis are analogy of Fog computing I just try to host those kinds of computation on them rather than clouds.
Kubernetes and evaluating them are my next project but I wanted to get a sense of cluster and its evaluation before.
Maybe it can be good idea to cluster far away devices for some processing and release them again.
therefore I also looked for some latencies in clusters ; delaying the message passing to simulate the fog nodes in far away distances; how it sounds??
I have created my MPICH cluster. now i wanna create my SLURM cluaster.
can i choose the same master node or i need to select another Pi as masater node?
(I wanna have both clusters simultaneously )
The master should be the same for both.

hosseinpink
Posts: 79
Joined: Fri Mar 22, 2019 9:53 pm

Re: Python programming on Cluster (MPICH or SLURM)

Thu Apr 18, 2019 5:53 am

I wanna see a document or explanation that can clarify the difference between MPICH cluster and SLRUM cluster:
their behavior in processing , differences and any attribute that can clarify that.

ejolson
Posts: 4842
Joined: Tue Mar 18, 2014 11:47 am

Re: Python programming on Cluster (MPICH or SLURM)

Thu Apr 18, 2019 6:03 am

hosseinpink wrote:
Thu Apr 18, 2019 5:53 am
I wanna see a document or explanation that can clarify the difference between MPICH cluster and SLRUM cluster:
their behavior in processing , differences and any attribute that can clarify that.
Instructions how to set up a cluster with both Slurm and MPICH may be found in this thread. It would also be reasonable to read the documentation on the web sites

https://slurm.schedmd.com
https://www.mpich.org

for an overview of what each software does.

oicfar
Posts: 25
Joined: Tue Sep 17, 2019 7:39 am

Re: Python programming on Cluster (MPICH or SLURM)

Thu Sep 19, 2019 8:46 pm

hosseinpink wrote:
Fri Apr 12, 2019 12:17 am
Hi Friends,
I wanna make a cluster for 4 Pis.
I read two articles: one MPICH cluster and another one SLURM cluster.
As my end goal is to evaluate my prepared Python code on the cluster , which one is better?
I don't wanna face any challenge or extra coding after building my cluster.
The question is what do you wan't to do with the cluster?

Setup cluster is easy. Use the power difficult.

My cluster has 1* Pi 4 4 GB as cluster manager. And then 4 Nodes. Each node is Pi 4 with 4 GB. I'm using slurm and open-mpi. If you want use all the cores, than you have to learn using MPI.

Return to “General discussion”