RedPanther
Posts: 19
Joined: Thu Sep 12, 2019 9:20 pm

Raspberry Pi distributed computer project

Thu Sep 12, 2019 11:11 pm

Welcome to my project build!

I’m newer to the whole working with code and terminal commands, but I’ve made it this far by doing simple research and actually learning from what I do. I’ve been into working on gaming computers for years, so this is a fun new area to me!

Here’s what I’ve got so far:
1- Raspberry Pi 3B+(powered from desktop power supply 5v via factory micro usb power slot)
1- 220W desktop power supply
1- 32Gb SD card for boot and recovery
1- Powered 4-port usb3 hub(powered via power supply 5v) for connecting HDDs
1- 500Gb 2.5” HDD(powered via power supply, connected via sata to usb3) Root FileSystem
1- 4Tb 2.5” HDD(“”””””) for PLEX media server
1- 1Tb 2.5” HDD(“”””””) for local network file storage and exchange
1- 1Tb 2.5” HDD(powered and connected via built in usb3.1) for personal file storage
3- 80mm 12v fans(powered via power supply)
server.jpg
server.jpg (195.85 KiB) Viewed 374 times
The idea behind this project was to build a simple simple server using my Pi, and not need a separate power supply for every device. Well, I got carried away with what I had laying around, and spent $45 for connectors. Three adapters to convert sata ports on HDDs to usb3, and a powered usb3 hub.

This build is currently two days into full testing, and has held up amazingly! As long as it’s only used for it’s intended purpose, that is. These little things aren’t great for computing power, so I came up with an idea to fix that in the near future. Until then, I usually connect via SSH from a Raspberry Pi ZeroW(which I connect to via VNC from my phone) to manage my whole setup.

I posted here because I know I am going to run into a lot of bumps along the way, and not know where to turn. So, what better place to come and share my journey.

For all those still following along, this project is FAR from over!

As I mentioned above, I still have more plans for this! In the future, I am going to be getting four more 2.5" HDDs(which I already have slots for in my case), another powered 4-port usb3 hub, 1200 Watt power supply to handle all of the 5v load, bigger fans, 7 Raspberry Pi 4s(4Gb variants, if I don't wait for a new version, in which case I will be temporarily installing three more 3B+s), and an actual desktop case to hide everything in eventually.

Yes, I said seven. My plan is to build a super computer/server using Pis. Because, why not? I'm sure you heard years ago about super computers being built from multiple PS3s somehow connected together. I've spent a lot of time here lately discovering just how that all works with these little Pi computers, and I would like to pull it together myself! Why? The possibilities are endless if you have a properly built computer. I learned that from years of gaming on PCs. Building a super computer of my own opens many more doors for programming and learning even more with these small boards we love so much. The main controller would initially be a Raspberry Pi 3B+, which would later be replaced by a 4 or newer model.

I still have an overall dislike for the Raspberry Pi 4B after realizing most of their updates were minimal and so far, not much of an improvement. The only reason the 4B is good at this point, in my opinion, is because it now has a more capable processor, 2 actual usb3 ports, and true Gb ethernet. While that may enough for most of you, software glitches and an extra 0.1 Ghz of processing power and 3Gb more RAM, in my experience, don't make a practical long term investment opportunity. I would rather wait for an updated model featuring an even faster processor, and still have the same 4Gb of RAM as the 4B(or more???).

If you've made it this far, there are a lot of details I did not get into here. Feel free to keep this thread alive if you have any questions! I am learning this all as I go, as far as the coding and programming, but I'm willing to help anyone with what I've done so far!
Last edited by RedPanther on Sat Sep 14, 2019 2:17 am, edited 3 times in total.

RedPanther
Posts: 19
Joined: Thu Sep 12, 2019 9:20 pm

Raspberry Pi distributed computer project

Fri Sep 13, 2019 9:49 pm

After having posted this, I set up a Plex server for my household on the Raspberry Pi3B+. While it did great for streaming directly to phones, laptops, and even a Roku, streaming from my phone to Chromecast was another story. The movies were choppy, buffered a lot, and barely played some movies for more than a couple seconds before needing to buffer again.

Lesson learned for running my mouth about an RPi4B not being a good investment.

I decided today to upgrade the main board in my server to the RPi 4 B 4Gb. With everything set up the same as it was on my 3B+, the 4B has not disappointed for running Plex, MiniDLNA, Samba, SSH and VNC on demand at all times. I even tested the same movies on the new system over Plex, and they play considerably longer before needing to buffer with slightly better quality.

I also got another Raspberry 3B+ while I was out today. I figured since I finally have a decently sized server set up, I am ready to work on incorporating my other two Pi’s into the mix, and work on a small distributed computing system.

The goal of this project is to have a fully functioning Linux based desktop environment, home and remote media server, household personal storage(local cloud server with independent user directories, each accessible by only the user assigned), personal smart voice control assistant, and a system more than capable of running intensive tasks in an efficient manner.

The 4B will be the system that calls the tasks to be run, then distribute the task to be processed on the other two boards, and send all output back to the 4B.

Onto the part I’m going to need help the most with, where would I begin with using the 4B as the main system which distributes tasks to my other two 3B+‘s? I don’t want to avoid using all processing and memory from the 4B, so I don’t want a simple SSH connection. I want to build a program on my 4B to help with distributing processes evenly based on task intensity. At some point I would like to have several “clusters” of paired Pi’s, each functioning in parallel, connected to the main board to make an even more powerful distributed system. The idea is to have the main board decide how many “clusters” need to be activated to perform a task.

I’ve rambled a lot here, but if you’re still with me and you’ve been into this sort of thing before, maybe you could help get me along. I’m looking for a starting point at the moment
Last edited by RedPanther on Sat Sep 14, 2019 2:18 am, edited 2 times in total.

LTolledo
Posts: 1766
Joined: Sat Mar 17, 2018 7:29 am
Location: Anime Heartland

Re: Raspberry Pi 3B+ Server

Sat Sep 14, 2019 12:34 am

Below is my just completed RPi4B-2G NAS
in ITX case
with 2 x 8TB 3.5" HDDs
ext4 formatted
samba shared
using simple HDD redundancy (instead of RAID)
Raspbian Buster Lite in 8GB microSD
main PSU is a single 12v 5A
12v 50x50x15 fan for HDD/case cooling
RPi4BNAS_project.jpg
RPi4BNAS_project.jpg (190.37 KiB) Viewed 332 times
picture was taken while copying all files from a 4TB RAID1 NAS.....
will get better image when copying is finished
"Don't come to me with 'issues' for I don't know how to deal with those
Come to me with 'problems' and I'll help you find solutions"

Some people be like:
"Help me! Am drowning! But dont you dare touch me nor come near me!"

RedPanther
Posts: 19
Joined: Thu Sep 12, 2019 9:20 pm

Re: Raspberry Pi distributed computer project

Sat Sep 14, 2019 5:26 am

Here is a brief write up on setting up a MiniDLNA media server by me. I hope it’s as easy and painless to follow as possible!
https://www.raspberrypi.org/forums/view ... 4#p1535864

ejolson
Posts: 3407
Joined: Tue Mar 18, 2014 11:47 am

Re: Raspberry Pi distributed computer project

Sun Sep 15, 2019 4:15 am

RedPanther wrote:
Thu Sep 12, 2019 11:11 pm
Yes, Building a super computer of my own opens many more doors for programming and learning even more with these small boards we love so much.
I built a super-cheap cluster using five Pi Zero computers, a Pi B+ and a USB hub. Getting the software to perform a parallel computation is the main difficulty with supercomputers and a Pi cluster provides a remarkable development platform for such code. More information on how I set things up and some sample programs are available in the following thread:

https://www.raspberrypi.org/forums/view ... 9&t=199994

RedPanther
Posts: 19
Joined: Thu Sep 12, 2019 9:20 pm

Re: Raspberry Pi distributed computer project

Sun Sep 15, 2019 6:05 am

ejolson wrote:
Sun Sep 15, 2019 4:15 am
RedPanther wrote:
Thu Sep 12, 2019 11:11 pm
Yes, Building a super computer of my own opens many more doors for programming and learning even more with these small boards we love so much.
I built a super-cheap cluster using five Pi Zero computers, a Pi B+ and a USB hub. Getting the software to perform a parallel computation is the main difficulty with supercomputers and a Pi cluster provides a remarkable development platform for such code. More information on how I set things up and some sample programs are available in the following thread:

https://www.raspberrypi.org/forums/view ... 9&t=199994
I’ll have to read this over when I get a chance. I’m in process of doing system upgrades at the moment and figuring out placement for clusters in my case

RedPanther
Posts: 19
Joined: Thu Sep 12, 2019 9:20 pm

Re: Raspberry Pi distributed computer project

Mon Sep 16, 2019 4:34 pm

If you've made it this far into my build, I recently went through some system upgrades. Over the weekend I was able to fully rewire a 300W power supply(will get upgraded again when i get more Pi's), upgrade to Raspberry Pi 4B, and finished getting my case together. My case now has enough room for about twelve more Pi's. I still need to add USB ports, power and shutdown switches, and lights for Power, HDD activity, and CPU activity all to the front of the case. I’ll also be adding HDMI, Ethernet, and more USB ports to the back of the case.
Attachments
server004.JPG
server004.JPG (116 KiB) Viewed 151 times
server003.JPG
server003.JPG (137.31 KiB) Viewed 151 times
server002.jpg
server002.jpg (74.71 KiB) Viewed 151 times

ejolson
Posts: 3407
Joined: Tue Mar 18, 2014 11:47 am

Re: Raspberry Pi distributed computer project

Tue Sep 17, 2019 4:07 am

RedPanther wrote:
Mon Sep 16, 2019 4:34 pm
If you've made it this far into my build, I recently went through some system upgrades.
It looks like things are coming together. Are you are making your own case? What is the case made of? Is it made from wood? Or is it made with some special type of plastic?

RedPanther
Posts: 19
Joined: Thu Sep 12, 2019 9:20 pm

Re: Raspberry Pi distributed computer project

Tue Sep 17, 2019 5:46 am

ejolson wrote:
Tue Sep 17, 2019 4:07 am
It looks like things are coming together. Are you are making your own case? What is the case made of? Is it made from wood? Or is it made with some special type of plastic?
I am building my own case. I am using flat rolled plastic. Corners are held together with brackets, all inside seams will be sealed, and a side panel will be removable for ease of access

Return to “Advanced users”