MuntyScruntfundle wrote: ↑
Tue Oct 23, 2018 3:16 am
Ok, so, after my messing about with led panels and clusters I've been approached to build a 128 node cluster. I'm sightly annoyed I don't have the money to build my own, I'll have to do with my modest 8 node system, but at least I'll get a go with this one I guess.
The advantage of the Bitscope solution, as I understand it, is the ability to remotely power on and off each Pi as well as network boot them. Both of these features become important once the size of the cluster scales beyond about 20 nodes.
As Pi reliability is greatly dependent on quality of the power supply, here is an area where you don't want to cut too many corners. I've always wondered where to obtain a high-capacity computer-grade 5V power supply or whether a higher voltage supply with separate 5V switching regulators for each Pi would be better.
From what I understand, not all network switches allow a Pi 3B+ to network boot, so these have to be chosen carefully. I think a 3B+ will continue to boot from the network even if it has an SD card provided the FAT-formatted boot partition on the card is empty. Thus, you could design an initial RAM filesystem that loads over the network, checks if the root partition on the SD card needs to be imaged or resynched and after that pivots root to the SD card. Hadoop clusters boot in this fashion as well as some Linux distributions designed for clusters. Rolling your own for the Pi should not be very difficult and would provide the centralised maintenance advantages of network booting along with the performance advantages of having the root filesystem mounted from local storage.
In the other hand, it might be preferable to go without SD cards entirely and mount the root filesystems of all 128 machines from a single server. That server should not be another Pi. Note that the first suggested method of imaging and resynching SD cards could be done using another Pi without needing a faster server.
It would be great to hear how things progress.