Pi4 Microk8s Cluster

[clicky]

Hi, here's just an inspirational post - I've just finished week long challenge (I've given myself) and here's the result:

• Pi4 2GB with JMicron(eh) M-key SSD adapter and 0.5TB Intel's SSD (short USB3-A extension cable is on the way) acting as tftp and nfs server
• 2x Pi4 4GB (two for now - another 2-3 allowed) as PXE net booted(*) workers
  [*[all three running 64bit Raspberry Pi OS (**)
• all three have snap version of microk8s installed(**) (server has scheduling switched off - it is 'cordoned')
• Microk8s's metallb happily talking to my Draytek server and assigning (static as requested in yaml) IP to 'pods' (read containers) (- not IPs associated with real interfaces, it uses BGP to talk to my router)
• cluster has its own private docker repo (working of nfs persisted volume)
• and my containers can happily use nfs persisted volumes, too

If I would like to add another node, it is as easy as copy /srv/tftp/<existing-mac-address> to new Pi's mac address and copy /srv/nfs/cluster-workerX to X+1 (with amending its .../etc/fstab to IP assigned by my router - maybe set hostname, too).

(*) speaking of PXE boot - it worked like a charm: in tftp root I had directories that match mac addresses of workers to contain their boot dir which only has cmdline.txt amended to match IP given to them by my rooter. Also, it order to avoid any messing up with secondary DHCP server and such malarkeys, I've added TFTP_IP in firmware config (with IP my router assigns to the server Pi4) and that was it. No other stuff was needed for PXE boot!

(**) in order for Microk8s to work on workers I had to recompile kernel (it was far, far less of an issue than I originally was worried might be - see here viewtopic.php?f=63&t=281061&p=1702371#p1702371)

I've been pleasantly surprised with BGP working just out of box and quite glad I avoided need for proxying all through the server.

Picture is work in progress - couple of bracket needed to be made and all hanged on the wall somewhere - only three cables are going away from the cluster: power for smart switch and USB power supply and cat 6 ethernet cable.

And to conclude: Raspberry Pi (in this case version 4!) is really a lots of fun!

[ejolson]

I've been pleasantly surprised with BGP working just out of box and quite glad I avoided need for proxying all through the server.

Thanks for the report. I've been working on a cluster for kubernetes here

viewtopic.php?p=1690721#p1690721

The tip about cgroups will be helpful.

I am interested how you used the border gateway protocol in your cluster to avoid proxying everything through the server.

What exactly did you do with BGP?

Are the Pi computers attached to that board? Will it be hung vertical?

[clicky]

The tip about cgroups will be helpful.

No worries. I've already hinted something here: viewtopic.php?f=63&t=281061&p=1702371#p1702371

Let me know if you need more.

The tip about cI am interested how you used the border gateway protocol in your cluster to avoid proxying everything through the server.

What exactly did you do with BGP?

Metallb talks to my router based on BGP. See here: https://metallb.universe.tf/configuration/

Instead of just enabling it through microk8s - I've copied config from above and applied to my cluster...

The tip about cAre the Pi computers attached to that board? Will it be hung vertical?

Yes. I'll upload next picture when I sort out current mess I am in... (I've done something bad to my router and needed to reset it to factory default - and, of course, didn't do backup on time )

[clicky]

I've recorded my journey (mostly to remind myself) here: https://blog.abstracthorizon.org/posts/ ... s-cluster/
It is not supposed to be tutorial nor 'the right way' - just reminder if I need to do it again. I'll update it if I remember anything else...

[ejolson]

I've recorded my journey (mostly to remind myself) here: https://blog.abstracthorizon.org/posts/ ... s-cluster/
It is not supposed to be tutorial nor 'the right way' - just reminder if I need to do it again. I'll update it if I remember anything else...

It's astonishing how many little gotchas you had to work around.

I like how the Pi computers have been placed at an angle so the USB C power cable has clearance to plug in. I had a similar problem with my Pi cloud and ended up getting some cables with L-plugs on the USB C end but still straight on the USB A end. Those were difficult to find and I don't see them for sale now.

If it were mine, I'd swap the location of the file server so it was in the middle, that way the flash stick does not hang off the edge of the board. I'd also think about adding a bracket to support the SSD and protect against earthquakes as well as unexpected clumsiness.

Thanks for posting. I'll let you know how I fare when I try to install kubernetes on the 2U Pi cloud.

[clicky]

It's astonishing how many little gotchas you had to work around.

I never though it would be easy and straight-forward and I've put time aside for it. And, yet, it was frustrating at times, but good thing was that even though I refused to back off my requirements (Raspi OS and such), I always managed to find way forward.

I like how the Pi computers have been placed at an angle so the USB C power cable has clearance to plug in. I had a similar problem with my Pi cloud and ended up getting some cables with L-plugs on the USB C end but still straight on the USB A end. Those were difficult to find and I don't see them for sale now.

At one point I thought of that, too - but forgot about it when I went to buy cables. I must admit that mounting them at angle was my son's idea

If it were mine, I'd swap the location of the file server so it was in the middle, that way the flash stick does not hang off the edge of the board. I'd also think about adding a bracket to support the SSD and protect against earthquakes as well as unexpected clumsiness.

That's good point, but what's on the picture is really just a temporary solution. I did order short USB 3 Type A extension cable and already had enclosure for SSD. See the attached picture. But, I am thinking of moving server to the right (empty place) so there can be space above and below it - on one I'll put SSD and another I'll put 3.5" disk (2TB or 4TB) as 'slow storage'

Thanks for posting. I'll let you know how I fare when I try to install kubernetes on the 2U Pi cloud.

Please do!