The little computer that could

Liz: Today we’ve got a guest post from the terrifyingly hirsute Pete Stevens. Pete’s from Mythic Beasts, our web hosts; and he’s the reason this website stands up to the absurd amounts of traffic you throw at it. (Yesterday we saw about a quarter of a million sessions – that goes up WAY above a million on some days.) Have at it, Pete!

After our successful test of using the Raspberry Pi 3 for hosting 5% of the traffic on Pi 3 launch day we celebrated by going to the pub. The conversation went something like this:

Eben: Is it possible to host the whole site on Pi 3s?
Pete: How would you do it?
Philip: Wouldn’t it be awesome to do it?
Liz: Dare you to try it!

The first part of the answer is quite easy: not on one Pi 3, it’s not fast enough. A better question is how many Pi 3s are required to host a typical day’s traffic. Extrapolating from some graphs and making up some numbers with the handy pub beermat service, we estimated between 4 and 6 should handle all the PHP code and file delivery. Unfortunately, the database server still looks out of scope: not enough RAM and not enough I/O.

Of course, only an idiot would replace thousands of pounds of highly specified hardware with a handful of £30 computers and expect it to still work.

A few weeks later we have this:

A mini rack of Raspberry Pi 3s

A mini rack of Raspberry Pi 3s

We’ve designed a custom plastic enclosure for holding Pi 3s securely, added power over ethernet HATs so we can power them directly from the switch, and a cheap 100Mbps PoE switch. We’ve put all the storage over the network and put a small storage server in the rack with the Pi 3 rack. We’ve used virtual LANs to have two effective network cards on each Pi 3: one just containing it and the storage server, the other with an IPv6 address that talks to the public internet and the load balancers. Ifconfig looks like this:

storage : eth0 : 10.46.189.X
public : eth0.131 : 2a00:1098:0:84:1000:1:0:X

As with all Pi servers, there is no public IPv4 connectivity to each server. To get out to legacy IPv4-only services such as Twitter / Akismet etc. they go through our NAT64 DNS proxy service. Inbound traffic lands on the front end load balancers and is shared between the Raspberry Pi 3s over IPv6.

 iftop, our network monitoring software showing traffic shuttling between the fileserver and the load balancers

 iftop, our network monitoring software showing traffic shuttling between the fileserver and the load balancers

If you do:

HEAD -E https://www.raspberrypi.org/

you’ll see a header which gives you the final octet of the address of the Pi that served you:

X-Served-By: Raspberry Pi 1e

The first person to tweet all the hex identifiers to Mythic Beasts wins absolutely nothing other than the respect of the Raspberry Pi community.

Is this a commercial hosted Pi service?

It’s not yet a commercially viable service. Scaled up we can fit 96 Pi3s in 4U of rack space including the switches, which is an impressive density. However, the Pis aren’t individually replaceable once in service. That means if a customer botches the SD card the Pi is dead until we can arrange downtime of all 96 Pis in the unit. Kernel upgrades involve a change on the SD card which carries a risk of bricking the Pi if the user gets it wrong. Not having access to the SD card other than via booting the Pi from it means that an enterprising user could compromise the kernel on the SD card and root-kit the machine, before cancelling the service and letting us sell it to another user.

But it’s close. Add in netboot with PXE and most of the above concerns go away, as we can remotely provision, remotely re-provision and remotely recover a broken Pi.

The Pi Rack under construction and testing

The Pi Rack under construction and testing

The Pi rack operational and waiting for your HTTP requests

The Pi rack operational and waiting for your HTTP requests

67 comments

Avatar

*chuckles*

Avatar

I wish you had posted this on any day other than April 1st.

Avatar

They should have slipped in Pi 4s instead of Pi 3s as a joke. ;)

Avatar

April 1, 2019 is a Monday….that would fit with Eben’s statements and be a typical day of the week for a Pi launch.

Avatar

Hmmm….fascinating.

Avatar

Hurries to to swag shop and orders 96 Pi’s . . . .

. . . . Oooppsss wrong Pi’s what’s your return policy Ginsters ??!?!

Avatar

impressive !
of this fish doesn’t get caught in the (Inter) Net

Avatar

@Liz

It’s 1st of April, but is this really an April fools joke? It seems too plausible.. and is entirely plausible to scale out a website with ARM chips, like PayPal are already doing.

My humble blog is also running on a Raspberry PI (2) but I was considering adding one or two more PI 3s and a load balancer. Requests per second is pretty low.. like 8-13 on average.

http://blog.alexellis.io/self-hosting-on-a-pi/

I’ve clustering 7 PIs for distributed load of a Node.js app (this was in LUD magazine issue 163) and retweeted by Docker.

http://blog.alexellis.io/linux-user-developer-magazine/

Avatar

It’s real – the biggest issues are the per core clock speed for rendering pages quickly enough and having enough RAM to deal with some of the corner cases. The Pi3 launch page with 800 comments takes around 7s to render on the Pi 3 (although Supercache means we don’t do it very often, but the odd user gets a slowish experience).

Avatar

I love the way Pete describes a 7s page render as “slowish”. I remember when I had my first contract to write a website we were targeting 35s or less per page load. That was back in the days of 56K modems of course ;)

You’re right though: I get grumpy of a site takes any longer than 2s to load on my fibre connection now.

Avatar

Late to the party, guys. Some french company by the name of scaleway has been offering bare-metal ARM servers for years.

Avatar

do you think this would work with other versions of the raspberry pi?

Avatar

thats cute

Avatar

Why do I think that to?

Avatar

Too*

Avatar

You could do it with the Pi2, but you’d need at least 12 to keep up. The Pi Zero and Pi 1 just aren’t fast enough to create pages to stop the users complaining.

Avatar

Ok… The post just came and I got my issue of the magpi44

Avatar

Nicely done ;-)

X-Clacks-Overhead:GNU Terry Pratchett
X-Served-By:Raspberry Pi a

Avatar

+ + + Re-do from start + + +

Avatar

The PIs I could find:

Raspberry Pi a
Raspberry Pi e
Raspberry Pi 6
Raspberry Pi 12
Raspberry Pi 1a
Raspberry Pi 2
Raspberry Pi 16
Raspberry Pi 1e

Code, in node.js:

var request = require(‘request’);
var async = require(‘async’);

var found = [];

async.until(function() {
return false;
}, function(cb) {
request.head(“https://www.raspberrypi.org/blog/the-little-computer-that-could/”,
function(err,res,body) {
var servedBy = res.headers[‘x-served-by’];
if(found.indexOf(servedBy)==-1){
console.log(servedBy);
found.push(servedBy);
}
cb();
});
});

Avatar

That’s the full set. There’s 8 in the production setup although the admin page is still on the primary x86 VMs – the faster per core clock speed makes deleting the spam comments and moderating a bit nicer. Every user that visits the site on April 1st will have all their content delivered direct from a Pi 3.

Avatar

I din’dt know where to find this info then kaboom it was here.

Avatar

“Of course, only an idiot would replace thousands of pounds of highly specified hardware with a handful of £30 computers and expect it to still work.”

After all “How hard can it be?” is I believe the traditional response to accepting a challenge..

Just noted todays date is 4^0 / 4^1 / 4^2

Avatar

Any set of three numbers will fit in a second-order equation.
Today we have a clean one though.

Avatar

Found
X-Served-By: Raspberry Pi 12
X-Served-By: Raspberry Pi 16
X-Served-By: Raspberry Pi 1a
X-Served-By: Raspberry Pi 1e
X-Served-By: Raspberry Pi 2
X-Served-By: Raspberry Pi 6
X-Served-By: Raspberry Pi a
X-Served-By: Raspberry Pi e##

for i in {1..100} ; do HEAD -E https://www.raspberrypi.org/ ; done | grep X-Served-By | sort | uniq

Avatar

This is much nicer than the node.js solution above!

Avatar

Bash is concise.. but not as cross-platform as node ;-)

The Node.js code looks better with padding, which isn’t supported by this comment system.

https://gist.github.com/alexellis/571f47491a91894f50b2ac58e298fd2c

Avatar

I think if the code works out of the box on a Pi with no additional packages it’s portable enough for the audience of this website!

Avatar

This can be made slightly more portable… HEAD isn’t installed on MacOS or on (Ubuntu?) Linux without extra packages… but curl is really common… and with cut we can get the formatting needed after the grep..

for i in {1..100} ; do curl -s -I https://www.raspberrypi.org/ ; done | grep X-Served-By | cut -d”:” -f 2 | sort | uniq

Avatar

Would love to see a PoE HAT for Zero.

Avatar

Actually I see you can already commercially buy them for the bigger Pi and I did not know that.

I think what would be useful to address the concerns that the author had about density and management would be to use the USB Gadget function to act as a KVM to a host machine with a USB hub. But I think the bigger Pi’s don’t support Gadget mode?

Avatar

We actually have a design for an Ethernet board for the Pi Zero ready to go. That could easily be paired with a PoE HAT for Pi Zero Ethernet :-)

Avatar

I build a webserver in go and it runs rather nice on a raspberry pi

Avatar

April 1st?
I guess we should also announce the arrival of pi 4 and 5 today. Also the pi store is selling used unicorns.

Avatar

Eating Humble Pi now, I am. Just wouldn’t believe it at first. Wow!

Guess it’s still the flashbacks to 1st April 2014 ;o) https://www.raspberrypi.org/blog/new-website-design-is-here/?fool Bring back the Green, Red and Black!

Avatar

Of course I’m now wondering how you’d host absolutely everything solely with Raspberry Pis. The multi-Gbps load balancers at the front end are only the first difficult problem…

Avatar

Solely on ARM?

http://host.rupy.se

Avatar

Any chance of releasing the case specs to iterate with?

Avatar

What is the cpu load and temperature of individual RPi3? Are they overheating? Do you have cooling in place?

Avatar

Currently no special cooling other than the pressure through the rack from the hot aisle / cold aisle. Temperatures range from 49C to 57C. Long term load averages about 1.5 on each Pi.

Avatar

Since starting to do some mass storage benchmarking, I have independently come to the conclusion that the Current weak pint to the Pi is I/O. Perhaps the Pi4 will solve at least some of that problem is 3 or 4 years.

Avatar

As PXE is mentioned in this posting: is there any documentation on how to setup PXE on the RasPi3 available?

Avatar

No. The mention of PXE in the post is to complain that it still doesn’t yet work and we had to boot from the flash card instead. Hopefully it’s coming soon though.

Avatar

Thanks for the info!

I managed to boot my C64 using PXE using a self-written firmware. (No joke.) The Pi3 is next. What is the best way to be notified, once the docs are available? Keep monitoring this blog?

Avatar

There’s always a certain amount of doubt with any story published on 1st April (I used to skip reading SlashDot on April Fools because it was always full of worse junk than usual).

I’m almost convinced there’s a X-Served-By:Raspberry Pi 6 in the bits & bytes I looked at.

Avatar

It’s real. We thought showing you all today would be kind of funny because we knew a significant proportion of you wouldn’t believe us!

Avatar

I’m trusting you on this one. I was embarrassed after telling my dad about “The Apple Device” http://hackaday.com/2016/04/01/apple-introduces-their-answer-to-the-raspberry-pi/
And then him telling me “no way! that is an April fools day trick.”
I then read the comments of the post and then all of a sudden remembed what day it was. :P

Avatar

Any update on when we could expect PXE booting?

Avatar

Anybody noticed? Date is 1.4.16, = .1416, part of Pi (to 4 places)

Avatar

Err.. isn’t any date and time part of pi. i.e.
1.4.16 20:37:22 is also part of pi in so far as
pi=3.141…some number of digits…1416203722…more digits

Avatar

Confirmation of Half Life 3 ? ;-)

Avatar

Hey Paddy! Yep, and you can find the complete works of Shakespeare and everything else ever written, if you look long enough :D If 10,000 Pii each generated 10,000 digits in 10,000 seconds, how many typewriter-equipped monkeys would be collecting unemployment? ;)

This is a very cool thing that only the kinds of folks in the Pi community would undertake … and succeed! Bravo Zulu, Pete and crew!

Avatar

this phrase is a bit unreadable “on just containing it and the storage server the other”

Avatar

It’s a typo. Should be “One [virtual network card] containing it [=the RPi itself]…”

So it’s April 3rd and I admit I’m not good at the whole spotting-a-prank thing. So is this an April Fool’s Day joke, or is (or was) the site actually hosted by a whole box of Pi 3’s?

Avatar

It was real!

Avatar

I can see a Raspberry Pi 3 with Ubuntu Mate is faster than a Core I7 pc with Windows 7/8/10 when you work or have fun on it. Now if VideoCore4 would allow more than 1GB of ram…

Avatar

Wow, a more in depth article with details of configuring would be amazing!

Avatar

Wonderful

Avatar

You guys need to design a cluster-able “blade” type of board.
Call it the RP Zero.3, and have a back plane something like:

http://www.geek.com/chips/raspberry-pi-zero-cluster-in-development-1645823/

With perhaps better I/O speeds, some common memory, and
HDD/SSD interfaces.

It would make a great Server and a great low cost light duty
supercomputer…

Avatar

Awesome guys !
I really like the “because you can do it” attitude. That’s what an open minded guy does and the Pi perfectly fits this area. Well done.
I’d like to hear more success stories like this one, thanks !

Ben

Avatar

Excellent work. I think the next step is to launch a Pi cloud with this Pi cluster.

Avatar

What POE HAT did you use?

Avatar

You guys might want to read this: http://serverfault.com/q/745711/273852

Replace “lightweight” with “a website that’s visited by millions”, although the “single point of failure” argument might have been eliminated in this setup.

Avatar

Single company of failure : if Raspberry Pi goes bust the lack of spare computers to run their website is not high on the list of problems. I think it unlikely eBay will ever run out. Contrast that to getting a replacement RAID controller in a reasonable time period or even a hot-swap SATA disk with the screws in the right place for super-awkward chassis #6.

Physically there to service it : We don’t yet have full PXE boot, but right now I can remotely re-image and extract the data from the network file system.

No hardware/OS support : Don’t get me started on server manufacturers and RAID controllers with toolsets that only run under Windows / have to embed their own z80 based computer to email you….

ARM & FOSS only : Unfortunately you can’t install terrible unmanageable propriety software that requires you to undergo regular licensing audits. Apparently this is a bad thing.

SD storage : Network filesystem

It’s looking a bit like the year 2000 all over again. People who paid $50k for big Sun boxes told us that Linux on desktop class machines weren’t proper servers and nobody would ever run a serious application on Linux. They sure showed us.

Avatar

Good points from Pete. speaking of which. I noticed the blog has returned to the usual/previous setup of beefy VMs?

Blog VM 1
Blog VM 2

Just out of curiosity.. do you have any plans to reinstate PIs again in the future as taking on some of the workload of the main site?

Avatar

I’ve been wanting PXE boot support since my first Pi; I use a lot of them as small network servers.

Alternatively network disk support; NFS, iSCSI or AoE (https://en.wikipedia.org/wiki/ATA_over_Ethernet) could work.

Leave a Comment

Comments are closed