Launch day – what happened to the website?

Liz: As you may recall, back on February 2 we launched a new product. This website buckled a little under the strain (as did some of our partners’ websites). At the time, we promised you a post about what happened here and how we dealt with it, with plenty of graphs. We like graphs.

Here’s Pete Stevens from Mythic Beasts, our hosts, to explain exactly what was going on. Over to you, Pete!

On Monday, the Raspberry Pi 2 was announced, and The Register’s predictions of global geekgasm proved to be about right. Slashdot, BBC News, global trending on Twitter and many other sources covering the story resulted in quite a lot of traffic. We saw 11 million page requests from over 700,000 unique IP addresses in our logs from Monday, around 6x the normal traffic load.

The Raspberry Pi website is hosted on WordPress using the WP Super Cache plugin. This plugin generally works very well, resulting in the vast majority of page requests being served from a static file, rather than hitting PHP and MySQL. The second major part of the site is the forums and the different parts of the site have wildly differing typical performance characteristics. In addition to this, the site is fronted by four load balancers which supply most of the downloads directly and scrub some malicious requests. We can cope with roughly:

Cached WordPress 160 pages / second
Non cached WordPress 10 pages / second
Forum page 10 pages / second
Maintenance page at least 10,000 pages / second

Back in 2012 for the original launch, we had a rather smaller server setup and we just put a maintenance page up and directed everyone to buy a Pi direct from Farnell or RS, both of whom had some trouble coping with the demand. We also launched at 6am GMT so that most of our potential customers would still be in bed, spreading the initial surge over several hours.

This time, being a larger organisation with coordination across multiple news outlets and press conferences, the launch time was fixed for 9am on Feb 2nd 2015 so everything would happen then, apart from the odd journalist with premature timing problems – you know who you are.

Our initial plan was to leave the site up as normal, but set the maintenance page to be the launch announcement. That way if the launch overwhelmed things, everyone should see the announcement served direct from the load balancers and otherwise the site should function as normal. Plan B was to disable the forums, giving more resources to the main blog so people could comment there.

The Launch

turtlebeach

It is a complete coincidence that our director Pete took off to go to this isolated beach in the tropics five minutes after the Raspberry Pi 2 launch.

At 9:00 the announcement went live. Within a few minutes traffic volumes on the site had increased by more than a factor of five and the forum users were starting to make comments and chatter to each other. The server load increased from its usual level of 2 to over 400 – we now had a massive queue of users waiting for page requests because all of the server CPU time was being taken generating those slow forum pages which starved the main blog of server time to deliver those fast cached pages. At this point our load balancers started to kick in and deliver to a large fraction of our site users the maintenance page containing the announcement – the fall back plan. This did annoy the forum and blog users who had posted comments and received the maintenance page back having just had their submission thrown away – sorry. During the day we did a little bit of tweaking to the server to improve throughput, removing the nf_conntrack in the firewall to free up CPU for page rendering, and changing the apache settings to queue earlier so people received either their request page or maintenance page more quickly.

Disabling the forums freed up lots of CPU time for the main page and gave us a mostly working site. Sometimes it’d deliver the maintenance page, but mostly people were receiving cached WordPress pages of the announcement and most of the comments were being accepted.

Super Cache not quite so super

Unfortunately, we were still seeing problems. The site would cope with the load happily for a good few minutes, and then suddenly have a load spike to the point where pages were not being generated fast enough. It appears that WP Super Cache wasn’t behaving exactly as intended. When someone posts a comment, Super Cache invalidates its cache of the corresponding page, and starts to rebuild a new one, but providing you have this option ticked… supercache-anonymouse …(we did), the now out-of-date cached page should continue to be served until it is overwritten by the newer version. After a while, we realised that the symptoms that we were seeing were entirely consistent with this not working correctly, and once you hit very high traffic levels this behaviour becomes critical. If cached versions are not served whilst the page is being rebuilt then subsequent requests will also trigger a rebuild and you spend more and more CPU time generating copies of the missing cached page which makes the rebuild take even longer so you have to build more copies each of which now takes even longer. Now we can build a ludicrously overly simple model of this with a short bit of perl and draw a graph of how long it takes to rebuild the main page based on hit rate – and it looks like this. Supercache performance This tells us that performance reasonably suddenly falls off a cliff at around 60-70 hits/second. At 12 hits/sec (typical usage) a rebuild of the page completes in considerably under a second, at 40 hits/sec (very busy) it’s about 4s, at 60 hits/sec it’s 30s, at 80hits/sec it’s well over five minutes – the load balancers kick in and just display the maintenance page, and wait for the load to die down again before starting to serve traffic as normal again. We still don’t know exactly what the cause of this was, so either it’s something else with exactly the same symptoms, or this setting wasn’t working or was interacting badly with another plugin, but as soon as we’d figured out the issue, we implemented the sensible workaround; we put a rewrite hack in to serve the front page and announcement page completely statically, then created the page fresh once every five minutes from cron picking up all the newest comments. As if by magic the load returned to sensible levels although there was now a small delay on new comments appearing.

Re-enabling the forums

 

With stable traffic levels, we turned the forums back on. And then immediately off again. They very quickly backed up the database server with connections, causing both the forums to cease working and the main website to run slowly. A little further investigation into the InnoDB parameters and we realised we had some contention on database locks, we reconfigured and this happened.

Our company pedant points out that actually only the database server process fell over, and it needed restarted not rebooting. Cunningly, we’d managed to find a set of improved settings for InnoDB that allowed us to see all the tables in the database but not read any data out of them. A tiny bit of fiddling later and everything was happy.

The bandwidth graphs

We end up with a traffic graph that looks like this. raspi-launch-bwgraph On the launch day it’s a bit lumpy, this is because when we’re serving the maintenance page nobody can get to the downloads page. Downloads of operating system images and NOOBS dominates the traffic graphs normally. Over the next few days the HTML volume starts dropping and the number of system downloads for newly purchased Raspberry Pis starts increasing rapidly. At this point were reminded of the work we did last year to build a fast distributed downloads setup and were rather thankful because we’re considerably beyond the traffic levels you can sanely serve from a single host.

Could do a bit better

The launch of Raspberry Pi 2 was a closely guarded secret, and although we were told in advance, we didn’t have a lot of time to prepare for the increased traffic. There’s a few things we’d like to have improved and will be talking to with Raspberry Pi over the coming months. One is to upgrade the hardware adding some more cores and RAM to the setup. Whilst we’re doing this it would be sensible to look at splitting the parts of the site into different VMs so that the forums/database/Wordpress have some separation from each other and make it easier to scale things. It would have been really nice to have put our extremely secret test setup with HipHop Virtual Machine into production, but that’s not yet well enough tested for primetime although a seven-fold performance increase on page rendering certainly would be nice.

Schoolboy error

Talking with Ben Nuttall we realised that the stripped down minimal super fast maintenance page didn’t have analytics on it. So the difference between our stats of 11 million page requests and Ben’s of 1.5 million indicate how many people during the launch saw the static maintenance page rather than a WordPress generated page with comments. In hindsight putting analytics on the maintenance page would have been a really good idea. Not every http request which received the maintenance page was necessarily a request to see the launch, nor was each definitely a different visitor. Without detailed analytics that we don’t have, we can estimate the number of people who saw the announcement to be more than 1.5 million but less than 11 million.

Flaming, Bleeding Servers

Liz occasionally has slightly odd ideas about exactly how web-servers work: 

is-this-thing-on

Now, much to her disappointment we don’t have any photographs of servers weeping blood or catching fire. [Liz interjects: it’s called METAPHOR, Pete.] But when we retire servers we like to give them a bit of a special send-off: here’s a server funeral, Mythic Beasts-style.

9 Comments

Big Birthday Bash – the aftermath

We are all very tender, aching and sleepy. It was a fantastic weekend.

1300 of you came to see us at the University of Cambridge Computer Laboratory over the weekend, where you listened to 24 lecture theatre talks, took part in 14 workshops, shared hundreds of incredible projects you’d made with your Pis, and ate 110 pizzas.

2015-02-28 13.21.11

2015-02-28 13.20.57

The workshops were amazing: thanks so much to everybody who helped run them. Here’s Imogen, age 10, who is a Scratch pro (we loved your maze game, Imogen!): this is the first time she’s ever done any robotics, and we thought her robot turned out just great.

Alan McCullagh came all the way from France, where he runs the Rhône Valley Raspberry Jams, to join the other volunteers teaching kids in the Beginners’ Workshop.

16508273319_41bbbd7aaa_z

(Private note for Alan: ROWER. I said ROWER.)

The projects on display were brilliant. Phil Atkin brought along PIANATRON, his Raspberry Pi synthesiser. Pete from Mythic Beasts (you can only see his hands), who is such a good pianist I’m always too embarrassed to play in front of him, was joined by Jonathan “Penguins” Pallant on the “drums”. (Jonathan gave me an update on the penguins project: the Pis all survived the Antarctic winter; however, the solar panels did not, so some more work’s being done on how to manage power.)

We loved watching kids see the music they were making.

magic keyboard

Some kids learned a bit of history.

2015-02-28 12.46.45

Others got to work on custom devices.

2015-02-28 14.34.33-1

Brian Corteil’s Easter Bunny (which he lent us last year for YRS) made an appearance, and laid several kilos of chocolate eggs.

16072840153_6b9898bddd_z

We found more kids in quiet corners, hacking away together.

kids

Workshops aren’t just for young learners: here’s Dave Hughes, the author of the PiCamera library, giving a PiCamera workshop to some grown-up users.

davepicamera

There were 24 talks: here’s our very own Carrie Anne explaining what we do in education.

16487145207_f8944e2252_z

A certain Amy Mather made a Pi photobooth, the results of which, in this particular instance, I found horrifying.

lizphotobooth

Vendors set up stands to sell Pis and add-ons on both days. Here’s Pimoroni’s stand, as gorgeous as ever.

16675607665_17b9204e05_z (1)

All the cool kids played retro games.

gaming

Poly Core (Sam Aaron and Ben Smith) provided live-coded evening entertainment. (My Mum, who came along for the day, is still adamant that there must have been a tape recorder hidden in a box somewhere.) They were amazing – find more snippets on their Twitter feed.

Dan Aldred brought a newly refined version of PiGlove. The capitalisation of its name is of utmost importance.

Screen Shot 2015-03-02 at 14.17.00

Ben Croston from the Fuzzy Duck Brewery (and author of RPi.GPIO) uses a Raspberry Pi controller in the brewing process, and made us a batch of very toothsome, special edition beer called Irration Ale (geddit?) for the Saturday evening event.

Screen Shot 2015-03-02 at 10.52.17

There was cake.

cake

It was a bit like getting married again.

weddingphoto

There was more cake.

B_AF-wlXIAA70AR

After the beer (and raspberry lemonade for the kids) and cake, several hundred people played Pass the Parcel.

The foyer centrepiece was a talking throne created for an exhibition at Kensington Palace, which we borrowed from its current home in Oxford (thank you to Henry Cooke and Tim Burrell-Saward from ELK for making it, and for your heroic work getting it to Cambridge!) We understand a door had to be removed from its frame to get it here.

throneempty

A selection of members of Team Pi were photographed on it. Please note the apposite labelling – the throne uses a Pi with RFID to read what’s on the slates out loud. (Ross has cheese on his mind because we interrupted his burger for this shot.)

gameofthrones

And we appear to have lost Eben. He was last seen heading towards Bedford in an outsized, Pi-powered Big Trak.

Enormous thanks to all the exhibitors and volunteers – and most especially to Mike Horne, Tim Richardson and Lisa Mather, who made this weekend what it was. We can’t thank the three of you enough.

There was so much more – we were so busy we didn’t get pictures of everything, and I didn’t manage to get to talk to anything like as many of you as I’d like to have done. (Does anybody have a picture of the gerbils?) I’ll add links to other people’s accounts of the weekend’s events as they come in.

Update: here’s Spencer’s blog post, Jarle’s pictures, Ben’s pictures, Les’s pictures, Leo’s pictures, Rob’s blog, Tom’s blog, Alex (RasPi.Tv)’s blog, Phil’s blog – more to come!

Thank you to the University of Cambridge Computer Laboratory for letting us take over the building for the weekend.

Thank you to our incredibly thoughtful and generous sponsors for the pass-the-parcel gifts, the contents of the goodie bags, and other giveaways:

  • 4tronix
  • @holdenweb
  • @ipmb
  • @whaleygeek
  • Adafruit
  • AirPi (Tom Hartley)
  • Bare Conductive
  • Brian Cortiel
  • CamJam
  • CPC
  • CSR
  • Cyntech
  • Dawn Robotics
  • Dexter Industries
  • Django
  • Eduboard
  • Energenie
  • Farnell
  • GitHub
  • IQaudIO
  • Low Voltage Labs
  • Manchester Girl Geeks
  • ModMyPi
  • MyPiFi
  • NewIT
  • No Starch Press
  • O’Reilly
  • PiBorg
  • Pimoroni
  • PiSupply
  • RasPi.TV
  • RealVNC
  • RS Components
  • RyanTeck
  • Sugru
  • The Pi Hut
  • UK Space Agency
  • Watterott
  • Wiley
  • Wireless Things

Tableware and Decorations were kindly sponsored by:

  • @WileyTech
  • @RealVNC

 Wood and Laser Cutting was generously sponsored by:

  • @fablabmcr (FabLab Manchester)

rainbow

33 Comments

Happy birthday to us!

It’s the Raspberry Pi’s third birthday today (or as near as we can get: we launched on February 29 in a leap year). To celebrate we’re having a huge party/conference/scrum over the weekend in Cambridge – we’ve sold 1,300 tickets and I’m currently hiding in the press room to get this post written. I’m on a really overloaded WiFi network, so I’m having trouble uploading pictures at the minute: we’ll have some for you next week. [Edited: here’s just one from this morning – we’ll have lots more to share soon!]

Experimenting with Sonic Pi in the beginners' workshop

Experimenting with Sonic Pi in the beginners’ workshop

 

Three years ago, we made 2,000 little computers, and I remember looking at the pallet, and thinking: “Cripes. Can’t believe we’ve made so many computers. That’s amazing.”

We’ve sold half a million of the things just this month. Thanks to everyone who’s joined us on this extraordinarily weird journey – you’re all brilliant.

This is becoming an annual tradition: Matt Timmons Brown, one of my favourite 15-year-olds, has made us another celebratory video. (Here’s last year’s.) Thank you Matt!

 

 

44 Comments

All change: meet the new MagPi!

Some of you may have sniffed this in the wind: there have been some changes at The MagPi, the community Raspberry Pi magazine. The MagPi has been run by volunteers, with no input from the Foundation, for the last three years. Ash Stone, Will Bell, Ian McAlpine and Aaron Shaw, who formed the core editorial team, approached us a few months ago to ask if we could help with what had become a massive monthly task; especially given that half the team has recently changed jobs or moved overseas.

We had a series of discussions, which have resulted in the relaunch of the MagPi you see today. Over the last few months we’ve been working on moving the magazine in-house here at the Foundation. There’s a lot that’s not changing: The MagPi is still your community magazine; it’s still (and always) going to be available as a free PDF download (CC BY-NC-SA 3.0); it’s still going to be full of content written by you, the community.

We don’t make any money out of doing this. Even if in the future we make physical copies available in shops, we don’t expect to break even on the magazine; but we think that offline resources like this are incredibly important for the community and aid learning, so we’re very happy to be investing in it.

Russell Barnes, who has ten years of experience editing technology magazines, has joined us as Managing Editor, and is heading up the magazine. He’s done an incredible job over the last couple of months, and I’m loving working with him. Russell says:

I’m really excited to be part of The MagPi magazine.

Like all great Raspberry Pi projects, The MagPi was created by a band of enthusiasts that met on the Raspberry Pi forum. They wanted to make a magazine for fellow geeks, and they well and truly succeeded. 

It might look a bit different, but the new MagPi is still very much a magazine for and by the Raspberry Pi community. It’s also still freely available under a Creative Commons license, so you can download the PDF edition free every issue to share and remix.

The MagPi is now a whopping 70 pages and includes a mix of news, reviews, features and tutorials for enthusiasts of all ages. Issue 31 is just a taste of what we’ve got in store. Over the coming months we’ll be showing you how the Raspberry Pi can power robots, fly to the edge of space and even cross the Atlantic!

The biggest thanks, of course, has to go to Ash, Will, Ian, Aaron and everybody else – there are dozens of you – who has worked on The MagPi since the beginning. You’ve made something absolutely remarkable, and we promise to look after The MagPi just as well as you have done.

So – want to see the new issue? Here it is! Click to find a download page.

31

 

103 Comments

Tangram: an open source map rendering library

I have a Raspberry Pi project that I’d love to use street maps for, but it would be a daunting challenge for me to figure out how to read map data and write the code to draw the maps on screen. It’s why I was delighted to discover Tangram ES, which is a library for rendering 2D and 3D maps using OpenGL ES 2 with data from OpenStreetMap. The library works on a number of devices, including of course Raspberry Pi.

Patricio Gonzalez Vivo (from the video above) and the team at Mapzen are responsible for the open source project, which is an offshoot of their WebGL map rendering library, Tangram. While Tangram ES is still a work-in-progress, they’ve been using Raspberry Pi 2 to speed up their development of the library and they’re ready for more people to take it for a spin.

Structured a lot like a research and development lab, Mapzen is a startup founded with the idea that mapping done collaboratively, transparently, and in the open can produce more resilient software, and ultimately, better maps. Their focus is on open source tools and using open data to create the building blocks for future mapping applications, including search & geocoding, routing, and transit, in addition to the rendering work they’re doing with Tangram.

Patricio is a graphics engineer on Tangram, responsible for implementing different graphical features such as tessellation, lights, materials, environmental maps, and other CG effects. The team also includes Brett Camper, who is Mapzen’s co-founder, as well as Peter Richardson, Ivan Willing, and Karim Naaji. The ES version of Tangram was started by Matt Blair and Varun Talwar.

“Last December Karim and I thought it could be interesting to get Tangram ES running on a Raspberry Pi,” said Patricio. “At the beginning we thought it would be difficult and probably slow, but at the end we were surprised by the speed of the app and how easy the implementation was. Cross-platform C++ development is possible!”

“In a way, the Pi is an ideal test platform for developing graphics software that targets low-power systems,” said Matt. “The OpenGL ES 2 implementation on the Pi is the strictest that we’ve encountered, so it has become our gold standard for ensuring correct usage of OpenGL. The only major missing piece on the Pi was a compiler that supports C++11, which Tangram uses extensively. However since the Pi is a complete Linux distribution, installing the packages we needed with apt was a breeze.”

Don’t have to take Matt’s word for it; you can install and test drive Tangram ES on the Raspberry Pi right now:

Installing Tangram ES

Using Raspbian, here’s how to install the Tangram ES library from the command line and execute the included sample code:

sudo apt-get update
sudo apt-get install cmake g++-4.7 libcurl4-openssl-dev
cd ~
git clone https://github.com/tangrams/tangram-es.git
cd tangram-es
git submodule init && git submodule update
make rpi
cd build/rpi/bin
./tangram

14 Comments

Announcement: Creative Technologists 2015-16

Hey everyone!

After much preparation we are super happy to announce an exciting new project from the Raspberry Pi Foundation.

 

Creative Technologists

The Raspberry Pi Creative Technologists is a mentoring programme for creative people interested in technology aged 16 – 21 years old. If your passion is the creative arts, and you’re wondering how you can use technology to enhance that, this is for you.

Ben and I are heading up the programme, and the first year will run from April 2015 to April 2016. We will provide individual and group mentoring via online video chats, industry networking and technical support. It’s free to participate. As well as costs of food, travel and accommodation, each participant will also receive a Raspberry Pi 2 starter kit and a £300 materials grant, and the group will receive a £1000 grant for exhibition costs.

Applications are now open and the deadline is 9am on 30th March 2015.

We are both certified Arts Award Gold Advisers – so participants will have the opportunity to complete Trinity College London’s Arts Award Gold accreditation; a Level 3 Award, a QCF credit value of 15, and 35 UCAS points.

We will also have some amazing partners helping us out with mentoring and site visits: Victoria and Albert Museum Digital Programmes, Writers’ Centre Norwich, FutureEverything, Pimoroni, Saladhouse and Hellicar&Lewis.

For full details on the programme, and how to apply, visit the new Creative Technologists page.

30 Comments

Welcome James to our Education Team

If you visited us at the Bett Show in January, or came to Picademy in October or February half term, then you will recognise James Robinson as one of our education team volunteers. He is a well-established member of the Computing At School community, as both a CAS Master Teacher and CAS Hub Leader for Cambridge. He is also a Raspberry Pi Certified Educator and a frequent attendee of Cambridge Raspberry Jams.

james-r-start

I’ve known James for roughly a year now. He is a hugely successful and experienced teacher whose opinion I have sought on regular occasions. We also seem to keep bumping into him at Computing education events like the CAS Conference, and PyconUK as well as at community events like Piwars. It seemed like we were destined to work together!

James says:

I have always enjoyed tinkering with technology and understanding exactly what’s going on under the surface. To learn more, I studied Computer Science at university, and graduated with first class honours. This enhanced my passion for the subject, and I worked at IBM for a while. I initially trained as a maths teacher, but within a term I was leading an ICT department in a middle school, and offering training to non-specialists. Most recently I worked at Soham Village College as lead teacher for Computing. I am very excited about the introduction of Computing to KS3 and 4, and enjoy testing and developing projects with students. My current interests and projects include: using Raspberry Pi in the classroom, Minecraft Pi, Sonic Pi and High Altitude Ballooning. Looking forward to working on the weather station and getting more schools involved with Pi in the sky!

As part of the Foundation’s Education Team, James will be writing educational resources for the website (especially schemes of work for teachers of KS4), as well as continuing to assist with Picademies and other outreach. James has the best case I’ve ever seen for all his Raspberry Pi bits and bobs, and as soon as I saw it I knew he would fit in around here.

james_lego.JPG

james_lego1

36 Comments

She said yes

Matt Broach made this box, which contains a Pi, to propose to his girlfriend Jackie.

Box o' love

She’s now his fiancée. The box does something at the end of this video that made my heart go boom-biddy-boom. Beautiful job, Matt.

Congratulations to you both from everybody at Pi Towers!

20 Comments

A history of Raspberry Pi in LEGO

There is a significant chance that this is the very best thing on the internet. Richard Hayler and his two boys have built a massive LEGO diorama tracking the history of the Raspberry Pi, from concept to Astro Pi’s visit to the ISS.

architects

The level of detail’s amazing. Here’s a group of mad scientists inventing the Pi:

madscientists

And here’s a primary school with its own Raspberry Pi setup, some deliveries going on in the background.

primaryschool

Here, for some reason, are a PIrate, MOnkey, RObot and NInja hiding in some bushes.

pimoroni

And here’s a lady in a pith helmet.

takingthepith

There are loads more pictures and much more explanation over at Richard’s website: click here, or on any of the pictures to marvel at the enormous detail Richard and the boys have gone into. Bonus points if you can work out what the hotdog guy is all about. (I couldn’t, and I work here.)

largelego

41 Comments

The Raspberry Pi Guy interviews Eben and Gordon

On Monday, Matt Timmons-Brown, The Raspberry Pi Guy, took a day out from revising for his GCSEs to come and do some video interviews with Eben and Gordon. We really enjoy working with Matt; he asks difficult questions, and I think that many of you will find this interview particularly interesting, as Eben talks about plans for open-sourcing the Pi’s graphics stack, what’s going on with the display board, what’s up with Windows 10, and much more.

Thanks Matt – come back to Pi Towers when your exams are over! (Next time, we want more Gordon!)

38 Comments