Pete Lomas is a trustee of the Raspberry Pi Foundation, and designed the final hardware that turned into the Raspberry Pi. We’ve had so many questions from you about the manufacturing process that Pete decided to put this post together – he’s been working on it for a couple of months, and we’re very, very grateful. Thanks Pete – and thank you to everybody at the Sony factory!
The basic idea is simple – Attach components a PCB with solder to make mechanical and electrically conductive joints, test and ship.
But how do Sony manage to make 4000 Raspberry Pi Model B’s a day – or more astoundingly, one every 7.5 seconds? On a recent trip to the facility we had a look at how the team at Pencoed actually do it, and some of the technical wizardry and skill they use to make it happen.
The Raspberry Pi design is what is termed double sided SMT and single sided PTH. Translated, this means it has surface mount components (SMT) mounted on both sides of the PCB and through hole (PTH) components just on the top side, with the pins pushing out of the bottom.
The PCBs are actually mounted in a panel (or plaque) of six, christened after Liz’s earlier post a “six-pack of Pis”. This has several functions: first it reduces handling at both bare PCB manufacture and assembly as the PCBs travel together all the way to final test and pack. It also provides an area (waste edge) around the PCBs that the machines use to clamp the panel firmly in place. When you are whacking components down at 5.5 parts per second, you want to avoid vibration. It also allows Sony to get round the fact that some components on the Raspberry Pi come right over the edges. If you look at the design in the photograph you can see there are also small areas of waste laminate between the board. Ideally the designer (i.e. me) wants to avoid this but it is not always possible.
This shows a partially finished six-pack of Pis. The strips with dots on will be broken away to separate out the Pis just before test. The bottom left Pi is being given a quick-process validation test.
The manufacturing line is actually made up of four key processes, bottom SMT, top SMT, PTH, test and pack, but the skill of the Sony team goes much further back than that into production engineering and component procurement.
This is the surface mount line used to build Pis. The front machine (white dome) is the one used to print the solder paste. On the very front of it is a loader that can be filled with PCBs. The line automatically takes them as required.
Ingredients
Sony have strict policy on component procurement, and this ensures that we only get good quality parts. Vendor assessment is rigorous, and we have spent many months trawling through the BOM with Sony and validating any proposed alternatives or substitutes. Remember the issue with the mag-jack we had early days with the factory we were using in China? The Sony team are dedicated to making sure that sort of thing cannot happen.
As well as getting the right parts, they also need them in the right packaging. Everything that is required for surface mount operations has to be on a reel, ideally as large a reel as possible. For some small components that will be 10,000 on a single reel, but these will be used up in just a couple of hours, as there are 42 on each PCB. Every time a reel runs out, the line stops and requires operator intervention, so the larger the better.
Reels of components in stores
To keep the wheels turning, all the machines monitor usage and send requests to the stores themselves for replacements! So by the time they are located and booked out of stores and brought to the line, they arrive just in time before the reel runs out. Empty reels are bar-code scanned off the machine, and the new reel is bar-code scanned back on to eliminate the chance of fitting incorrect components.
This shows a screen from the system that shows the usage of each component on the machine and when it is due to run out.
Printing the solder paste
The first physical operation is to “paste” the PCB with solder paste. The paste is made up of tiny spheres of solder approximately 25μm across. The other component in the paste is flux. This is designed not only to bind the paste together but also prevents oxidation during the soldering process and is an important aspect of getting a reliable joint.
The machine uses a thin stainless-steel stencil, and the solder paste is pushed through tiny apertures onto the PCB. When the stencil is removed, you are left with tiny prints of solder. On the BGA pads for the BCM2835 these solder prints are only 300μm in diameter. If any of these prints is missing, then a solder joint will not be formed. Just sometimes, rather than the paste sticking to the PCB, it stays in the stencil. To check for this the machine does an optical check immediately after solder paste print just to make sure it is there. If there is a problem, the paste can be removed and the PCB re-printed.
This is a typical solder paste stencil (before anyone posts a comment – no, it’s not a Pi one, but shows the general idea.) These are made of stainless steel and are just 0.004″ thick.
The solder paste can be seen on top of the gold pads on the PCB. On larger areas the paste is cut into segments. This controls the amount of paste more accurately and also provides an escape route for gasses that can build up during reflow as the solvents evaporate.
Once the paste print is verified, the surface mount components can be added to the PCB using a SMT mounting machine. Sony use their own-brand machines, but there are lots to choose from. In principle they all operate the same way, with only subtle differences.
The parts are in “pockets” on each reel of tape, and they are picked out using a vacuum nozzle that is fitted to a moving chunk of mechanics and sensor electronics (the mounting head). The problem is that the parts move around in their little pockets, so the alignment of the part relative to the nozzle is somewhat inaccurate. So, if you place a part straight onto the PCB from the reel, it would be out of line. In modern placement machines this problem is solved using a small camera that looks at the part on the nozzle, figures out its exact location and rotation, and then applies correction factors to ensure that when the part is actually placed it is within 40μm of the optimum position. This accuracy is vitally important when you are placing parts that are only 0.5 by 1.0mm, or if you have tens of tightly spaced legs or pads. When they are running at full speed, these machines can place 25,000 components per hour. When you watch them, they move so quickly that it is almost impossible to see the parts being placed; they just sort of “appear”.
Some tantalum capacitors, on a reel. The pockets that hold the individual components can be seen in the tape. A secondary “cover tape” is peeled off by the machine just before the component is used.
It is difficult to move the mounting head quickly; to stop the larger parts being ripped off the nozzle by inertia the head has to accelerate and decelerate smoothly, and this all takes time. The guys who design these placement machines are real speed demons. To speed things up even further the machines can have up to 12 nozzles, and in some cases two heads working alternately. Whilst one is picking up parts the other is placing then on the PCB, and then the roles reverse.
These dual headed-machines can place at 60,000 components per hour (cph). As it turns out, they are over-specified to help us with Pi, which only has 173 surface-mount components. A 25,000 cph machine with 12 heads is sufficient. Typically, a pack of six Pis has its SMT components mounted in just 150 seconds.
Time to do some baking
Once the parts are placed on the PCBs they have to be soldered. (OK: I could say baked – it is a big oven – but don’t try this at home). The technical term is reflow. What we want to do is heat up the solder (and components) so that the little balls fuse to the component, the PCB and each other.
The reflow oven consists of a number of zones, each one getting progressively hotter until the solder melts (reflows), whereupon the joint is made. The subsequent stages then cool the PCB in a controlled way.
Getting this part of the process wrong can be a recipe for disaster. Too cold, and some of the joints will not form; too quick and the ends of some parts will not heat up evenly, leading to an embarrassing process defect called tombstoning. Too hot, and you fry some of the more delicate components, not to mention the PCB; too slow and the flux burns off before it does it job leaving poor joints: you get the idea.
Again, this is an area where technical skill and years of experience come into play; what the process engineers need to do is find what is called the “process window”, where all the factors are good enough to get reliable joints, and then aim for the ideal point somewhere in the middle. They do this by sacrificing some boards (poor Pi), and adding thermocouple instrumentation to critical parts of the board. These are connected to a mole that follows the PCBs, recording the temperature/time profile for each point. These can then be passed through the reflow oven as many times as is required to get the result. I’ve learnt after explaining this to a group of students that you need to know that the mole is in fact a piece of electronics that is heavily protected against the heat, and not a small furry animal. It took me a few seconds to figure why one of them had a really horrified expression!
Now the parts are solidly attached to the PCB the next step is AOI (Automatic Optical Inspection). Here the PCBs are inspected with a high resolution cameras, and the resulting images processed and compared with images from golden (known good) PCBs. This allows all the parts to be checked for presence, correct rotation, joint soldering and generally anything else that looks odd. Anything that looks out of kilter is then checked by a skilled operator. Sometimes the AOI can be over-fussy and give a false fail, but once I had just a few chips in a reel of 5000 that had whole corner missing, the AOI picked it up in an instant.
All these quality processes are important as it allows Sony to drive down the defect rates to almost unbelievable low levels. They utilise quality principles like Kaizen and lean along with six sigma quality targets. Six sigma is a challenging defect rate of 3.4 faults per million. Unfortunately, that does not mean that only 3.4 Pis in a million will have a problem (in the factory); it relates to everything that could go wrong. These potential problems are termed “Defect Opportunities”. From an assembly point of view, a typical Pi has 100’s opportunities for things to go wrong, like a paste print defect, a missing component or a defective solder joint. The screening and test programs are designed to ensure that these Pis do not leave the manufacturing floor. Even after the Pis have passed everything, there is a quality team checking the final output, effectively checking the quality of the quality processes! The objective is that zero defects reach the customer.
Sony are proud of their quality, and make sure that everyone working the line knows what is being achieved.
PoP on top
So having gotten the underside SMT mounted, the whole process is repeated on the next SMT line for the top-side SMT components. In principle, this is just the same, but the processor and its package-on-package (PoP) memory are mounted on this side and some additional trickery is required.
Once the bulk of the components are mounted on a couple of SMT machines, the panel of boards is passed to a special placement machine that does the PoP. The BCM2835 is placed as any normal part, but the memory has to be placed on top of it. Remember: every surface mount pad that is used on the board to connect a component has had solder paste printed. So how do they get it printed onto the top of the processor? Turns out they don’t: they have a clever little tray full of solder paste in which they dip the memory chip gently, to coat the solder balls on the underside of the part, and then place that (carefully) atop the processor, job done!
I said “clever little tray” as it is constantly rotating and has a scraper bar that sets the exact depth of the paste. Coupled with that there is an automatic dispenser control system that adds more paste as required. It’s really neat, and as expected from Sony, it is the best solution giving phenomenal yields in the volumes required.
The round disk at the front is the tray that contains the solder paste – we will replace this with a short video as soon as we can.
The boards then go further down the line for topside reflow (soldering) and AOI inspection, and the SMT processes are complete. One really good process check is to have a look at joint quality, in particular finish and shape. You can get a hint from inspecting at this point that the process is wandering before it becomes an issue. The joints on the PoP memory package and BGAs are in general difficult to see, but Sony has an optical arrangement that allows an expert to have a look at least at the edges of the package. Any minor problems with shape and alignment can be observed early and adjusted out of the process. Sony also have X-ray facilities to detect bridges and missing solder balls on the underside of all BGA devices.
If you look at the maths, something does not add up. Taking 150 seconds to mount the 147 SMT parts on six Raspberry Pis does not equate to the production rate of one every 7.5 seconds. In fact there are three machines contributing to this figure, one building the underside and two for the topside. This points up another important aspect of efficient manufacture: load balancing. In the whole of the Sony process, the production engineers have ensured that each process step on average takes the same time. If it is too slow, it governs the production rate and they add additional equipment to resolve it.
Final assembly: the PTH components
The through-hole (PTH) parts are actually inserted by hand. On the Pi there are just five. The panel of PCBs are mounted on a solder- and heat-resistant carrier. This shields all the surface-mount components on the underside of the PCB so they don’t get desoldered and end up in the bottom of the solder bath when they are soldered by the wave (flow) soldering machine.
This shows the six pack of Pis with all their through-hole components added, about to be wave soldered.
So, how does this work? As the PCBs enter, the area on the underside is sprayed with flux. This stops the pins, pads and solder oxidising, and ensures a good joint. The PCBs are then (pre) heated. This is important, as it stops the solder cooling too quickly when it comes into contact with the PCB. Further on in the machine, there is a wave of molten solder (hence the name). This is continuously flowing and is pumped out and back into a heated solder pot. The height and shape of the wave and the amount of preheat, the solder pot temperature plus the speed of the PCBs is carefully controlled to ensure a quality joint. It needs skill and judgement to get this exactly right.
If the wave is too tall, it can cause components to be pushed out of the PCB, and in the worst case can cause the top of the board to be flooded with solder. This then resembles a lava flow – yep, I’ve done it! When I asked the guys at Sony about lava accidents, I just got a wry, knowing smile.
End of the line
That’s it: the six-pack of Pis is fully assembled. The Pis are then moved over to the test and packing stations. Here the individual Pis are broken out of the panel and placed on a test unit. Each station has two test units so that one can be used for loading/unloading, whilst the other is used to run the tests and also program the various setup codes such as the model and where it was made.
Here, the operator has two test rigs. One is being unloaded/reloaded while the other is testing a Pi.
A Raspberry Pi that passes the test is placed into its antistatic bag and straight into the box ready for dispatch. Any Pi that fails is going to be pretty lonely: when we were there to take these pictures the “Fail” boxes were empty bar one. That failed unit will be investigated quickly to find out what the problem is, and the defect analysed to see if a process optimisation would help. [Liz interjects: I visited Sony last week, and was told that fewer than 20 Pis have ended up in the "Fail" box since Sony started manufacture. Not bad!]
Making products in this volume also puts stress on the component parameters and the design. With such a number of boards going through, the chances of a group of parts ganging up on the designer to create a “corner case” that causes the Pi to fail test is very real. We work closely with Sony to help identify and correct those where possible. We have already identified a couple of minor tweaks to the PCB that help. For me, even after 30 years in the business, there is always something to learn and the talented team at Sony make great teachers.
The Sony team who make your Raspberry Pi. And some interlopers. Click for bigness.
After Liz, Eben and I met all the team, we can confidently say that the Raspberry Pi is made with tender, loving care in Pencoed. We know: we’ve watched it happen.
