jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 26685
Joined: Sat Jul 30, 2011 7:41 pm

Re: Yesterday, another 3 Pi4Bs 2GB died within 2 hours. Now it's 5 out of 20 dead.

Wed Jul 01, 2020 2:51 pm

ganzgustav22 wrote:
Wed Jul 01, 2020 12:42 pm
Out of curiosity just opened two brand-new Pi4B 4GB packages (also bought from Reichelt, just a few months later) and it's the same. Multiple solder balls on different places on the PCB.

(Made some close-ups/crops so it's easier to spot them)

Pi4B 4GB 1
Image

Pi4B 4GB 2
Image
Solder balls are not uncommon, and are generally harmless. Since you bought you Pi4's a long distance in time apart they will have been from very different batches, so I don't think that is the problem.

I think the problem lies in something in your kiosks blowing the Pi4's up. Presumably something the previous models were not as susceptible to, but still something out of spec somewhere. I think that if you run a Pi4 outside that scenario, on a standard monitor, running whatever software you are using, it won't fail.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed.
I've been saying "Mucho" to my Spanish friend a lot more lately. It means a lot to him.

User avatar
jbeale
Posts: 3682
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: Yesterday, another 3 Pi4Bs 2GB died within 2 hours. Now it's 5 out of 20 dead.

Wed Jul 01, 2020 2:57 pm

Correct me if I am wrong, but it seems to me it only takes one of those stray solder balls to make a (possibly intermittent or temperature-dependent) fault, if for example it gets wedged underneath a BGA part, or trapped between pins in a SD card socket. Is the kind of through-hole pin manual rework shown in one of those photos part of the standard manufacturing practice?

aBUGSworstnightmare
Posts: 1996
Joined: Tue Jun 30, 2015 1:35 pm

Re: Yesterday, another 3 Pi4Bs 2GB died within 2 hours. Now it's 5 out of 20 dead.

Wed Jul 01, 2020 2:59 pm

jbeale wrote:
Wed Jul 01, 2020 2:57 pm
Correct me if I am wrong, but it seems to me it only takes one of those stray solder balls to make a (possibly intermittent or temperature-dependent) fault, if for example it gets wedged underneath a BGA part, or trapped between pins in a SD card socket. Is the kind of through-hole pin manual rework shown in one of those photos part of the standard manufacturing practice?
as there is flux my guess is manual repair job where they skipped the cleaning process

ejolson
Posts: 5424
Joined: Tue Mar 18, 2014 11:47 am

Re: Yesterday, another 3 Pi4Bs 2GB died within 2 hours. Now it's 5 out of 20 dead.

Wed Jul 01, 2020 3:06 pm

jbeale wrote:
Wed Jul 01, 2020 2:57 pm
Correct me if I am wrong, but it seems to me it only takes one of those stray solder balls to make a (possibly intermittent or temperature-dependent) fault, if for example it gets wedged underneath a BGA part, or trapped between pins in a SD card socket. Is the kind of through-hole pin manual rework shown in one of those photos part of the standard manufacturing practice?
Given the white lines drawn between the pads likely to short, it seems possible that a robotic AI checks to see if the line is visible and if not sends the Pi to the room where the socially-distanced solder people live. As I doubt the process is top secret nor the exact details needed, it would be of educational value to have a 10 minute video to show students how Pi computers are manufactured and tested. A quick discussion of the statistics involved in quality assurance would be nice to include somewhere in the video.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 26685
Joined: Sat Jul 30, 2011 7:41 pm

Re: Yesterday, another 3 Pi4Bs 2GB died within 2 hours. Now it's 5 out of 20 dead.

Wed Jul 01, 2020 3:23 pm

Hi ganzgustav22

Can you email me at james.hughes@raspberrypi.com? We'd like to get one or more of the broken Pi's with solder balls back for internal testing/checking. All returns will obviously be replaced along with some goodies to cover the cost of postage etc.

Thanks.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed.
I've been saying "Mucho" to my Spanish friend a lot more lately. It means a lot to him.

Puffergas
Posts: 94
Joined: Thu Dec 19, 2019 12:16 am

Re: Yesterday, another 3 Pi4Bs 2GB died within 2 hours. Now it's 5 out of 20 dead.

Wed Jul 01, 2020 6:42 pm

I have damaged about 3 Pi's. All from static discharge and my fault. I know the Pi's are tested for static discharge, during design phase, however my dry winter months can zap anything. Just something to consider when handling any bare board. A zapped board can fail months after being zapped.

I inspect every Pi board for solder balls and remove them before the first power up. Pi's are a consumer product and would not require the quality standards needed for custody transfer or the aerospace industry.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 26685
Joined: Sat Jul 30, 2011 7:41 pm

Re: Yesterday, another 3 Pi4Bs 2GB died within 2 hours. Now it's 5 out of 20 dead.

Wed Jul 01, 2020 6:48 pm

Puffergas wrote:
Wed Jul 01, 2020 6:42 pm
I have damaged about 3 Pi's. All from static discharge and my fault. I know the Pi's are tested for static discharge, during design phase, however my dry winter months can zap anything. Just something to consider when handling any bare board. A zapped board can fail months after being zapped.

I inspect every Pi board for solder balls and remove them before the first power up. Pi's are a consumer product and would not require the quality standards needed for custody transfer or the aerospace industry.
But we don't expect too many solder balls, so we are investigating.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed.
I've been saying "Mucho" to my Spanish friend a lot more lately. It means a lot to him.

tombsar
Posts: 46
Joined: Wed May 13, 2020 12:40 am

Re: Yesterday, another 3 Pi4Bs 2GB died within 2 hours. Now it's 5 out of 20 dead.

Wed Jul 01, 2020 7:30 pm

jamesh wrote:
Wed Jul 01, 2020 8:31 am
To have a >25% failure rate would be quite noticeable if it were happening to everyone. So it cannot be happening to everyone. This implies that it is something you are doing that is breaking these Pi4's. So we need to figure out what.
Just anecdotal, but I've so far had four Pi4s (two bought within a week of release, one a few months later), and exactly one of them had to be returned as it died completely after two days' use.

rwcooper
Posts: 41
Joined: Wed Dec 30, 2015 9:04 pm

Re: Yesterday, another 3 Pi4Bs 2GB died within 2 hours. Now it's 5 out of 20 dead.

Wed Jul 01, 2020 11:34 pm

I've just looked at the PCB of one of my 4GB RPi 4's and it's clean. I notice that the silk screen of TP3 and TP23 are messy. I've examined a couple of other Pi's as well and the silk screening on them is perfect as well. So here's a wacky idea: counterfeit?

Heater
Posts: 15998
Joined: Tue Jul 17, 2012 3:02 pm

Re: Yesterday, another 3 Pi4Bs 2GB died within 2 hours. Now it's 5 out of 20 dead.

Wed Jul 01, 2020 11:42 pm

rwcooper wrote:
Wed Jul 01, 2020 11:34 pm
So here's a wacky idea: counterfeit?
Yes, that is wacky.

Firstly it's pretty much impossible to by the Broadcom SoC used on the Raspberry Pi.

Secondly, even if one could I very much doubt anyone could make Pi clones using them cheaper than they are already. There would be no profit in it.

Of course one could try to sell Pi look a likes that don't actually use the Broadcom SoC. But they would not work at all and we would all soon know about it.
Memory in C++ is a leaky abstraction .

ejolson
Posts: 5424
Joined: Tue Mar 18, 2014 11:47 am

Re: Yesterday, another 3 Pi4Bs 2GB died within 2 hours. Now it's 5 out of 20 dead.

Thu Jul 02, 2020 1:03 am

Heater wrote:
Wed Jul 01, 2020 11:42 pm
rwcooper wrote:
Wed Jul 01, 2020 11:34 pm
So here's a wacky idea: counterfeit?
Yes, that is wacky.
If the person who made the original post actually sends one in as requested, it would be nice to have an official followup posted here about what was wrong and what has been done about it.

I agree that counterfeit is not probable. From what I understand, the manufacturers of similar looking single-board computers are happy to differentiate themselves from the Raspberry Pi in order to sell at a higher price. For example, compared to the 4GB Pi at $55 (shipping included) the 4GB Odroid C4 costs $50 plus shipping (about $70 total). Similarly, the 4GB FriendlyARM M4V2 lists at $70 (shipping is again extra). Then there's the 4GB Jetson Nano at $99. Though the corrosion on the solder looks like something that may have been left out in the rain, it seems unlikely a third party has been searching through the rubbish to find Pi computers that failed the quality testing in order to repackage and resell them.

As a prototype to develop software for provisioning a high-security cloud made out single-tenant Pi computers, I've been stuffing raspberry pies into a 2U chassis. So far all have booted; however, one of them came with a red LED that doesn't light up. I asked the local expert here who barked something about bathtubs. From what I understand, semiconductors either break immediately (even without 5V connected to the data lines) or not at all.

Doug_
Posts: 47
Joined: Tue Oct 08, 2019 6:40 pm

Re: Yesterday, another 3 Pi4Bs 2GB died within 2 hours. Now it's 5 out of 20 dead.

Thu Jul 02, 2020 9:06 am

No mention is made of the actual kiosk application, I don’t intend to pry but if people are motivated to break security or force an external reset then I’ve seen some drastic means taken. I’ve seen ceramic gas igniters (HV spark generators) used to try breaking into video game machines - something like this could cause damage like you have observed.
My comment may be out of line - but some lateral thinking may be required.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 26685
Joined: Sat Jul 30, 2011 7:41 pm

Re: Yesterday, another 3 Pi4Bs 2GB died within 2 hours. Now it's 5 out of 20 dead.

Thu Jul 02, 2020 9:56 am

ejolson wrote:
Thu Jul 02, 2020 1:03 am
If the person who made the original post actually sends one in as requested, it would be nice to have an official followup posted here about what was wrong and what has been done about it.
The solder balls should not be there , so we will talk to the factory and get that sorted out. In this particular case I do no think the solder balls are the culprit - I think its something else in the kiosk application (We are not seeing a large number of problems with Pi4 in general). Getting the boards back will help us sort out the solder balls issue, and might help with the other issue.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed.
I've been saying "Mucho" to my Spanish friend a lot more lately. It means a lot to him.

chwe
Posts: 154
Joined: Tue Jul 31, 2018 1:35 pm

Re: Yesterday, another 3 Pi4Bs 2GB died within 2 hours. Now it's 5 out of 20 dead.

Thu Jul 02, 2020 12:09 pm

ejolson wrote:
Thu Jul 02, 2020 1:03 am
For example, compared to the 4GB Pi at $55 (shipping included) the 4GB Odroid C4 costs $50 plus shipping (about $70 total). Similarly, the 4GB FriendlyARM M4V2 lists at $70 (shipping is again extra). Then there's the 4GB Jetson Nano at $99. Though the corrosion on the solder looks like something that may have been left out in the rain, it seems unlikely a third party has been searching through the rubbish to find Pi computers that failed the quality testing in order to repackage and resell them.
ahh well these prices depend on a lot of where you life.. Here you won't get much cheapter then ~68$ for a 4GB Pi4 including shipping (and those selling it locally are more in the 70-73$ range).. For the other boards you've to import them and then pay the import fees (well taxes wouldn't be that hight but shipping companies here have a bad habit of adding a rediculus amount of 'import tax processing'... so your 7$ taxes might blow up to 20-30$ when the processing fee is added). But the boards mentioned are IMO different purpose boards.. m4v2 is a double CSI board with exposed 2x PCIe. The nano is a SoM so it would more be like compute module with a cheap carrier.. and the odroid is a TV box without case. :lol: (okay. it's a beefie tv box).

I had a quick lock at one of my RP4 and I didn't spot such big solderballs on the bottom side. Let's check the other one at home in the evening.

chipace
Posts: 119
Joined: Sat Jun 29, 2019 2:56 am
Location: brown paper bag in a septic tank

Re: Yesterday, another 3 Pi4Bs 2GB died within 2 hours. Now it's 5 out of 20 dead.

Fri Jul 03, 2020 7:02 am

Physical damage might be done to the USB-C socket. Could the boards be powered by driving 5V to the 5V pin on the 40 pin header? I am guilty of powering off my board by pulling the USB-C connector. I really should be pulling the wall adapter out.
It would be interesting to find the root cause.

User avatar
HawaiianPi
Posts: 5856
Joined: Mon Apr 08, 2013 4:53 am
Location: Aloha, Oregon USA

Re: Yesterday, another 3 Pi4Bs 2GB died within 2 hours. Now it's 5 out of 20 dead.

Fri Jul 03, 2020 7:35 am

ganzgustav22 wrote:
Wed Jul 01, 2020 10:13 am
It's an LCD Panel mounted inside a kiosk stand. From a technical perspective, I don't see a difference between a panel mounted inside a kiosk stand or inside a monitor case, after all, it's just a case.
Well, that doesn't appear to be a standard HDMI display, it's an LVDS panel, and it would require a driver board to use with the Pi. Since that is not a standard configuration most here would be using, I suspect we may have found the source of your troubles. What driver board are you using, how is it powered, and how is it connected to the Pi?

ejolson wrote:
Wed Jul 01, 2020 2:40 pm
Is there a known Pi and computer monitor combination where a solid green screen is displayed or is it supposed to be striped?
I tested a RPi Imager recovery SD card on my 4B4 1.1 with a normal HDMI 1080p monitor (HP Pavilion 27xw) and I got a solid green display. Didn't take a picture, sorry, but it was all green with no stripes.
My mind is like a browser. 27 tabs are open, 9 aren't responding,
lots of pop-ups...and where is that annoying music coming from?

hippy
Posts: 7792
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Yesterday, another 3 Pi4Bs 2GB died within 2 hours. Now it's 5 out of 20 dead.

Fri Jul 03, 2020 1:04 pm

HawaiianPi wrote:
Fri Jul 03, 2020 7:35 am
ganzgustav22 wrote:
Wed Jul 01, 2020 10:13 am
It's an LCD Panel mounted inside a kiosk stand. From a technical perspective, I don't see a difference between a panel mounted inside a kiosk stand or inside a monitor case, after all, it's just a case.
Well, that doesn't appear to be a standard HDMI display, it's an LVDS panel, and it would require a driver board to use with the Pi. Since that is not a standard configuration most here would be using, I suspect we may have found the source of your troubles. What driver board are you using, how is it powered, and how is it connected to the Pi?
Indeed. The OP hasn't been entirely clear about exactly how their display is connected to the PI. If it ( or its driver board ) is plugged in via HDMI I cannot see how that would be destroying a Pi but if connected some other way it could.

Others have reported unexplained "green light of death" issues, or the green LED being always on but dimmed, and no one seems to know why, but they are a very small minority with respect to how many Pi 4B's have been sold.

But millions of devices "working perfectly" doesn't prove the absence of an issue. It may be that others simply never encounter the situation which causes the issue to materialise. Toyota and almost all their drivers said there wasn't a problem with "sudden unexpected acceleration" until it was finally proven there was. I remember the LAN9512 overheating issue when many refused to accept there was an issue.

rwcooper
Posts: 41
Joined: Wed Dec 30, 2015 9:04 pm

Re: Yesterday, another 3 Pi4Bs 2GB died within 2 hours. Now it's 5 out of 20 dead.

Fri Jul 03, 2020 8:21 pm

Heater wrote:
Wed Jul 01, 2020 11:42 pm
rwcooper wrote:
Wed Jul 01, 2020 11:34 pm
So here's a wacky idea: counterfeit?
Yes, that is wacky.

Firstly it's pretty much impossible to by the Broadcom SoC used on the Raspberry Pi.

Secondly, even if one could I very much doubt anyone could make Pi clones using them cheaper than they are already. There would be no profit in it.

Of course one could try to sell Pi look a likes that don't actually use the Broadcom SoC. But they would not work at all and we would all soon know about it.
I thought that would be the case. After I posted I searched for 'counterfeit Raspberry Pi' and everyone seems to agree it's extremely unlikely.

chwe
Posts: 154
Joined: Tue Jul 31, 2018 1:35 pm

Re: Yesterday, another 3 Pi4Bs 2GB died within 2 hours. Now it's 5 out of 20 dead.

Mon Jul 06, 2020 1:20 pm

I had a closer look at my RPi4 and I could not spot many solder balls on mine. But what is noticeable is that the solder joints for the sockets (especially USB3) look a bit blurry (maybe cold joint, I'm not an expert on this). See here in comparison RPi4 and NanoPi M4V2 which has IMO nice looking socket joints.

RPi: Image



NanoPi: Image

ejolson
Posts: 5424
Joined: Tue Mar 18, 2014 11:47 am

Re: Yesterday, another 3 Pi4Bs 2GB died within 2 hours. Now it's 5 out of 20 dead.

Mon Jul 06, 2020 4:20 pm

chwe wrote:
Mon Jul 06, 2020 1:20 pm
I had a closer look at my RPi4 and I could not spot many solder balls on mine. But what is noticeable is that the solder joints for the sockets (especially USB3) look a bit blurry (maybe cold joint, I'm not an expert on this). See here in comparison RPi4 and NanoPi M4V2 which has IMO nice looking socket joints.

RPi: Image



NanoPi: Image
I like those photographs--they remind me of outer space art. Maybe the originals could be framed and sold. Do you know what type of solder is used for each board and whether either is lead free?
Last edited by ejolson on Mon Jul 06, 2020 4:52 pm, edited 2 times in total.

thatchunkylad198966
Posts: 244
Joined: Thu Jul 04, 2019 10:21 am
Location: UK, Birmingham

Re: Yesterday, another 3 Pi4Bs 2GB died within 2 hours. Now it's 5 out of 20 dead.

Mon Jul 06, 2020 4:45 pm

ganzgustav22 wrote:
Wed Jul 01, 2020 11:22 am
Just had a closer look on the PCBs. Maybe they shorted themselves out when one of the many solder balls I found on the PCBs fell off? On the other hand, one of them doesn't have any visible solder balls and two died without physically touching them as I worked over network on them. Also it doesn't make sense for the ones that died after a reboot. But still, finding multiple solder balls on almost every Pi is not very convincing.

(Click on the pictures for larger images)

Image Image Image Image Image Image Image
Both of my 4GB Pi4's look exactly like this.
One man's trash is another man's treasure! :) Pi's I have; Pi Zero, Pi Zero W, Pi 2 x2, Pi 3 x2, Pi 4 4GB x2.

Heater
Posts: 15998
Joined: Tue Jul 17, 2012 3:02 pm

Re: Yesterday, another 3 Pi4Bs 2GB died within 2 hours. Now it's 5 out of 20 dead.

Mon Jul 06, 2020 5:06 pm

trejan wrote:
Mon Jul 06, 2020 4:48 pm
All Pi boards are complaint with RoHS + REACH so will be lead free.
That is no doubt contributing to these occasional issues.
Memory in C++ is a leaky abstraction .

chwe
Posts: 154
Joined: Tue Jul 31, 2018 1:35 pm

Re: Yesterday, another 3 Pi4Bs 2GB died within 2 hours. Now it's 5 out of 20 dead.

Mon Jul 06, 2020 6:35 pm

ejolson wrote:
Mon Jul 06, 2020 4:20 pm
I like those photographs--they remind me of outer space art. Maybe the originals could be framed and sold. Do you know what type of solder is used for each board and whether either is lead free?
that was quick and dirty 'put it under the microscope, make 20 pictures on different high' and then stack them with a free tool to get depth of field. If you wanna make art of it feel free just let me know when you sold one for a million or two...Then some cashback would be appreciated.. :lol:

light is obviously not perfect... this microscope is used to look at crystals where normally reflected-light microscopy isn't used. But both joints have the same zoom and roughly the same size, so they're imo comparable.

Technocolour
Posts: 122
Joined: Thu Jul 04, 2019 6:23 pm

Re: Yesterday, another 3 Pi4Bs 2GB died within 2 hours. Now it's 5 out of 20 dead.

Mon Jul 06, 2020 7:11 pm

chwe wrote:
Mon Jul 06, 2020 1:20 pm
I had a closer look at my RPi4 and I could not spot many solder balls on mine. But what is noticeable is that the solder joints for the sockets (especially USB3) look a bit blurry (maybe cold joint, I'm not an expert on this). See here in comparison RPi4 and NanoPi M4V2 which has IMO nice looking socket joints.

RPi: Image



NanoPi: Image
If it's non leaded (and it should), a high quality solder joint should look matte.

That fact was something that Japanese firms had a hard time coming to terms with.

(The previous blocks of text was brought to you by iirc)

trejan
Posts: 2200
Joined: Tue Jul 02, 2019 2:28 pm

Re: Yesterday, another 3 Pi4Bs 2GB died within 2 hours. Now it's 5 out of 20 dead.

Mon Jul 06, 2020 7:28 pm

Heater wrote:
Mon Jul 06, 2020 5:06 pm
trejan wrote:
Mon Jul 06, 2020 4:48 pm
All Pi boards are complaint with RoHS + REACH so will be lead free.
That is no doubt contributing to these occasional issues.
Hmm. My previous post that you quoted has disappeared so presumably it was deleted by a moderator. Are we not allowed to even mention the name of the other SBC companies?

Return to “General discussion”