High RPI 3 board failures

[LuizCressoni]

Hi all,

There is a project we've been working on for a few years with around 300 units sold, using the Rpi 3.
So far, we go some boards that failed for different reasons:

- 9 without video output on the HDMI connector
- 2 with Ethernet not working
- 9 with I2C not working (there is just a DS3232M RTC (correctly) connected to the I2C bus)
- 2 'burned' - some ICs get very hot when bod is powered
- 2 with 'burned' GPIO pins

Most of them after some time running - from a week up to a few month
That makes around 8% of malfunctioning boards. Except for the GPIO pins that may have got burned due to improper use of the hardware, I'm a bit concerned about the other ones, specially the ones with no video signal - it is just a HDMI cable connected to a Samsung monitor.
To compare, we have a similar product (also similar quantity sold) running on Beaglebone Black with *NO* failures at all so far (knock on wood).

Have you guys found similar percentage of failures?


Thanks,

[wh7qq]

You don't say if the failures are in your hands or the customers. If the customers, perhaps the level of sophistication of the user is in play. The RPis may seem to be more approachable by the uninitiated while the more costly Beagle Bones may appeal to a client who is more familiar with the SBC genrre. The BB's may also be a bit more robust...I can't speak to that...but it would be interesting to know how many failures are the result of user error or failure of attached devices as opposed to spontaneous component death. You should know the root cause of failures before you start suggesting a high failure rate of RPis due to quality.

My own experience is limited to my small sample of personally used RPis of various models but I have yet to experience a failure even due to my ham-fisted handling of the RPi and the occasional miss-connection of pins. I have zero's, B+'s, 2 and 3B models and some have run 24/7 without a hiccup for years.

[ejolson]

You don't say if the failures are in your hands or the customers. If the customers, perhaps the level of sophistication of the user is in play. The RPis may seem to be more approachable by the uninitiated while the more costly Beagle Bones may appeal to a client who is more familiar with the SBC genrre. The BB's may also be a bit more robust...I can't speak to that...but it would be interesting to know how many failures are the result of user error or failure of attached devices as opposed to spontaneous component death. You should know the root cause of failures before you start suggesting a high failure rate of RPis due to quality.

My own experience is limited to my small sample of personally used RPis of various models but I have yet to experience a failure even due to my ham-fisted handling of the RPi and the occasional miss-connection of pins. I have zero's, B+'s, 2 and 3B models and some have run 24/7 without a hiccup for years.

Almost all replies you get here will be anecdotal with respect to a smaller sample size than 300. My experience with about 10 units is that the single Pi 3B I purchased never worked quite properly but every other Pi including the current 3B+ have worked flawlessly. Unless you have a need for 1.2 GHz peak performance, in applications where reliability is important I would recommend routinely under clocking the 3B to 2B speeds.

Given the market leadership that the Raspberry Pi currently enjoys, I think it is important to take reliability extra seriously. To this end it would be useful if the manufacturer provided a complete field-deployable hardware diagnostic tool. If the tool had an opt-in capability for remote gathering of anonymised performance metrics, my suspicion is that your reliability question could soon be answered based on statically significant data.

[jamesh]

Hi all,

There is a project we've been working on for a few years with around 300 units sold, using the Rpi 3.
So far, we go some boards that failed for different reasons:

- 9 without video output on the HDMI connector
- 2 with Ethernet not working
- 9 with I2C not working (there is just a DS3232M RTC (correctly) connected to the I2C bus)
- 2 'burned' - some ICs get very hot when bod is powered
- 2 with 'burned' GPIO pins

Most of them after some time running - from a week up to a few month
That makes around 8% of malfunctioning boards. Except for the GPIO pins that may have got burned due to improper use of the hardware, I'm a bit concerned about the other ones, specially the ones with no video signal - it is just a HDMI cable connected to a Samsung monitor.
To compare, we have a similar product (also similar quantity sold) running on Beaglebone Black with *NO* failures at all so far (knock on wood).

Have you guys found similar percentage of failures?


Thanks,

That is a very unusually high failure rate. I would not expect ANY failures of the 3B under those circumstances. I'll flag this up to the rest of the team for comment.

EDIT: Please read your PM's, I sent one with the email of someone who will try and help you out.

[jamesh]

Hi all,

There is a project we've been working on for a few years with around 300 units sold, using the Rpi 3.
So far, we go some boards that failed for different reasons:

- 9 without video output on the HDMI connector
- 2 with Ethernet not working
- 9 with I2C not working (there is just a DS3232M RTC (correctly) connected to the I2C bus)
- 2 'burned' - some ICs get very hot when bod is powered
- 2 with 'burned' GPIO pins

Most of them after some time running - from a week up to a few month
That makes around 8% of malfunctioning boards. Except for the GPIO pins that may have got burned due to improper use of the hardware, I'm a bit concerned about the other ones, specially the ones with no video signal - it is just a HDMI cable connected to a Samsung monitor.
To compare, we have a similar product (also similar quantity sold) running on Beaglebone Black with *NO* failures at all so far (knock on wood).

Have you guys found similar percentage of failures?


Thanks,

Please read your PM's. We would really like to understand the problem here.

[DougieLawson]

Please read your PM's. We would really like to understand the problem here.

ucp.php?i=ucp_pm is a quick link to forum PMs.

[LuizCressoni]

WH7QQ, I read my post multiple times trying to find out if I wrote it indicating that I lack confidence in the RPI hardware but I found no evidence of that. And, to be clear, I don't. The RPI is an amazing piece of hardware.
All the concern is on the fact that I just sold another 800 units of my product and I must figure out the real cause of issues.

The mention to another platform was to try to say something like "the attached extra hardware is the same on both implementations, so you can't blame me for burning the module due to some basic error without discussing it a little bit deeper". Of course my hardware may have flaws and be the source of problems in a way that the BB can deal with it and the RPI can't.

And yes, the project that uses the BB runs inside a metal cabinet, grounded, with no external access to the hardware while the one on the RPI is sent to the costumers without any sort of protection - and this MAY be the issue.

DougieLawson sent me a contact so we'll discuss this and see what I did wrong
BTW, I guess I'm a bit dumb, but I can't find where are the PMs...

Thanks,
Luiz

[jamesh]

WH7QQ, I read my post multiple times trying to find out if I wrote it indicating that I lack confidence in the RPI hardware but I found no evidence of that. And, to be clear, I don't. The RPI is an amazing piece of hardware.
All the concern is on the fact that I just sold another 800 units of my product and I must figure out the real cause of issues.

The mention to another platform was to try to say something like "the attached extra hardware is the same on both implementations, so you can't blame me for burning the module due to some basic error without discussing it a little bit deeper". Of course my hardware may have flaws and be the source of problems in a way that the BB can deal with it and the RPI can't.

And yes, the project that uses the BB runs inside a metal cabinet, grounded, with no external access to the hardware while the one on the RPI is sent to the costumers without any sort of protection - and this MAY be the issue.

DougieLawson sent me a contact so we'll discuss this and see what I did wrong
BTW, I guess I'm a bit dumb, but I can't find where are the PMs...

Thanks,
Luiz

Unfortunately PM's have just been turned off for safeguarding reasons, so can you please send an email to info@raspberrypi.org, for the attention of me (James Hughes), with your contact details and information on the issue, and I will put you in touch with an engineer who should be able to help you. We are interested to find out why you are getting this level of failure, as we don't see it elsewhere.

[mark3112]

The company I work for has sold about 500 Pi based Lighting Control Desks and expect to move about 1000 this year (want to test ruggedness? Give it to the Stage Crew).
A new Pi product (office type lighting controller) that I am involved in has just reached the point of bulk deliveries to installers.
Another is in development with prototype boards already sent to the App Integration Engineers.

So far the number of returns is 1. My product, and when I examined it I didn’t find any problems.

The only problem with the Pi that I am aware of is when we were trying to get the Desk through the UK/EU RFI compliance tests. For some reason the HDMI extension lead used was a real problem at the time. My board has the HDMI and Audio turned off. It passed with only one problem and that wasn’t with the Pi but the DC-DC convertors driving the CAN interfaces.

The Desk uses very few connections between the Pi and host board (excepting HDMI and Touchscreen). The Lighting Controller is maxed out on the Pi’s 40 pin header and uses i2c GPIO Expanders to handle the non urgent stuff.

I understand that the number of faults you report are concerning and whilst they manifest in different ways I would suggest there is something fundamental going wrong and the damage is a symptom not the cause.

My suggestion – Opto Isolate the Pi (to protect it) and blow a fuse if overvoltage (when running on a generator).

[jamesh]

One of our engineers has received some data from the customer. None of the failures being reported are something we have ever heard of, so we are pretty certain its something being done by the customer. We'll be getting in contact to see if we can narrow down the problem, but initial thoughts are it's external to the Pi itself.

[LuizCressoni]

Hello all,

Just to update the post, we keep selling our product and now the total count is 930 units, running from Mexico down to Chile as both vending machines and Video Lottery Terminals (slot machines).
We had only ONE new issue, so the percentage dropped to 2.6% of board failures.
More than that, we are (obviously) quite confident that the initial large number of failures was due to some local (client) reasons.
On the carrier board there was an important issue to fix, regarding a MAX232 RS-232 driver, where the chip randomly got extremely hot (we observed brown spots on the back of the PCB!) but after a brief research we found out this is a known MAX232 issue and that it didn't affect the behavior or stability of the Raspberry. A fix for that was applied and we're running fine so far.

Thanks a lot!!!!

[jamesh]

Thanks for reporting back - glad you have got it all sorted!