ganzgustav22
Posts: 59
Joined: Tue Feb 11, 2020 1:04 pm

Yesterday, another 3 Pi4Bs 2GB died within 2 hours. Now it's 5 out of 20 dead.

Wed Jul 01, 2020 7:42 am

Yesterday, another 3 Pi4B 2GB Rev 1.1 died within about 2 hours. Now it's 5 dead out of 20.

One just stopped responding while working on it logged in via SSH. Re-plugged power, now it shows a steady red light and a seemingly PWM-dimmed green light. The next one died after I did a reboot via the console by typing "reboot" and that was it. It did shutdown, but did not come back up again. The last one died like the first one, i.e. it just stopped responding.

The other two ones died a few months ago in the same ways. Until yesterday, I put it off as "well, maybe I was just unlucky" (Although I have dozens of older Pi models that never died without a reason). Now after 5 dead out of 20, I don't believe in bad luck anymore.

The 5 dead Pis now show the following behaviour (without sdcard inserted and nothing connected but the official power supply):
1. Steady red LED, steady PWM-dimmed green LED
2. Steady red and green LED, then the green LED goes off and blinks on 4 times repeatedly
3. Steady red and green LED, then the green LED goes on and blinks off 4 times repeatedly
4. Red led blinks very short repeatedly, green LED stays off
5. Steady red LED, steady PWM-dimmed green LED

All those Pi4Bs were always properly mounted in cases (i.e. I didnt' short something), never got hotter than 65 degrees and were run solely on official power supplies.

Haven't tried EEPROM recovery yet. Is there anything I can to do diagnose this further?

Is it correct, that the firmware has the ability to log to the serialport during bootup, but that functionality is disabled by default and now it cannot be enabled anymore because that would require a working Pi4B to update the eeprom? Would it make sense to enable this by default in the future so that we can see what is going on?


ganzgustav22
Posts: 59
Joined: Tue Feb 11, 2020 1:04 pm

Re: Yesterday, another 3 Pi4Bs 2GB died within 2 hours. Now it's 5 out of 20 dead.

Wed Jul 01, 2020 7:57 am

No, they also don't work with fresh sdcards, the Pis are broken. (Edit: Maybe not totally broken as I haven't tried the eeprom recovery yet, will report on that later when I find time).

ganzgustav22
Posts: 59
Joined: Tue Feb 11, 2020 1:04 pm

Re: Yesterday, another 3 Pi4Bs 2GB died within 2 hours. Now it's 5 out of 20 dead.

Wed Jul 01, 2020 8:11 am

Well great. I just tried the eeprom recovery on a working Pi. As it says in the documentation, the screen turned green to indicate success (it wasn't solid green, but green bars), the green LED blinked fast to indicate success. Powered it off, put in a working sdcard, powered it back on, now it's number six that is also dead.

6. Steady red LED, green light flickers shortly, then stays off.

timg236
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 505
Joined: Thu Jun 21, 2018 4:30 pm

Re: Yesterday, another 3 Pi4Bs 2GB died within 2 hours. Now it's 5 out of 20 dead.

Wed Jul 01, 2020 8:21 am

ganzgustav22 wrote:
Wed Jul 01, 2020 7:42 am

The 5 dead Pis now show the following behaviour (without sdcard inserted and nothing connected but the official power supply):
1. Steady red LED, steady PWM-dimmed green LED
2. Steady red and green LED, then the green LED goes off and blinks on 4 times repeatedly
3. Steady red and green LED, then the green LED goes on and blinks off 4 times repeatedly
4. Red led blinks very short repeatedly, green LED stays off
5. Steady red LED, steady PWM-dimmed green LED
The ROM doesn't drive the activity LED so except for case 4 it looks as though the second stage has been loaded from the EEPROM and run.

2,3 would indicate that the error is trapped and reported correctly.
1,5 is possibly an untrapped SD-CARD error where it's retrying although that's pretty unusual.

The most likely explanation is damage to the SD voltage switch or SOC IO pads. It might be worth checking that there is no physical damage to the SD-cards or anything which could cause a short.

There's no built-in debug that can be enabled via a GPIO/UART because there are not enough pins to have a standard DEBUG pin.

If you have a USB serial cable then creating a rescue image with the Raspberry Pi Imager and adding uart_2ndstage=1 to config.txt would force some debug output from recovery.bin.

This will probably report SD card errors or a hash mismatch if the read fails or returns corrupted data. However, if it does happen to work then the new bootloader EEPROM will have USB boot support and the HDMI diagnostics screen.

timg236
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 505
Joined: Thu Jun 21, 2018 4:30 pm

Re: Yesterday, another 3 Pi4Bs 2GB died within 2 hours. Now it's 5 out of 20 dead.

Wed Jul 01, 2020 8:23 am

ganzgustav22 wrote:
Wed Jul 01, 2020 8:11 am
Well great. I just tried the eeprom recovery on a working Pi. As it says in the documentation, the screen turned green to indicate success (it wasn't solid green, but green bars), the green LED blinked fast to indicate success. Powered it off, put in a working sdcard, powered it back on, now it's number six that is also dead.

6. Steady red LED, green light flickers shortly, then stays off.
That's almost certainly a power supply issue. The solid green is just the HVS background colour register since there is no frame-buffer.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 26398
Joined: Sat Jul 30, 2011 7:41 pm

Re: Yesterday, another 3 Pi4Bs 2GB died within 2 hours. Now it's 5 out of 20 dead.

Wed Jul 01, 2020 8:31 am

This is going to sound harsh, but...

To have a >25% failure rate would be quite noticeable if it were happening to everyone. So it cannot be happening to everyone. This implies that it is something you are doing that is breaking these Pi4's. So we need to figure out what.

So, every detail of the installation is needed. Power supply, case design, peripherals plugged in, environment, SD card type. Everything you can think of.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed.
I've been saying "Mucho" to my Spanish friend a lot more lately. It means a lot to him.

ganzgustav22
Posts: 59
Joined: Tue Feb 11, 2020 1:04 pm

Re: Yesterday, another 3 Pi4Bs 2GB died within 2 hours. Now it's 5 out of 20 dead.

Wed Jul 01, 2020 8:36 am

These Pis were run with different (all official) power supplies. I just took a brand-new official power supply out of it's box and tried with that, same thing.

This is how the "green sucess screen" looks:

Image

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 26398
Joined: Sat Jul 30, 2011 7:41 pm

Re: Yesterday, another 3 Pi4Bs 2GB died within 2 hours. Now it's 5 out of 20 dead.

Wed Jul 01, 2020 8:41 am

That is an LCD panel not a standard monitor or TV. What LCD panel is it? Are they all connected to LCD's? Please tell us EVERYTHING about the setup as requested above.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed.
I've been saying "Mucho" to my Spanish friend a lot more lately. It means a lot to him.

chwe
Posts: 150
Joined: Tue Jul 31, 2018 1:35 pm

Re: Yesterday, another 3 Pi4Bs 2GB died within 2 hours. Now it's 5 out of 20 dead.

Wed Jul 01, 2020 9:15 am

ganzgustav22 wrote:
Wed Jul 01, 2020 7:42 am
Is it correct, that the firmware has the ability to log to the serialport during bootup, but that functionality is disabled by default and now it cannot be enabled anymore because that would require a working Pi4B to update the eeprom? Would it make sense to enable this by default in the future so that we can see what is going on?
That would maybe interfere with their previous behavior to use hats the way RPi does. Other SBCs have a dedicated UART header for that which is IMO the most sane implementation (in fact for most of the testing I do, I connect over UART to the boards).
If it doesn't boot up with a freshly written known working SD-card then updating the eeprom might be your last option.
kerry_s wrote: by kerry_s » Wed Jul 01, 2020 7:56 am
https://www.raspberrypi.org/documentati ... arnings.md

sounds to me like your doing something that's trashing the sdcard's.
can be tested easy. Just run f3 (linux) or h2testw for windows, format the whole card and then test it.

ganzgustav22
Posts: 59
Joined: Tue Feb 11, 2020 1:04 pm

Re: Yesterday, another 3 Pi4Bs 2GB died within 2 hours. Now it's 5 out of 20 dead.

Wed Jul 01, 2020 10:13 am

Maybe it makes sense to split the issue?

I see two issues currently:
1. Pis dying without apparent reason
2. A perfectly working Pi being bricked by the recovery eeprom procedure with nothing but official Power supply and monitor connected
jamesh wrote:
Wed Jul 01, 2020 8:41 am
That is an LCD panel not a standard monitor or TV. What LCD panel is it? Are they all connected to LCD's? Please tell us EVERYTHING about the setup as requested above.
It's an LCD Panel mounted inside a kiosk stand. From a technical perspective, I don't see a difference between a panel mounted inside a kiosk stand or inside a monitor case, after all, it's just a case. After googling the numbers I found on the back of the panel used in the kiosk stand it appears to be this one: http://www.udmgroup.com/db/upload/webda ... 054634.pdf

To make sure it's not the monitor, I used the recovery image again with a standard monitor (BenQ G2420HD) on Pi number 2 and I get the same "not-solid-green" image:
Image

Interestingly, on Pi number 2, I can reproduce the behaviour with the recovery image giving a green-bar-screen, but not on the one that got bricked by running the recovery image.

Why didn't I get a solid green screen (Or is this "green-bar-screen" supposed to look like that)?
Why did the recovery image brick a perfectly working Pi?



Other infos you requested:

Case design:
The Pis are mounted inside Kiosk stands. In those Kiosk stands, the Pis are mounted in these DIN Rail cases: https://www.phoenixcontact.com/online/p ... tegory=ALL


Power supply:
Official RPI power supply, the Pis died on different power supplies in different kiosk stands.

Peripherals plugged in:
For the first two that died a few months ago, I don't remember.

Two ones that died yesterday had monitor, ethernet and keyboard connected.
The third one that died yesterday had only monitor and ethernet connected.

The switch they were connected to is an 8-Port Trendnet TPE-S44. That switch is years old and has 6 other devices connected (computers, IP-Phones, etc.) that are also years old and work fine.


Environment:
A room inside a building. No water, humidity or heat. Residential area, no industry nearby, no thunderstorms outside (i.e. mains issues unlikely). Not living in a third world country with power surges or other mains issues, house wiring is fine.


SD cards:
Different ones from Sandisk and Kingston. They work fine in working Raspberrys as well as in two different USB sdcard readers.
Last edited by ganzgustav22 on Wed Jul 01, 2020 10:52 am, edited 1 time in total.

ganzgustav22
Posts: 59
Joined: Tue Feb 11, 2020 1:04 pm

Re: Yesterday, another 3 Pi4Bs 2GB died within 2 hours. Now it's 5 out of 20 dead.

Wed Jul 01, 2020 10:16 am

The most likely explanation is damage to the SD voltage switch or SOC IO pads. It might be worth checking that there is no physical damage to the SD-cards or anything which could cause a short.
Checked all the cards, all fine. Also didn't short out anything as the Pis were always inside the plastic case.

If you have a USB serial cable then creating a rescue image with the Raspberry Pi Imager and adding uart_2ndstage=1 to config.txt would force some debug output from recovery.bin.

This will probably report SD card errors or a hash mismatch if the read fails or returns corrupted data. However, if it does happen to work then the new bootloader EEPROM will have USB boot support and the HDMI diagnostics screen.
Thanks, I'll try to get a USB serial cable and do what you said.

User avatar
neilgl
Posts: 2038
Joined: Sun Jan 26, 2014 8:36 pm
Location: Near Aston Martin factory

Re: Yesterday, another 3 Pi4Bs 2GB died within 2 hours. Now it's 5 out of 20 dead.

Wed Jul 01, 2020 10:40 am

Is that 8-Port Trendnet TPE-S44 switch being used in PoE mode for any pi?

ganzgustav22
Posts: 59
Joined: Tue Feb 11, 2020 1:04 pm

Re: Yesterday, another 3 Pi4Bs 2GB died within 2 hours. Now it's 5 out of 20 dead.

Wed Jul 01, 2020 10:51 am

No, the switch has four PoE ports and the Pis have always been connected to the non-PoE ports.

ganzgustav22
Posts: 59
Joined: Tue Feb 11, 2020 1:04 pm

Re: Yesterday, another 3 Pi4Bs 2GB died within 2 hours. Now it's 5 out of 20 dead.

Wed Jul 01, 2020 11:22 am

Just had a closer look on the PCBs. Maybe they shorted themselves out when one of the many solder balls I found on the PCBs fell off? On the other hand, one of them doesn't have any visible solder balls and two died without physically touching them as I worked over network on them. Also it doesn't make sense for the ones that died after a reboot. But still, finding multiple solder balls on almost every Pi is not very convincing.

(Click on the pictures for larger images)

Image Image Image Image Image Image Image

User avatar
rpdom
Posts: 16980
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: Yesterday, another 3 Pi4Bs 2GB died within 2 hours. Now it's 5 out of 20 dead.

Wed Jul 01, 2020 11:32 am

The soldering on a few of those looks very dodgy. A lot of flux on the board and badly soldered. It looks like a bad repair job.

I suspect RPT will be interested in getting the manufacturing details of those if you can get a good picture of the 2D barcode label on each one.
Unreadable squiggle

Heater
Posts: 15794
Joined: Tue Jul 17, 2012 3:02 pm

Re: Yesterday, another 3 Pi4Bs 2GB died within 2 hours. Now it's 5 out of 20 dead.

Wed Jul 01, 2020 11:37 am

I don't see any solder balls or obviously dry joints.

But some of that soldering does look pretty poor.

The sloppy flux residue looks like a repair job. Some of that solder looks like corrosion after the boards have been left out in a field for a year.
Memory in C++ is a leaky abstraction .

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 26398
Joined: Sat Jul 30, 2011 7:41 pm

Re: Yesterday, another 3 Pi4Bs 2GB died within 2 hours. Now it's 5 out of 20 dead.

Wed Jul 01, 2020 11:42 am

Where did you purchase the devices from?
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed.
I've been saying "Mucho" to my Spanish friend a lot more lately. It means a lot to him.

ganzgustav22
Posts: 59
Joined: Tue Feb 11, 2020 1:04 pm

Re: Yesterday, another 3 Pi4Bs 2GB died within 2 hours. Now it's 5 out of 20 dead.

Wed Jul 01, 2020 12:21 pm

They've been purchased from www.reichelt.de

I've never soldered on the Pis and also they have never seen water or moisture. I personally removed them from the original not-opened-before packaging, so I'm sure nobody at Reichelt tampered with them.

Looked through the forum for that issue, seems I'm not the only one, the newer 8GB Pi4s also show this:
viewtopic.php?f=63&t=276652&p=1676343&h ... r#p1676343

ganzgustav22
Posts: 59
Joined: Tue Feb 11, 2020 1:04 pm

Re: Yesterday, another 3 Pi4Bs 2GB died within 2 hours. Now it's 5 out of 20 dead.

Wed Jul 01, 2020 12:42 pm

Out of curiosity just opened two brand-new Pi4B 4GB packages (also bought from Reichelt, just a few months later) and it's the same. Multiple solder balls on different places on the PCB.

(Made some close-ups/crops so it's easier to spot them)

Pi4B 4GB 1
Image

Pi4B 4GB 2
Image

ejolson
Posts: 5127
Joined: Tue Mar 18, 2014 11:47 am

Re: Yesterday, another 3 Pi4Bs 2GB died within 2 hours. Now it's 5 out of 20 dead.

Wed Jul 01, 2020 1:50 pm

ganzgustav22 wrote:
Wed Jul 01, 2020 8:36 am
This is how the "green sucess screen" looks:
I had to reset the EEPROM in a Pi 4B last week. The noisy screen with green bars looked so much like something had crashed, I took a picture of it. Fortunately, the EEPROM reset had actually worked fine and I was up and running again in short time.

Image

I think it is possible a bad SD card might cause repeated failures of the devices it was plugged into. With 20 Pi computers my solution would be to buy just one new SD card from a different retailer and use that card to configure network boot on each of the 20 Pi computers and then use them without SD cards.
Last edited by ejolson on Wed Jul 01, 2020 2:31 pm, edited 3 times in total.

Heater
Posts: 15794
Joined: Tue Jul 17, 2012 3:02 pm

Re: Yesterday, another 3 Pi4Bs 2GB died within 2 hours. Now it's 5 out of 20 dead.

Wed Jul 01, 2020 1:55 pm

Great, the Green Screen Of Not Dead Yet. GSONDY.
Memory in C++ is a leaky abstraction .

timg236
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 505
Joined: Thu Jun 21, 2018 4:30 pm

Re: Yesterday, another 3 Pi4Bs 2GB died within 2 hours. Now it's 5 out of 20 dead.

Wed Jul 01, 2020 2:34 pm

ejolson wrote:
Wed Jul 01, 2020 1:50 pm
ganzgustav22 wrote:
Wed Jul 01, 2020 8:36 am
This is how the "green sucess screen" looks:
I had to reset the EEPROM in a Pi 4B last week. The noisy screen with green bars looked so much like something had crashed, I took a picture of it. Fortunately, the EEPROM reset had actually worked fine and I was up and running again in short time.

Image

I think it is possible a bad SD card might cause repeated failures of the devices it was plugged into. With 20 Pi computers my solution would be to buy just one new SD card from a different retailer and use that card to configure network boot on each of the 20 Pi computers and then use them without SD cards.
The solid green screen is only displayed if recovery.bin has updated the EEPROM and verified the update was successful. Any SD card or EEPROM issues would cause a red screen to be displayed.
The display mode is always 640x480 DVI - does the panel support that mode?

ejolson
Posts: 5127
Joined: Tue Mar 18, 2014 11:47 am

Re: Yesterday, another 3 Pi4Bs 2GB died within 2 hours. Now it's 5 out of 20 dead.

Wed Jul 01, 2020 2:40 pm

timg236 wrote:
Wed Jul 01, 2020 2:34 pm
ejolson wrote:
Wed Jul 01, 2020 1:50 pm
ganzgustav22 wrote:
Wed Jul 01, 2020 8:36 am
This is how the "green sucess screen" looks:
I had to reset the EEPROM in a Pi 4B last week. The noisy screen with green bars looked so much like something had crashed, I took a picture of it. Fortunately, the EEPROM reset had actually worked fine and I was up and running again in short time.

Image

I think it is possible a bad SD card might cause repeated failures of the devices it was plugged into. With 20 Pi computers my solution would be to buy just one new SD card from a different retailer and use that card to configure network boot on each of the 20 Pi computers and then use them without SD cards.
The solid green screen is only displayed if recovery.bin has updated the EEPROM and verified the update was successful. Any SD card or EEPROM issues would cause a red screen to be displayed.
The display mode is always 640x480 DVI - does the panel support that mode?
Correct, 640x480 is the mode the monitor claimed to be running in. I did not take a picture of that, but remember as I was curious about the same. Is there a known Pi and computer monitor combination where a solid green screen is displayed or is it supposed to be striped?

timg236
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 505
Joined: Thu Jun 21, 2018 4:30 pm

Re: Yesterday, another 3 Pi4Bs 2GB died within 2 hours. Now it's 5 out of 20 dead.

Wed Jul 01, 2020 2:45 pm

ejolson wrote:
Wed Jul 01, 2020 2:40 pm
timg236 wrote:
Wed Jul 01, 2020 2:34 pm
ejolson wrote:
Wed Jul 01, 2020 1:50 pm

I had to reset the EEPROM in a Pi 4B last week. The noisy screen with green bars looked so much like something had crashed, I took a picture of it. Fortunately, the EEPROM reset had actually worked fine and I was up and running again in short time.

Image

I think it is possible a bad SD card might cause repeated failures of the devices it was plugged into. With 20 Pi computers my solution would be to buy just one new SD card from a different retailer and use that card to configure network boot on each of the 20 Pi computers and then use them without SD cards.
The solid green screen is only displayed if recovery.bin has updated the EEPROM and verified the update was successful. Any SD card or EEPROM issues would cause a red screen to be displayed.
The display mode is always 640x480 DVI - does the panel support that mode?
Correct, 640x480 is the mode the monitor claimed to be running in. I did not take a picture of that, but remember as I was curious about the same. Is there a known Pi and computer monitor combination where a solid green screen is displayed or is it supposed to be striped?
It's always solid green. There's no frame-buffer so you just see the HVS background color. The stripes and rough edges might indicate an HDMI signal issue. It would be interesting to know if there are monitors where this is always the case.

Return to “General discussion”