iosabi
Posts: 4
Joined: Tue Oct 08, 2019 12:12 am

Raspberry Pi 4 bootloader corrupt, can't update EEPROM with recovery.bin

Tue Oct 08, 2019 12:33 am

I have two Raspberry Pi 4 4GB, one of them works just fine, but the other one doesn't boot when using the exact same SDCARD. I suspect the SPI EEPROM is corrupted somehow.

I followed the instructions in https://www.raspberrypi.org/documentati ... teeprom.md but that didn't work, it doesn't seem to be updating the EEPROM. I tried formatting the SDCARD in several ways (FAT16/FAT32, different sizes), including formatting it the same way that the regular boot works on the other Pi. I also tried using the recovery.bin from https://github.com/raspberrypi/rpi-eepr ... r/firmware but no change at all.

The symptoms:
* When plugged in PWR led is solid red.
* green ACT led blinks once very briefly after powering, then remains off.

It doesn't matter if it has a SDCARD or not, the symptoms are the same (ACT blinks only once). Shorting GLOBAL_EN to GND and releasing it causes the green ACT led to blink once.

Some questions for the engineers:
1. Does the rom always try to update the bootloader if the recovery.bin is present? Is there any way to force this?
2. Can you provide a dd image of an sdcard that's proven to update the eeprom?
3. I see two chips near the AV jack that look like two SPI EEPROMS. I can try to hook up the oscilloscope there to see if the rom is trying to load the eeprom and what's there. Which one of the two is the bootloader eeprom? (I assume the one labeled 4H916 064838). Are the SPI pins for this broken out to any of the Test Points or external header? a more complete schematic would be very helpful =/
4. Any concerns with flashing the eeprom using an external programmer?

Thanks!

User avatar
Imperf3kt
Posts: 3215
Joined: Tue Jun 20, 2017 12:16 am
Location: Australia

Re: Raspberry Pi 4 bootloader corrupt, can't update EEPROM with recovery.bin

Tue Oct 08, 2019 2:35 am

I assume you've read the Pi4b section of the.boot sticky?
https://www.raspberrypi.org/forums/view ... 1#p1485558
55:55:44:44:4C
52:4C:52:42:41

iosabi
Posts: 4
Joined: Tue Oct 08, 2019 12:12 am

Re: Raspberry Pi 4 bootloader corrupt, can't update EEPROM with recovery.bin

Tue Oct 08, 2019 10:31 am

Imperf3kt wrote:
Tue Oct 08, 2019 2:35 am
I assume you've read the Pi4b section of the.boot sticky?
https://www.raspberrypi.org/forums/view ... 1#p1485558
Yes, from that and the https://www.raspberrypi.org/documentati ... teeprom.md all tends to indicate corrupted EEPROM. The problem here is that I can't manage to rewrite the eeprom following that procedure.

timg236
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 287
Joined: Thu Jun 21, 2018 4:30 pm

Re: Raspberry Pi 4 bootloader corrupt, can't update EEPROM with recovery.bin

Tue Oct 08, 2019 10:46 am

"* green ACT led blinks once very briefly after powering, then remains off."

Does this happen if you remove the SD-CARD and power on ?

When running the recovery.bin from the downloads page do you get different behaviour with the LEDs ? The first thing that the recovery.bin executable does is enable the green LED then starts printing to the UART. If the green LED doesn't change state then it would indicate that the on-board chip ROM is unable to read recovery.bin from the SD-CARD which would probably indicate that the Pi4 SDCARD reader is damaged.

N.B. It's worth checking in Windows that this really is single FAT32 partition. 64GB cards will be exFAT and the ROM cannot ready exFAT which also causes the brief ACT led then nothing behaviour.

iosabi
Posts: 4
Joined: Tue Oct 08, 2019 12:12 am

Re: Raspberry Pi 4 bootloader corrupt, can't update EEPROM with recovery.bin

Tue Oct 08, 2019 12:08 pm

"Does this happen if you remove the SD-CARD and power on ?"

Yes, same symptoms with or without an sdcard installed.

I probed around the EEPROM with the oscilloscope and found out that the EEPROM is not even read:

1) On my healthy Pi4B without an sdcard, when probing the #CS (pin 1) and CLK (pin 6) of the chip labeled 4H916 next to the AV jack, I see the following sequence:
* #CS stays up for ~100ms, then down for ~180 ms while the EEPROM is read
* CLK goes up for ~12ms, then down for 90ms, then I see a clock of about 3.5MHz with ~1 clock pauses every 8 pulses. This is clearly reading the EEPROM (it looks to me like less than 4Mbit, but didn't measure super accurately)

2) On the other Pi4B, again without the sdcard at all I see:
* #CS goes up and stays there forever (2s+)
* CLK goes up for ~12ms, then down forever
The pi then sits there draining 300mA @ 5V. Nothing obviously dead in the power side.

The green LED blinks very briefly, I measured ~12ms but this happens very early. It starts ~70ms after applying power to the 5V rails in the GPIO header. This is at the same time that CLK goes up for 12ms, so well before we even read the EEPROM (which doesn't happen in the broken Pi4B).

Is there anything earlier than the 4Mbit EEPROM that could be broken? Any hints on how to debug this?
This rpi4 is new, I have booted it only a few times before and if I remember correctly it broke when updating the eeprom from the running OS.

I created the sdcard in linux with gparted, selecting fat16 or fat32. I also tried formating the partition with mkfs.fat which lets you force fat16 or fat32.

timg236
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 287
Joined: Thu Jun 21, 2018 4:30 pm

Re: Raspberry Pi 4 bootloader corrupt, can't update EEPROM with recovery.bin

Tue Oct 08, 2019 1:40 pm

iosabi wrote:
Tue Oct 08, 2019 12:08 pm
"Does this happen if you remove the SD-CARD and power on ?"

Yes, same symptoms with or without an sdcard installed.

I probed around the EEPROM with the oscilloscope and found out that the EEPROM is not even read:

1) On my healthy Pi4B without an sdcard, when probing the #CS (pin 1) and CLK (pin 6) of the chip labeled 4H916 next to the AV jack, I see the following sequence:
* #CS stays up for ~100ms, then down for ~180 ms while the EEPROM is read
* CLK goes up for ~12ms, then down for 90ms, then I see a clock of about 3.5MHz with ~1 clock pauses every 8 pulses. This is clearly reading the EEPROM (it looks to me like less than 4Mbit, but didn't measure super accurately)

2) On the other Pi4B, again without the sdcard at all I see:
* #CS goes up and stays there forever (2s+)
* CLK goes up for ~12ms, then down forever
The pi then sits there draining 300mA @ 5V. Nothing obviously dead in the power side.

The green LED blinks very briefly, I measured ~12ms but this happens very early. It starts ~70ms after applying power to the 5V rails in the GPIO header. This is at the same time that CLK goes up for 12ms, so well before we even read the EEPROM (which doesn't happen in the broken Pi4B).

Is there anything earlier than the 4Mbit EEPROM that could be broken? Any hints on how to debug this?
This rpi4 is new, I have booted it only a few times before and if I remember correctly it broke when updating the eeprom from the running OS.

I created the sdcard in linux with gparted, selecting fat16 or fat32. I also tried formating the partition with mkfs.fat which lets you force fat16 or fat32.
Are you able to measure the SD clock and command lines ? Best guess is that some fault related to the sd-card which has caused the ROM to hang and fail to proceed to the EEPROM, although I've never seen this when the sd-card slot is empty.

iosabi
Posts: 4
Joined: Tue Oct 08, 2019 12:12 am

Re: Raspberry Pi 4 bootloader corrupt, can't update EEPROM with recovery.bin

Wed Oct 09, 2019 9:30 am

Timg236, first of all thank you for following up on this issue. Really appreciate your time looking into it.

I measured the sdcard slot lines without any sdcard in it. All of them remain low, except for the sdcard Vdd which comes high (3v3) ~70ms after power is applied to the 5v lines in the gpio port.

In the healthy pi4b i also have, I see a clock in the CLK line after about 15ms of sdcard Vdd coming up, but this doesn't happen on my other pi4b. I don't see any physical damage, and as I say I was trying to flash the new eeprom from a booted OS when this stopped working.

Is there any JTAG or other debug port I can look at for the main cpu?

timg236
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 287
Joined: Thu Jun 21, 2018 4:30 pm

Re: Raspberry Pi 4 bootloader corrupt, can't update EEPROM with recovery.bin

Wed Oct 09, 2019 11:21 am

iosabi wrote:
Wed Oct 09, 2019 9:30 am
Timg236, first of all thank you for following up on this issue. Really appreciate your time looking into it.

I measured the sdcard slot lines without any sdcard in it. All of them remain low, except for the sdcard Vdd which comes high (3v3) ~70ms after power is applied to the 5v lines in the gpio port.

In the healthy pi4b i also have, I see a clock in the CLK line after about 15ms of sdcard Vdd coming up, but this doesn't happen on my other pi4b. I don't see any physical damage, and as I say I was trying to flash the new eeprom from a booted OS when this stopped working.

Is there any JTAG or other debug port I can look at for the main cpu?
I don't think there's much more that you can do to debug it. However, we'd be interesting in getting the board back so that our hardware engineers can take a look. We'll send you a replacement + Pi goodies to cover the postage & hassle.

Please can you e-mail [email protected] FAO timg236 so that we can exchange contact details ?

timg236
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 287
Joined: Thu Jun 21, 2018 4:30 pm

Re: Raspberry Pi 4 bootloader corrupt, can't update EEPROM with recovery.bin

Tue Oct 22, 2019 11:20 am

The reason that the board was no longer booting is that a resistor had become dislodged which prevented the SOC from powering up properly. It's a tiny resistor so not easy to spot but I've attached an image showing where it was re-worked which allowed the board to boot and run Raspbian. The bootloader EEPROM seemed to be fine.

N.B. The brief flash of the green LED at power on is just due to the power sequencing rather than the ROM enabling it.

https://drive.google.com/open?id=1KtjSr ... begfSVrWDe

User avatar
rpdom
Posts: 16080
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: Raspberry Pi 4 bootloader corrupt, can't update EEPROM with recovery.bin

Tue Oct 22, 2019 11:36 am

Would this have been a manufacturing fault or something that happened during delivery/usage?

Just curious.

Nice reworking, BTW :)

andrum99
Posts: 1021
Joined: Fri Jul 20, 2012 2:41 pm

Re: Raspberry Pi 4 bootloader corrupt, can't update EEPROM with recovery.bin

Tue Oct 22, 2019 1:32 pm

timg236 wrote:
Tue Oct 22, 2019 11:20 am
The reason that the board was no longer booting is that a resistor had become dislodged which prevented the SOC from powering up properly. It's a tiny resistor so not easy to spot but I've attached an image showing where it was re-worked which allowed the board to boot and run Raspbian. The bootloader EEPROM seemed to be fine.

N.B. The brief flash of the green LED at power on is just due to the power sequencing rather than the ROM enabling it.

https://drive.google.com/open?id=1KtjSr ... begfSVrWDe
It's not easy to spot on that photo either - is it the thing to the left of the 2D barcode label?

andrum99
Posts: 1021
Joined: Fri Jul 20, 2012 2:41 pm

Re: Raspberry Pi 4 bootloader corrupt, can't update EEPROM with recovery.bin

Tue Oct 22, 2019 1:35 pm

Comparing that picture to my own Pi 4B4 I see a tiny little resistor next to a larger rectangular block with a shiny top. Just out of interest, can you tell me what the shiny thing is?

andrum99
Posts: 1021
Joined: Fri Jul 20, 2012 2:41 pm

Re: Raspberry Pi 4 bootloader corrupt, can't update EEPROM with recovery.bin

Tue Oct 22, 2019 1:38 pm

rpdom wrote:
Tue Oct 22, 2019 11:36 am
Would this have been a manufacturing fault or something that happened during delivery/usage?

Just curious.

Nice reworking, BTW :)
I was under the impression that all boards were individually tested at the factory, but I may be wrong. It perfectly possible that it was attached firmly enough to test OK then due to mechanical stresses, vibration etc somewhere along the way came loose.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7895
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Raspberry Pi 4 bootloader corrupt, can't update EEPROM with recovery.bin

Tue Oct 22, 2019 1:51 pm

The wire to the left of the 2D barcode shouldn't be there - that's the rework.
Where the left hand end of the wire is soldered down there should be a tiny resistor, which was missing in this case.

Yes it will have been tested in the factory, so somewhere along the line it has become detached.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

rsalazar
Posts: 2
Joined: Tue Oct 22, 2019 2:42 pm

Re: Raspberry Pi 4 bootloader corrupt, can't update EEPROM with recovery.bin

Tue Oct 22, 2019 2:47 pm

I do not know what the problem is for not turning on or detecting the boot .. I do not know if the PI is damaged and I need this to work ... what do I do.

I did what you recommend and it does NOT work .. what can I do

rsalazar
Posts: 2
Joined: Tue Oct 22, 2019 2:42 pm

Re: Raspberry Pi 4 bootloader corrupt, can't update EEPROM with recovery.bin

Tue Oct 22, 2019 3:20 pm

I already solved it .... format the sd card with sd Card Formatter but in overwrite format and then download the rpi-boot-eeprom-recovery-2019-09-10 and insert it into the PI and it worked as it started to flash green led

4d6178
Posts: 1
Joined: Sat Dec 21, 2019 12:20 am

Re: Raspberry Pi 4 bootloader corrupt, can't update EEPROM with recovery.bin

Sat Dec 21, 2019 1:12 am

Hi,
I have a Raspberry Pi 4 with the same issue. Are you sure it is the resistor? I measure 10 Ohms and 16mV Voltage Drop when powered, so there seems to be some current flowing. (1.6mA if what I remember from my Electronics Classes is right)
The Voltages on the Pins of the SD Card Slot come up once at startup and stay there but there seems to be no CLK. I didn't check in depth, it is 2 A.M. right now... :shock:

spandananupam
Posts: 12
Joined: Mon Feb 03, 2020 7:03 pm

Re: Raspberry Pi 4 bootloader corrupt, can't update EEPROM with recovery.bin

Mon Feb 03, 2020 7:36 pm

rsalazar wrote:
Tue Oct 22, 2019 3:20 pm
I already solved it .... format the sd card with sd Card Formatter but in overwrite format and then download the rpi-boot-eeprom-recovery-2019-09-10 and insert it into the PI and it worked as it started to flash green led
Even with this, nothing seems to work for me. I tried different cards, different power supplies etc. My green LED will NOT blink. It will not do anything, and the red LED will just stay on. Is this a native problem in RPi 4?

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 24933
Joined: Sat Jul 30, 2011 7:41 pm

Re: Raspberry Pi 4 bootloader corrupt, can't update EEPROM with recovery.bin

Mon Feb 03, 2020 9:05 pm

spandananupam wrote:
Mon Feb 03, 2020 7:36 pm
rsalazar wrote:
Tue Oct 22, 2019 3:20 pm
I already solved it .... format the sd card with sd Card Formatter but in overwrite format and then download the rpi-boot-eeprom-recovery-2019-09-10 and insert it into the PI and it worked as it started to flash green led
Even with this, nothing seems to work for me. I tried different cards, different power supplies etc. My green LED will NOT blink. It will not do anything, and the red LED will just stay on. Is this a native problem in RPi 4?
Looks like a broken device. If you have tried all that stuff and it still doesn't work.

With no SD card in and power up, do you get any green flashes?
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I own the world’s worst thesaurus. Not only is it awful, it’s awful."

Moonmarch
Posts: 249
Joined: Thu Mar 07, 2019 1:34 am

Re: Raspberry Pi 4 bootloader corrupt, can't update EEPROM with recovery.bin

Mon Feb 03, 2020 10:35 pm

Any LED not blinking or staying on does not mean your RPI computer is broken if you are using a laptop that uses LED lights to tell you when CAPS LOCK or NUM LOCK is on, then you replace the keyboard with a third keyboard that does not have any LED lights you can't assume the keyboard is broken no instead you would say the third party keyboard has less features, because you decided to purchase a cheap replacement keyboard for the laptop, if the LED does not turn on when you use CAPS LOCK check to see if CAPS LOCK is on use a text editor Mousepad then write in the text file to see if CAPS LOCK is actually on.

When you start the RPI computer for the first time the boot up can take a long time after the first boot after downloading, and installing all of the updates the RPI computer should boot faster afterwards, if you do not know how to flash the Raspbian OS image to a SD card then you would need to purchase a SD card that contains NOOBS, after booting the RPI computer with the SD card that contains NOOBS if the computer refuses to start then your RPI computer is out the door, you need to replace the computer forget about soldering or using multimeters replace the computer, here is a link to a SD card for sale with NOOBS installed:

64 Gb micro SD For Raspberry Pi 4 / 3B+ / 3A+ /3B With NOOBS v3.2 (Ready To Use)
https://www.ebay.com/i/283491540894?chn ... 5HEALw_wcB

spandananupam
Posts: 12
Joined: Mon Feb 03, 2020 7:03 pm

Re: Raspberry Pi 4 bootloader corrupt, can't update EEPROM with recovery.bin

Tue Feb 04, 2020 11:44 am

jamesh wrote:
Mon Feb 03, 2020 9:05 pm
spandananupam wrote:
Mon Feb 03, 2020 7:36 pm
rsalazar wrote:
Tue Oct 22, 2019 3:20 pm
I already solved it .... format the sd card with sd Card Formatter but in overwrite format and then download the rpi-boot-eeprom-recovery-2019-09-10 and insert it into the PI and it worked as it started to flash green led
Even with this, nothing seems to work for me. I tried different cards, different power supplies etc. My green LED will NOT blink. It will not do anything, and the red LED will just stay on. Is this a native problem in RPi 4?
Looks like a broken device. If you have tried all that stuff and it still doesn't work.

With no SD card in and power up, do you get any green flashes?
Nope. I have a Pi3, which goes in a loop of green when the SD card isn't installed. This one, however, doesn't do anything. There is a very short pulse of green light that comes up only at the start and that's about it. It's completely red after that.
Here's what all I have tried:
1. I changed the card, tried the same card with a different Pi.
2. Tried changing the power supply
3. Tried an 8GB card instead if a 32GB card
4. Tried reflashing the EEPROM, but it won't even show the green it's supposed to show when it is updating the EEPROM

Is there something that is missing? If this board is done with, is there somewhere I could send it to? Can I do anything?
Last edited by spandananupam on Tue Feb 04, 2020 11:51 am, edited 1 time in total.

spandananupam
Posts: 12
Joined: Mon Feb 03, 2020 7:03 pm

Re: Raspberry Pi 4 bootloader corrupt, can't update EEPROM with recovery.bin

Tue Feb 04, 2020 11:45 am

Moonmarch wrote:
Mon Feb 03, 2020 10:35 pm
Any LED not blinking or staying on does not mean your RPI computer is broken if you are using a laptop that uses LED lights to tell you when CAPS LOCK or NUM LOCK is on, then you replace the keyboard with a third keyboard that does not have any LED lights you can't assume the keyboard is broken no instead you would say the third party keyboard has less features, because you decided to purchase a cheap replacement keyboard for the laptop, if the LED does not turn on when you use CAPS LOCK check to see if CAPS LOCK is on use a text editor Mousepad then write in the text file to see if CAPS LOCK is actually on.

When you start the RPI computer for the first time the boot up can take a long time after the first boot after downloading, and installing all of the updates the RPI computer should boot faster afterwards, if you do not know how to flash the Raspbian OS image to a SD card then you would need to purchase a SD card that contains NOOBS, after booting the RPI computer with the SD card that contains NOOBS if the computer refuses to start then your RPI computer is out the door, you need to replace the computer forget about soldering or using multimeters replace the computer, here is a link to a SD card for sale with NOOBS installed:

64 Gb micro SD For Raspberry Pi 4 / 3B+ / 3A+ /3B With NOOBS v3.2 (Ready To Use)
https://www.ebay.com/i/283491540894?chn ... 5HEALw_wcB
I did test my flashes on a different Pi3. It boots up fine and goes through the usual. There is something very specific with this Pi4. I do not know if anything can or cannot be done but in the distant case it can be, I will have to try because these cost a lot here in India. If taking this or mailing this into some shop will do the job, then that might be a lot cheaper.

Return to “General discussion”