RonR
Posts: 1683
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Serious Problem in Bootloader EEPROM

Sun Aug 30, 2020 3:19 am

There's a nasty problem in the bootloader EEPROM configuration logic that can cause the bootloader EEPROM to be reprogrammed over and over endlessly.

The following steps will reproduce the problem:

1. Disable the rpi-eeprom-update.service (I run this way to prevent automatic EEPROM updates):

systemctl disable rpi-eeprom-update.service

2. With no USB storage devices attached, boot into an SD card.

3. Change BOOT_ORDER to 0x1 in the bootloader configuration file.

4. Reboot to apply the change.

5. Observe that in /boot, recovery.bin has been renamed to recovery.000, but pieeprom.sig and pieeprom.upd are still present.

6. Attach a USB storage device containing RaspiOS.

7. Change BOOT_ORDER to 0xf14 in the bootloader configuration file (I run with this value to give USB storage devices priority).

8. Reboot to apply the change.

9. Observe that the USB storage device boots normally and the SD card's /boot directory still contains the files observd in step 5 above.

10. Change BOOT_ORDER to 0x1 in the bootloader configuration file.

11. Reboot to apply the change.

12. Observe in a serial port display that the bootloader EEPROM is being endlessly reprogrammed, alternately from the SD card and the USB storage device, without ever booting RaspiOS.

I've attached a serial port capture (bootloader.log) of the procedure outlined above. I removed power after the EEPROM had been reprogrammed 4 times, but it will continue endlessly.

I discovered this earlier today after my Raspberry Pi 4 had unknowingly been in one of these loops for about 30 minutes. It could have continued for hours or days if I hadn't needed to use the Pi.
Attachments
bootlloader.log.zip
(1.81 KiB) Downloaded 19 times

Kendek
Posts: 265
Joined: Thu Jul 25, 2019 4:39 pm
Location: Kaposvár, Hungary

Re: Serious Problem in Bootloader EEPROM

Sun Aug 30, 2020 9:29 am

Yeah, with ENABLE_SELF_UPDATE=1, the bootloader will updates itself when the pieeprom.upd (and .sig) is present and differ from the contents of the EEPROM. You have now caused a logical cyclicality with the ever-changing boot order.

hippy
Posts: 8531
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Serious Problem in Bootloader EEPROM

Sun Aug 30, 2020 1:12 pm

I thought this issue had been previously identified, though I have no idea if it was supposedly fixed or not, or if it can be.

I certainly experienced and reported similar issues when an SD Card and USB device were causing an updated Boot Eeprom to be unexpectedly reverting on reboot - though I didn't end up in an endless boot or Boot Eeprom update cycle.
hippy wrote: So I guess the issue is that Boot Eeprom updates happen before the boot media is determined so it can update from USB before booting SD and update from SD before booting from USB. And then one ends up in a circle of hell when what's updated doesn't match what's just booted.

I guess that if everything matches on everything one is booting, everything is set exactly the same, then everything is fine but when it isn't, that's when things break.

RonR
Posts: 1683
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Serious Problem in Bootloader EEPROM

Sun Aug 30, 2020 9:11 pm

The crux of the problem is a fragile bootloader EEPROM update / bootloader EEPROM configuration process.

Delaying the actual bootloader EEPROM update until the next reboot and relying on a service to run after that reboot to delete the files that initiated the update opens the door to unexpected results. If BOOT_ORDER is changed, the update files can be left behind, only to be unexpectedly executed the next time that device is booted, possibly resulting in bootloader configuration values also being changed (embedding a user configuration file in a firmware file was not a great idea). In addition, the service that runs on every boot also forces automatic bootloader EEPROM updates to the latest version to occur that can't be regressed out of easily. You can regress to an older bootloader EEPROM version, but an unexpected update to the latest version will occur on the next boot. Simply inserting/removing an SD card or powering a USB device off/on during a bootloader EEPROM or bootloader configuration file change (or with one left behind as previously described) can result in totally unexpected results.

With the current scheme, in addition to verifying that a bootloader update/configuration change occurred as desired, it's very important to inspect the /boot directory of all devices to ensure that no surprises have been left behind. This is obviously asking a lot of the most users, but failure to do so almost certainly accounts for some of the 'mysterious' bootloader EEPROM update scenarios that have been encountered. Limiting the system to a single storage device (an SD card but no USB storage devices or no SD card and one USB storage device) during the entire bootloader EEPROM update and verification process will greatly reduce the possibility of unexpected results.

cleverca22
Posts: 1878
Joined: Sat Aug 18, 2012 2:33 pm

Re: Serious Problem in Bootloader EEPROM

Mon Aug 31, 2020 5:05 am

i think the simplest fix is to not just rename recovery.bin but also to rename the pieeprom.upd file, even with the self-update mechanism

then it cant loop because the eeprom cant find an update on the card/usb

the only issue, is that network boot cant rename files, but if you get into an sd/net cycle, the sd will rename, and then netboot wins the fight and stops reflashing

making that change take effect, will the foundation to change the eeprom firmware, but once done, it would entirely solve this issue

timg236
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 690
Joined: Thu Jun 21, 2018 4:30 pm

Re: Serious Problem in Bootloader EEPROM

Mon Aug 31, 2020 8:31 am

The mistake was "1. Disable the rpi-eeprom-update.service (I run this way to prevent automatic EEPROM updates):"
This would have deleted the update files when the system booted.

There's no need to rename the .upd files and you can't do that via TFTP anyway. If we change anything it would be to disable self-update from SD boot.

RonR
Posts: 1683
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Serious Problem in Bootloader EEPROM

Mon Aug 31, 2020 4:59 pm

timg236 wrote:
Mon Aug 31, 2020 8:31 am
The mistake was "1. Disable the rpi-eeprom-update.service (I run this way to prevent automatic EEPROM updates):"
This would have deleted the update files when the system booted.

Not so. Don't perform that step and the problem remains unchanged.

It's very easy for the current process to go into an endless reprogramming loop even with the service enabled.

timg236
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 690
Joined: Thu Jun 21, 2018 4:30 pm

Re: Serious Problem in Bootloader EEPROM

Mon Aug 31, 2020 5:23 pm

If you hadn't disabled step (1) then this would not have happened
"5. Observe that in /boot, recovery.bin has been renamed to recovery.000, but pieeprom.sig and pieeprom.upd are still present."

The files would have been deleted so the problem would not have occurred.

So I disagree with your assessment that this is a serious problem, swapping bootable media between re-configuring the bootloader is rather contrived.

The latest bootloader beta disables self-update from the sd-card since recovery.bin can be used instead.

RonR
Posts: 1683
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Serious Problem in Bootloader EEPROM

Mon Aug 31, 2020 5:31 pm

Simple test case on a stock, unmodified system:

1. With an SD card present but while booted from USB, change BOOT_ORDER to 0xf41.

2. Reboot.

3. When the SD card runs, change the BOOT_ORDER to 0xf14.

4. Reboot.

You're in an endless reprogramming loop.

timg236
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 690
Joined: Thu Jun 21, 2018 4:30 pm

Re: Serious Problem in Bootloader EEPROM

Mon Aug 31, 2020 5:52 pm

RonR wrote:
Mon Aug 31, 2020 5:31 pm
Simple test case on a stock, unmodified system:

1. With an SD card present but while booted from USB, change BOOT_ORDER to 0xf41.

2. Reboot.

3. When the SD card runs, change the BOOT_ORDER to 0xf14.

4. Reboot.

You're in an endless reprogramming loop.
Well done you've created a loop - don't do that. Removing either the sd-card or the USB would have gotten you out of that or see the previous post which about SD self-update being removed.
btw: There's not raspi-config option for 0xf14 so presumably you already knew about the boot order

RonR
Posts: 1683
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Serious Problem in Bootloader EEPROM

Mon Aug 31, 2020 6:20 pm

timg236 wrote:
Mon Aug 31, 2020 5:52 pm
don't do that.

What a great solution - Very universal advice for any problems that arise.

W. H. Heydt
Posts: 13604
Joined: Fri Mar 09, 2012 7:36 pm
Location: Vallejo, CA (US)

Re: Serious Problem in Bootloader EEPROM

Mon Aug 31, 2020 6:24 pm

RonR wrote:
Mon Aug 31, 2020 6:20 pm
timg236 wrote:
Mon Aug 31, 2020 5:52 pm
don't do that.

What a great solution - Very universal advice for any problems that arise.
Sometimes...that IS the answer.

RonR
Posts: 1683
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Serious Problem in Bootloader EEPROM

Mon Aug 31, 2020 6:30 pm

W. H. Heydt wrote:
Mon Aug 31, 2020 6:24 pm
RonR wrote:
Mon Aug 31, 2020 6:20 pm
timg236 wrote:
Mon Aug 31, 2020 5:52 pm
don't do that.

What a great solution - Very universal advice for any problems that arise.
Sometimes...that IS the answer.

And sometimes it's turning a blind eye...

W. H. Heydt
Posts: 13604
Joined: Fri Mar 09, 2012 7:36 pm
Location: Vallejo, CA (US)

Re: Serious Problem in Bootloader EEPROM

Mon Aug 31, 2020 8:07 pm

RonR wrote:
Mon Aug 31, 2020 6:30 pm
W. H. Heydt wrote:
Mon Aug 31, 2020 6:24 pm
RonR wrote:
Mon Aug 31, 2020 6:20 pm


What a great solution - Very universal advice for any problems that arise.
Sometimes...that IS the answer.

And sometimes it's turning a blind eye...
The problem with building a better mousetrap is that Nature inevitably builds a better mouse. Likewise, anything that is written to be proof against human folly, will run into a human who can be foolish enough to defeat the protections. At some point, one simply has to stop and say, "Do that at your own risk," and be done with it. Perhaps that's why I prefer C to several other languages. It allows you to do what you want and bear the consequences of your own choices instead of restrictive hand holding.

If you want to create boot loops, you will probably be able to find a way to do so, regardless of what the RPF/RPT devs do to prevent it from happening. I can think of many things that could be done that would be more productive than chasing this particular rabbit down this particular hole, and so--I suspect--can they.

hippy
Posts: 8531
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Serious Problem in Bootloader EEPROM

Mon Aug 31, 2020 8:09 pm

I've said it before but I think Boot Eeprom updating is trying to be too clever for its own good. The whole thing is far too complicated and, as critical feature of the Pi 4B, it needs to be KISS.

cleverca22
Posts: 1878
Joined: Sat Aug 18, 2012 2:33 pm

Re: Serious Problem in Bootloader EEPROM

Mon Aug 31, 2020 8:36 pm

hippy wrote:
Mon Aug 31, 2020 8:09 pm
I've said it before but I think Boot Eeprom updating is trying to be too clever for its own good. The whole thing is far too complicated and, as critical feature of the Pi 4B, it needs to be KISS.
the problem though, is that the rom cant load recovery.bin from usb/net, so you need self-updating to get any updating at all on usb boot
timg236 wrote:
Mon Aug 31, 2020 5:23 pm
The latest bootloader beta disables self-update from the sd-card since recovery.bin can be used instead.
and that sounds perfect, no point in self-update from SD, since recovery.bin can do the job for you
and then the renaming of recovery.bin neuters the SD card, and it cant trigger any more updates

the only way a user can break it then, is for usb and netboot to both be active, and both have a pieeprom.upd that says to boot the other, but you would have to really go out of your way to do that

RonR
Posts: 1683
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Serious Problem in Bootloader EEPROM

Mon Aug 31, 2020 8:40 pm

W. H. Heydt wrote:
Mon Aug 31, 2020 8:07 pm
The problem with building a better mousetrap is that Nature inevitably builds a better mouse. Likewise, anything that is written to be proof against human folly, will run into a human who can be foolish enough to defeat the protections. At some point, one simply has to stop and say, "Do that at your own risk," and be done with it. ...

If you want to create boot loops, you will probably be able to find a way to do so, regardless of what the RPF/RPT devs do to prevent it from happening. I can think of many things that could be done that would be more productive than chasing this particular rabbit down this particular hole, and so--I suspect--can they.

I didn't want or try to create a boot loop. I simply changed the BOOT_ORDER to match what some Raspberry Pi's have been shipped with to run a test while helping another user (Pi 4 8GB Won't boot from USB). When I switched BOOT_ORDER back to its original value and rebooted, I didn't discover until 30 minutes later that my Raspberry Pi was in an endless reprogramming loop. Only after this problem occurred without warning did I research it further. As I outlined earlier, there are very reasonable scenarios that will put the bootloader into this death spiral.

If the only allowable value for BOOT_ORDER is the one that raspi-config sets it to, then there should be huge warnings to that effect. If one can't have an SD card and a USB storage device online simultaneously when doing bootloader updates, then there should be huge warnings to that effect.

One of the attributes of well written software is that it doesn't self-destruct if a user does something seemingly reasonable.

RonR
Posts: 1683
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Serious Problem in Bootloader EEPROM

Mon Aug 31, 2020 8:55 pm

cleverca22 wrote:
Mon Aug 31, 2020 8:36 pm
timg236 wrote:
Mon Aug 31, 2020 5:23 pm
The latest bootloader beta disables self-update from the sd-card since recovery.bin can be used instead.
and that sounds perfect, no point in self-update from SD, since recovery.bin can do the job for you
and then the renaming of recovery.bin neuters the SD card, and it cant trigger any more updates

That may stop the endless reprogramming loop, but it doesn't prevent the bootloader EEPROM firmware or the bootloader configuration file from being reverted unexpectedly.

RonR
Posts: 1683
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: Serious Problem in Bootloader EEPROM

Tue Sep 01, 2020 2:53 am

Raspberry Pi EEPROM Manager has been enhanced to circumvent the issue described in this topic by ensuring that no previous EEPROM updates are pending.

While I would rather see a more robust core EEPROM update process, I believe the use of Raspberry Pi EEPROM Manager eliminates the possiblilty of an endless EEPROM reprogramming loop.

ganzgustav22
Posts: 115
Joined: Tue Feb 11, 2020 1:04 pm

Re: Serious Problem in Bootloader EEPROM

Tue Sep 01, 2020 11:54 am

While looking at the rpi-eeprom-update script for other reasons I just noticed something that really seems serious to me:

The checksum is generated on-the-fly by the script. Which means, should the firmware file be corrupted, the checksum generated will still match and the corrupted file will be flashed. Shouldn't these checksums be generated by the same guys that build the firmware files and then be delivered together with the firmware files?

hippy
Posts: 8531
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Serious Problem in Bootloader EEPROM

Tue Sep 01, 2020 12:11 pm

ganzgustav22 wrote:
Tue Sep 01, 2020 11:54 am
The checksum is generated on-the-fly by the script. Which means, should the firmware file be corrupted, the checksum generated will still match and the corrupted file will be flashed.
AIUI the checksum is merely to verify that a pieeprom.bin is a Boot Eeprom image and not just some random file placed on the card with that name, not a means to guarantee correctness of the image.

The checksum has to be created from the script because it changes as the Boot Eeprom configuration is altered.

It would be possible for the Boot Eeprom image to include a checksum for each file contained within it, and for the file list itself.

I imagine corrupting Boot Eeprom isn't considered much of a problem as it is always possible to flash a Boot Eeprom recovery image from SD Card.

timg236
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 690
Joined: Thu Jun 21, 2018 4:30 pm

Re: Serious Problem in Bootloader EEPROM

Tue Sep 01, 2020 12:48 pm

hippy wrote:
Tue Sep 01, 2020 12:11 pm
ganzgustav22 wrote:
Tue Sep 01, 2020 11:54 am
The checksum is generated on-the-fly by the script. Which means, should the firmware file be corrupted, the checksum generated will still match and the corrupted file will be flashed.
AIUI the checksum is merely to verify that a pieeprom.bin is a Boot Eeprom image and not just some random file placed on the card with that name, not a means to guarantee correctness of the image.

The checksum has to be created from the script because it changes as the Boot Eeprom configuration is altered.

It would be possible for the Boot Eeprom image to include a checksum for each file contained within it, and for the file list itself.

I imagine corrupting Boot Eeprom isn't considered much of a problem as it is always possible to flash a Boot Eeprom recovery image from SD Card.
Correct. The .sig file guards against FAT corruption or the randomly renaming something of exactly the right size etc.

The rpi-eeprom-update script also verifies the 'source' .bin files against the APT manifest so you have to try *really hard* in order to flash a corrupted image.

If your rootfs is randomly corrupting files then you have bigger problems.

ganzgustav22
Posts: 115
Joined: Tue Feb 11, 2020 1:04 pm

Re: Serious Problem in Bootloader EEPROM

Wed Sep 02, 2020 11:47 am

The checksum has to be created from the script because it changes as the Boot Eeprom configuration is altered.
But that applies only to the bootloader firmware, the VL805 firmware doesn't have any configurable options, or?

The .sig file guards against FAT corruption or the randomly renaming something of exactly the right size etc.
I'd say, if you manage to unknowingly and randomly rename something of the exactly the right size, you have bigger problems (like ghosts in your computer, or split personality disorder, dementia, etc. ;)), a corrupted file on the root file systems seems more likely to me, especially on an sdcard.

But seriously, is it correct, that no matter what garbage might be flashed as bootloader firmware or VL805 firmware, it's always recoverable, i.e. there is no way to permanently brick the Pi4 by an update or re-flash gone wrong?

hippy
Posts: 8531
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Serious Problem in Bootloader EEPROM

Wed Sep 02, 2020 12:01 pm

ganzgustav22 wrote:
Wed Sep 02, 2020 11:47 am
But seriously, is it correct, that no matter what garbage might be flashed as bootloader firmware or VL805 firmware, it's always recoverable, i.e. there is no way to permanently brick the Pi4 by an update or re-flash gone wrong?
The internal Boot ROM inside the chip cannot be changed and it will always check the SD Card and run the Boot Eeprom Recovery app if present.

So, as long as you have a readable SD Card with a non-corrupted Boot Eeprom Recovery app and image, the Boot Eeprom will be reflashed to this non-corrupted image.

Only hardware damage or failure should prevent that from working.

Return to “General discussion”