okenido
Posts: 73
Joined: Thu Aug 02, 2018 11:47 am

Fail-safe firmware upgrades

Fri Jun 05, 2020 12:49 pm

Hi
My RPI3B+ has a custom made firmware (with bare metal code), and the firmware can be upgraded via USB.

When the upgrade happens, the kernel8.img file is copied from USB to the SD card. If a power loss occurs at this moment, the file can be corrupted and the device bricked.

A common method to make fail-safe upgrades is to have two firmware files, if file A is used to boot then new firmware is copied to file B, its checksum is verified and if everything is correct a switch is toggled in a config file, which tells to use firmware B for the next boot. Then the process repeats for next upgrades.

Is there a way to do that using the RPI ? It seems it will only boot from kernel8.img

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 26875
Joined: Sat Jul 30, 2011 7:41 pm

Re: Fail-safe firmware upgrades

Fri Jun 05, 2020 1:11 pm

You cannot brick a Raspberry Pi.

In this circumstance, it just requires the SD card to be reimaged.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed.
I've been saying "Mucho" to my Spanish friend a lot more lately. It means a lot to him.

okenido
Posts: 73
Joined: Thu Aug 02, 2018 11:47 am

Re: Fail-safe firmware upgrades

Fri Jun 05, 2020 1:16 pm

I wasn't very clear sorry, the RPI isn't bricked but the device embedding it is (unless you tell the user to open his device which isn't very practical)


okenido
Posts: 73
Joined: Thu Aug 02, 2018 11:47 am

Re: Fail-safe firmware upgrades

Fri Jun 05, 2020 1:22 pm

It seems start_file in config.txt would simply do what I need, i'll try that thanks

DarkElvenAngel
Posts: 941
Joined: Tue Mar 20, 2018 9:53 pm

Re: Fail-safe firmware upgrades

Fri Jun 05, 2020 2:06 pm

okenido wrote:
Fri Jun 05, 2020 1:22 pm
It seems start_file in config.txt would simply do what I need, i'll try that thanks
I was thinking more about the kernel configuration so you could use a different kernel file.

I do this with my embedded projects the update path is done with a reboot and a GPIO switch the pi is then set to boot with a different kernel and an initramfs that has the updater and rescue if things go wrong.

okenido
Posts: 73
Joined: Thu Aug 02, 2018 11:47 am

Re: Fail-safe firmware upgrades

Fri Jun 05, 2020 2:09 pm

Interesting I'll check that too - I was simply thinking about swapping between the two firmware files, then updating config.txt programatically after the write is completed, so any power loss simply leave the system working on the already existing firmware

okenido
Posts: 73
Joined: Thu Aug 02, 2018 11:47 am

Re: Fail-safe firmware upgrades

Fri Jun 05, 2020 2:28 pm

* it's kernel parameter, not start_file

DarkElvenAngel
Posts: 941
Joined: Tue Mar 20, 2018 9:53 pm

Re: Fail-safe firmware upgrades

Fri Jun 05, 2020 4:26 pm

okenido wrote:
Fri Jun 05, 2020 2:09 pm
Interesting I'll check that too - I was simply thinking about swapping between the two firmware files, then updating config.txt programatically after the write is completed, so any power loss simply leave the system working on the already existing firmware
I had a problem with that just a few days ago the config.txt file was corrupted the EOF marker was missing so the pi wouldn't boot at all I had to put the SD card in another machine and rewrite the config.txt file but first I had to fix the boot partition so I could delete it. Was a real mess.

okenido
Posts: 73
Joined: Thu Aug 02, 2018 11:47 am

Re: Fail-safe firmware upgrades

Fri Jun 05, 2020 4:48 pm

I'm only changing one byte from the file, leaving its size untouched so it should be fine to prevent corruptions (eg. kernel8.img => kernel9.img)

However I'm not sure how to deal with the problem regarding the firmware file itself. Maybe I can simply add lots of zero bytes at the end, providing enough margin so futures upgrades do not change the file size too ? Not sure if the file will still be valid, need to try...

DarkElvenAngel
Posts: 941
Joined: Tue Mar 20, 2018 9:53 pm

Re: Fail-safe firmware upgrades

Fri Jun 05, 2020 5:04 pm

okenido wrote:
Fri Jun 05, 2020 4:48 pm
I'm only changing one byte from the file, leaving its size untouched so it should be fine to prevent corruptions (eg. kernel8.img => kernel9.img)

However I'm not sure how to deal with the problem regarding the firmware file itself. Maybe I can simply add lots of zero bytes at the end, providing enough margin so futures upgrades do not change the file size too ? Not sure if the file will still be valid, need to try...
I don't know if the corruption was caused by it changing size maybe because it was left open when when the kernel received a reset command. All I know is that it wasn't caused by a power issue.

User avatar
dickon
Posts: 1657
Joined: Sun Dec 09, 2012 3:54 pm
Location: Home, just outside Reading

Re: Fail-safe firmware upgrades

Fri Jun 05, 2020 5:51 pm

okenido wrote:
Fri Jun 05, 2020 4:48 pm
I'm only changing one byte from the file, leaving its size untouched so it should be fine to prevent corruptions (eg. kernel8.img => kernel9.img)
That isn't how it works, I'm afraid. Your uSD card has a small CPU on it -- probably an ARM -- to do wear-levelling, so it can remap 'hot' blocks to ones which haven't been touched in some time, to make the whole device statistically more reliable over time.

Copy it to a different file, call 'sync' to ensure it's on the card, then move it over the top of the old one. mv is atomic -- as much as anything can be on an SD card -- so that should be about as safe as you can get.

okenido
Posts: 73
Joined: Thu Aug 02, 2018 11:47 am

Re: Fail-safe firmware upgrades

Fri Jun 05, 2020 11:52 pm

I fear there is no 100% fail-safe software method against the SD card doing its wear leveling.

Anyway for the files going with the firmware I replaced the direct overwrite with a temporary copy + move, it's already way better as the critical time window where things can go wrong is considerably reduced (at least avoiding a blank file - risk of corrupting FAT and wear leveling is still there). This + the "double-buffered" firmware, it should be good enough !

User avatar
dickon
Posts: 1657
Joined: Sun Dec 09, 2012 3:54 pm
Location: Home, just outside Reading

Re: Fail-safe firmware upgrades

Fri Jun 05, 2020 11:57 pm

Do the sync before the move. It's belt-and-braces, but you want that in a situation like this.

okenido
Posts: 73
Joined: Thu Aug 02, 2018 11:47 am

Re: Fail-safe firmware upgrades

Sat Jun 06, 2020 12:13 am

Do you use FatFS like me ? It seems f_close() does the sync so it's not needed to call it manually

User avatar
dickon
Posts: 1657
Joined: Sun Dec 09, 2012 3:54 pm
Location: Home, just outside Reading

Re: Fail-safe firmware upgrades

Sat Jun 06, 2020 12:19 am

No, I only use SD cards as a last resort.

cleverca22
Posts: 1036
Joined: Sat Aug 18, 2012 2:33 pm

Re: Fail-safe firmware upgrades

Sat Jun 06, 2020 12:21 am

i think the safest route, would be to use something like a journaled FS (such as ext4) for the real kernel and boot config
and then have a bootloader on the fat32 (perhaps just u-boot) which will read its config from the 2nd fs

then properly configured, the journaled FS should undo partial writes after an improper shutdown, and then you just need to write things in the correct order, and the fat32 is never modified, so its not at risk

Return to “General discussion”