[SOLVED] custom recovery.bin files

[cleverca22]

from the documentation that is available, recovery.bin basically has the same job and format as bootcode.bin

if present, the boot rom will just load it into ram and execute it

the rpi4 then adds a second source, the SPI flash, which it will load&execute if recovery.bin cant be found

based on the above, i have used a vc4 gcc toolchain to compile a hello-world example, patched it to deal with the new pullup config, and the faster base clock (54mhz vs 19.2mhz), and renamed it to recovery.bin

but, the boot rom entirely ignores it!!

https://github.com/raspberrypi/rpi-eepr ... -568576065

this comment says there is extra validation to verify if the recovery.bin file is valid

the question then, is what kind of validation is done? and how can i make my own recovery.bin "valid" ?

[jamesh]

I don't believe that will be made public, I suspect it would create a massive security hole.

I may be wrong, but would need the engineer who deals with it to confirm.

[cleverca22]

even if the recovery.bin was signed, does the boot rom also check the signatures of the blobs within the SPI flash?

and https://github.com/ptesarik/vc4boot implies that there is no checks at all on the final start4.elf that gets loaded

my theories on secure-boot with the pi, is that a public key would have to first be burned into the OTP, and only if set, would the boot rom verify the next stage (recovery.bin or the spi flash)

and i'm guessing that all publicly sold pi's just have the key unset and the whole signing procedure undocumented?

[dividuum]

I don't believe that will be made public, I suspect it would create a massive security hole.

I'm curious why that's so. If you control any part the boot code, from the perspective of anything that follows, all is lost anyway. What does controlling that step of the boot process add?

[trejan]

my theories on secure-boot with the pi, is that a public key would have to first be burned into the OTP, and only if set, would the boot rom verify the next stage (recovery.bin or the spi flash)

and i'm guessing that all publicly sold pi's just have the key unset and the whole signing procedure undocumented?

Secure boot isn't available on the publicly availably Pi boards AFAIK. I guess you might be able to get the necessary documentation and tools if you're paying large sums for the customisation service but it'll all be under NDA.

The Pi firmware including the latest Pi 4 still mentions 4 boot signing key so it wasn't completely removed. I'm not sure if the OTP keys are actually the keys themselves though as OTP memory is limited. I suspect it might be similar to other implementations like NXP which stores a hash of the key signing keys and then looks for a table containing the key signing key and then the actual code signing key.

The other widely available device with the BCM2835 apart from the Raspberry Pi was the first generation Roku boxes. They had a per device key to verify the bootloader that then verified the kernel etc... The chain of trust was broken on those boxes but it was due to some flaws by Roku in their implementation and not the secure boot process itself.

[cleverca22]

another thing i notice, that might be of use to me

https://github.com/hermanhermitage/vide ... umpbootenv

Code: Select all

r26: 60003602 ; lr - link register

this implies that the entrypoint for bootcode.bin (and probably recovery.bin) is loaded with bl, rather then b (or it is loaded with b, and thats one call-stack up, tail-call optimization kind of thing)

so, you could just toss a conditional return in there, if certain conditions are met

what would the boot rom do, if the recovery.bin was to just return control back to the link register? would it consider the boot failed and go on to another mode, like usb boot?

are there other trivial ways for the spi eeprom to force a fallback into usb device mode, without bricking itself and doing a hard reset?

[cleverca22]

@jamesh are you able to reveal if its signed, or if it just has some kind of header or self-hash somewhere?

any eta on being able to reveal the full details, if at all?

[LdB]

As stated above I doubt it would be made public it is same problem with old AMI Bios upgrades if anyone ever targeted with a virus it is problematic to get rid of it

[cleverca22]

@LdB if somebody wanted to put malware onto the eeprom, they dont even need a custom recovery.bin

the existing recovery.bin file will happy burn anything you give it to the eeprom, provided you update the .sig file (which is only a sha256)

it is also trivial to reflash the eeprom from root using flashrom

so the recovery.bin validation itself is giving you zero security, and it must exist on the SD card, so it cant persist in a dark corner you cant see

but if the eeprom bootcode.bin has similar validation, that would cause problems for my plans, and would allow for the things you mention

[cleverca22]

@jamesh any reply from the engineer yet?

some of the things i'm wanting to look into, is writing my own firmware to handle booting from usb mass-storage
and if its possible to put `start4.elf` onto the SPI chip (assuming it fits obviously)

i have figured out the filesystem being used within the SPI chip, so adding more files to it would be easy enough

[jamesh]

@jamesh any reply from the engineer yet?

some of the things i'm wanting to look into, is writing my own firmware to handle booting from usb mass-storage
and if its possible to put `start4.elf` onto the SPI chip (assuming it fits obviously)

i have figured out the filesystem being used within the SPI chip, so adding more files to it would be easy enough

We've had a chat. What it comes down to is that the first customisable part of the boot sequence is start*.elf, so custom recovery.bin's are not and won't be supported. We also will not be supporting custom writes to the SPI EEPROM, although you are welcome to use it if you want, but the next update will overwrite it, there is no guarantee the EEPROM will stay the same size, and there is no guarantee it will continue to be used in future models. The main problem with using the EEPROM is that is pretty much full already.

This might be useful

https://github.com/ptesarik/vc4boot

[cleverca22]

i have been able to confirm that https://github.com/ptesarik/vc4boot works on my rpi4

i know that the spi chip may change size, and will be overwritten when applying updates

does the current spi blob ever look for `start4.elf` on the spi, or only ever on the sd and ethernet? (based on `bootconf.txt`)

just trying to see if i could configure it to run some basic custom program, without any sd card or network support

[hippy]

i have been able to confirm that https://github.com/ptesarik/vc4boot works on my rpi4

Excellent news. Any documentation or detail on how 'recovery.bin' can be used as 'bootcode.bin' used to be, how the Boot Eeprom is structured, will be useful for people who don't want to have to figure it all out for themselves, don't want to be limited to pre-4B Pi's.

The risk of recovery.bin or Boot Eeprom being updated isn't really a problem, not even a likelihood, for people only using VC4 Bare Metal, and the Boot Eeprom getting smaller probably won't be either.

[cleverca22]

for the spi eeprom image:
https://gist.github.com/cleverca22/2a7e ... 2c0aaa152a

each "file" has a 32bit magic number (identifies the type or contents)
then a 32bit length field
then $length bytes of body, with padding to make the next entry be aligned to 8 bytes

55aaf00f is the magic for what i'm calling bootcode.bin, it has the same 0x200 hole at the start, and looks strikingly similar to bootcode.bin when decompiled

55aafeef is the magic for "unused" space, to prevent un-needed wear on the flash, certain files are kept at known offsets (aligned to the erase block size of 4096 bytes), this magic is used to pad things out, to keep the diff minimal, recovery.bin checks for differences and only overwrites the blocks that have changed

55aaf22f is the magic for internal files, the body has a 16 byte filename prepended to it

55aaf11f is the magic for user accessible files, it also has a 16byte filename prepended to it, and the only known one is bootconf.txt

@jamesh, are you able to confirm if 55aaf00f has the same validation as recovery.bin, and we can never run custom code from that "file" ?


@jamesh: seperately from that answer, which PLL is the DDR4 controller running from? i want to start configuring them in my start4.elf, but need to know which one to not touch

[hippy]

Many thanks for the details. That moved my work on and I can now dump and build my own Boot Eeprom Image files.

It's pretty clear why RPT are looking to reduce the physical Eeprom size to save BOM costs. The last 256KB is completely unprogrammed so the Eeprom could easily be halved in size to 256KB.

It would probably be more difficult to reduce to 128KB or 64KB but, even if it were 64KB, it still looks to be suitable for ersatz 'bootcode.bin' use. Many will be far smaller than 64KB so it seems to confirm any reduction in size shouldn't affect most Bare Metal VideoCore programmers.

[cleverca22]

the biggest unknown in terms of a custom SPI image, is the ddr4 firmware

Code: Select all

-rw-r--r-- 1 clever users 21K Dec  3 21:24 memsys00.bin
-rw-r--r-- 1 clever users 21K Dec  3 21:24 memsys01.bin
-rw-r--r-- 1 clever users 21K Dec  3 21:24 memsys02.bin
-rw-r--r-- 1 clever users 21K Dec  3 21:24 memsys03.bin
-rw-r--r-- 1 clever users 21K Dec  3 21:24 memsys04.bin
-rw-r--r-- 1 clever users 21K Dec  3 21:24 memsys05.bin
-rw-r--r-- 1 clever users 21K Dec  3 21:24 memsys06.bin
-rw-r--r-- 1 clever users 21K Dec  3 21:24 memsys07.bin

i believe these files are firmware for the ddr4 controller

my current guess, is that the bootcode.bin will just treat that file as an uint32_t[], and then just copy each 32bit chunk to a magic register

but the VPU and the ddr4 controller have different byte-orders, so each 4byte chunk gets re-ordered during that process, which un-scrambles all of the strings in those files

something ive wanted to experiment with, but havent gotten to yet, what happens if i create an SPI image with only 1 of those 8 files?

why do 8 exist?

could some multi-file compression algo merge them together and save more space?

[hippy]

i believe these files are firmware for the ddr4 controller
why do 8 exist?

I haven't really looked at them other than to reverse byte order and look at the strings. My hunch from that, looking at how the strings differ, is that each sets up a different characteristic of the memory so they likely all have to be executed in order.

One thing I did note is that their sizes are all 20,512 bytes, which is 20KB+32 so I guess there's some header and/or configuration with 20KB of code which I suspect gets paged-in/overlaid and executed as needed. Could that 20KB be 16KB code + 4KB data/config ?

The code could be VC4, at least in part, because those strings and any associated data presumably gets formatted and sent out to the UART when debugging. I would have expected any hardware block to simply return status values in registers but it's just another hunch.

[cleverca22]

2019-12-13 14:46:55 < felix_> well, vc4 is a dual core arcompact and some vector or simd units
2019-12-13 14:38:34 < felix_> clever: my guess for the isa of those blobs would be arcompact. broadcom uses quite a lot of synopsys stuff and iirc at least one of their ddr4 memory controllers has an arc core for memory training. at least i think i read soething like that a few days ago
2019-12-13 14:42:01 < felix_> clever: https://www.synopsys.com/dw/ipdir.php?ds=dwc_ddr43_PHY via https://www.crowdsupply.com/mnt/reform/ ... ing-reform
2019-12-13 15:01:01 < felix_> the memsys0x.bin files don't look like arcompact code; the hello.bin does though

there is a ddr4 controller in another machine, that has very similar behavior

attempting to decompile memsys00.bin as vc4 results in total garbage output, so its not that

and i think the byte order is being swapped around, because the vc4 is little-endian, but the ddr4 controller is big-endian

my theory, is that there is basically an on-chip uart (or maybe just a fifo) between the ddr4 and vc4 cores, and the firmware should read it to see those error messages, if any occur, though such errors may corrupt the firmware that is responsible for reading it

felix_ also mentioned, that the vc4 opcode set, is very similar to arcompact, the hello.bin i gave him was from vc4-elf-gcc

[hippy]

My hunch on 16KB+4KB might be promising. Differences between each file and another file seems always roughly 16KB.

But it might be 2KB+2KB rather than 4KB. The first 2KB is roughly the same in all, and the first long in the image is 0x0000902 which is about 2K. That seems to be a bit-pattern array which is perhaps what would be expected for 'shmoo' graphing.

I expect figuring it out is going to be a case of collecting breadcrumbs until we have enough to make a cake, finding where else DPFE is used and who's had their fingers in the mix, Googling and hoping to get lucky. This looks like it could be one good starting point -

https://github.com/torvalds/linux/blob/ ... stb_dpfe.c

I'm more interested in running purely 'bootcode.bin' code, don't care if memory is initialised or not, and won't care how if using 'start4.elf', so I don't plan to really investigate this aspect further. But if curiosity does turn anything up I'll let you know.

On making things smaller; just zipping the memsys??.bin files gives about 64KB so it's feasible they could be pushed into a 128KB Eeprom. But that limits the ability to add more stuff which may be needed to overlay 'bootcode.bin' for boot options etc. I would guess RPT are looking to drop the Eeprom down to 256KB, which would not necessitate compressing. KISS is probably the current path they're on.

[cleverca22]

after studying the rpi4 mask rom for an entire day, and working on the hint that its an HMAC, i have confirmed that it is just a plain HMAC-SHA1, with the 20 byte signature appended to the end of the .bin file

getting a recovery.bin that actually functions will be a separate ordeal, but this is a major stepping stone towards one of my original goals of usb device boot and making it into a serprog device that flashrom could talk to

[hippy]

Just noticed the latest 'pieeprom-2020-03-04.bin' file format has changed in the beta release channel.

There's a new 55AAF33F header, 'bootcode.bin' is significantly larger, much larger than 64KB, 'memsysXX.bin' files now vary in size, VL805 files included along with others, 'bootconf.txt' alignment moves accordingly.

Older 2020-01-17

Code: Select all

00000 : 55AAF00F  61796   Bootcode File   'bootcode.bin'  61796
0F170 : 55AAFEEF  3720    Unused
10000 : 55AAF22F  20528   Internal File   'memsys00.bin'  20512
15038 : 55AAF22F  20528   Internal File   'memsys01.bin'  20512
1A070 : 55AAF22F  20528   Internal File   'memsys02.bin'  20512
1F0A8 : 55AAF22F  20528   Internal File   'memsys03.bin'  20512
240E0 : 55AAF22F  20528   Internal File   'memsys04.bin'  20512
29118 : 55AAF22F  20528   Internal File   'memsys05.bin'  20512
2E150 : 55AAF22F  20528   Internal File   'memsys06.bin'  20512
33188 : 55AAF22F  20528   Internal File   'memsys07.bin'  20512
381C0 : 55AAF22F  3852    Internal File   'mcb.bin'       3836
390D8 : 55AAFEEF  3872    Unused
3A000 : 55AAF11F  242     User File       'bootconf.txt'  226
3A100 : FFFFFFFF  286464  Unprogrammed

Latest 2020-03-04

Code: Select all

00000 : 55AAF00F  88960   Bootcode File   'bootcode.bin'  88960
15B88 : 55AAFEEF  42096   Unused
20000 : 55AAF33F  13894   Internal File   'memsys00.bin'  13878
23650 : 55AAF33F  12180   Internal File   'memsys01.bin'  12164
265F0 : 55AAF33F  14083   Internal File   'memsys02.bin'  14067
29D00 : 55AAF33F  14185   Internal File   'memsys03.bin'  14169
2D478 : 55AAF33F  14040   Internal File   'memsys04.bin'  14024
30B58 : 55AAF33F  13457   Internal File   'memsys05.bin'  13441
33FF8 : 55AAF33F  13372   Internal File   'memsys06.bin'  13356
37440 : 55AAF33F  13217   Internal File   'memsys07.bin'  13201
3A7F0 : 55AAF33F  3131    Internal File   'mcb.bin'       3115
3B438 : 55AAF33F  7028    Internal File   'vl805hub.bin'  7012
3CFB8 : 55AAF33F  60228   Internal File   'vl805mcu.bin'  60212
4BB08 : 55AAF33F  7645    Internal File   'logo.bin'      7629
4D8F0 : 55AAF33F  8007    Internal File   'font.bin'      7991
4F840 : 55AAF33F  1744    Internal File   'qrcode.bin'    1728
4FF18 : 55AAFEEF  224     Unused
50000 : 55AAF11F  242     User File       'bootconf.txt'  226
50100 : FFFFFFFF  196352  Unprogrammed

[cleverca22]

i believe the main rule for `bootcode.bin` is that it be under 128kb (the L2 cache size) and that it be properly signed

the large jump in size might be new features being added, possibly allowing the firmware to boot from other sources

the differences in memsys, could be compression or patching, ideas ive had on how to reclaim space, but ive not had time to look into the DDR4 stuff yet


a quick glance with `strings` says that the SPI firmware, now has the capability to re-flash the VL805, like `recovery.bin` did
that would imply that its able to re-flash things, with less files on the SD card, though its not clear yet how its told to do so