Netboot issues with mod 3 B+

[stephanst]

Hi guys,

It seems there is a problem with the default bootcode on the Raspberry Pi 3 B+, specifically related to netboot.
The foundation annouced upgrades for netboot support with this model, specfically the fact that netboot is activated by default on RPI 3B+ :


A per the announcement page on https://www.raspberrypi.org/blog/raspbe ... le-now-35/

1/ "So, without further ado, Raspberry Pi 3 Model B+ is now on sale for $35 (the same price as the existing Raspberry Pi 3 Model B), featuring:
A 1.4GHz 64-bit quad-core ARM Cortex-A53 CPU
Dual-band 802.11ac wireless LAN and Bluetooth 4.2
Faster Ethernet (Gigabit Ethernet over USB 2.0)
Power-over-Ethernet support (with separate PoE HAT)
Improved PXE network and USB mass-storage booting
Improved thermal management"

2/ "Raspberry Pi 3B was our first product to support PXE Ethernet boot. Testing it in the wild shook out a number of compatibility issues with particular switches and traffic environments. Gordon has rolled up fixes for all known issues into the BCM2837B0 boot ROM, and PXE boot is now enabled by default."

With this in mind we ordered the B+ and started testing...

Indeed, on RPi3B+ the PXE boot is enabled by default but the internal bootloader doesn’t enumerate USB devices where is the ethernet and fails to boot from the network when SD card is not present or doesn’t contain OS to boot. The only way to use PXE is to prepare SD card with updated RPi bootloader so it is not possible to use the SD card to boot and fall back to network boot if the SD card is not present or damaged.

Further:
There is nothing on the serial console when no SD card is present.
There is nothing on the serial console when we put a SD card that has just UART empty file to enable.

When we put SD card with UART and relatively old bootcode.bin (probably the same as is build in the RPi 3 B+ eprom) we got this:



When we put newer bootcode.bin as is suggested in this GitHub issue by the guy who developed it (Gordon Hollingworth mentioned above in the product announcement I assume) , we got a normal PXE boot:

https://github.com/raspberrypi/firmware/issues/988







The conclusion is this: the bootcode in the RPi3B+ units that we have is buggy. It was fixed later as we can see on Github, but this is not the version that we have in the units that we received.

Now the big question is: is there a way to fix this? Or just consider that netboot without an SD card is permanently broken on these boards?

[mahjongg]

I have notified all moderators.

[PhilE]

1. All 3B+s ship with the same SoC Boot ROM. There is no EEPROM. The only way to change the first stage loader is with a new spin of the chip, which would then be 2837C0.

2. A consequence of 1 is that if Gordon has fixed something so that booting works then that fix is available for all 3+s, not devices with a new ROM (because there isn't one).

3. A bug has been fixed in the firmware that was preventing a reboot from working. The fix hasn't reached the Raspbian firmware package yet but can be downloaded using "rpi-update".

4. The boot ROM does not access the UART - that feature is provided by bootcode.bin. If you are expecting to get some UART debugging output with no sdcard then the TFTP server must have a relatively up-to-date bootcode.bin that has been patched to enable the UART - see https://github.com/raspberrypi/linux/is ... -400649549.

If the above hasn't resolved your problem, please explain what you mean by the phrase "the internal bootloader doesn’t enumerate USB devices where is the ethernet", and post some logs from your dnsmasq/tftp server showing the (non-)boot sequence.

[ShiftPlusOne]

3. A bug has been fixed in the firmware that was preventing a reboot from working. The fix hasn't reached the Raspbian firmware package yet but can be downloaded using "rpi-update".

It should be in apt, I snuck it in last week.

[gsh]

Yes, as PhilE says above, there is a difference between 'bootcode' and 'bootrom' the boot sequence is:

1) BootROM executes on the chip, enumerates the USB devices and looks for a device matching the LAN9500 USB to Ethernet controller. It broadcasts a DHCP request and then gets a reply that includes the TFTP server address. Asks the TFTP server for bootcode.bin and then executes it.

2) Bootcode continues the ethernet connection from 1) and reads config.txt start.elf fixup.dat and then executes start.elf

3) start.elf continues the ethernet connection reading config.txt cmdline.txt, the dtb files, the linux kernel and filesystem if necessary. Then it kicks the ARM to start executing the linux kernel.


When you put 'just' the bootcode.bin onto an sdcard then it loads bootcode.bin from the sdcard, initialises the ethernet / DHCP / TFTP and continues from 2) above. So updates I make to bootcode.bin may fix problems at that initialisation stage but it cannot be 'programmed' into the Pi.

So if you cannot boot a Pi 3B+ without an SD card, it's nothing to do with the programming of the Pi 3B+ but more likely to be a problem with the network and server setup that the bootcode.bin is working around.

I would suggest that you run a tcpdump on the server while the Pi 3B+ is first plugged in (ignore the LEDs on the Pi 3B+, they are not enabled by the bootrom or bootcode)

[palavrov]

Hi guys,

I'm Sephan's colleague working on our RPi based firmware.

During my research there was nothing into the dnsmasq/tftp logs even for bootcode.bin so my conclusion was that the bootrom doesn't load anything from tftp. Let'm double check that issue and will post here logs, wireshark traces, etc if it still occurs.

Atanas

[gsh]

It's not clear what you mean here...

The bootcode.bin UART output Stephan has captured shows that it is correctly initialised and sending DHCP requests. I would suggest that your TCP dump is not working correctly.

[palavrov]

Oh, yes - Stephan post wrong logs in the first message, this is what I captured on the UART:

There is nothing on the serial console when no SD card is present.
There is nothing on the serial console when we put a SD card that has just UART empty file to enable debug logs.

When we put SD card with UART and relatively old bootcode.bin (I think it was from 6th of May - that is the current RPi firmware package in buildroot) we got this:

Raspberry Pi Bootcode

Found SD card, config.txt = 0, start.elf = 0, recovery.elf = 0, timeout = 0
Trying USB
USB Host enumeration failed, no device detected at root

When we put newest bootcode.bin on the SD card, we got a normal PXE boot:

Raspberry Pi Bootcode
Found SD card, config.txt = 0, start.elf = 0, recovery.elf = 0, timeout = 0
Trying USB
Hub device found at addr 4, enumerating HUB
Initialise hub
Found 4 ports, multi_tt = 1
Setting interface 0
Enabling PORT POWER on port 1
Enabling PORT POWER on port 2
Enabling PORT POWER on port 3
Enabling PORT POWER on port 4
Waiting for devices to respond to reset
Found device on port 1
Found highspeed device
Hub device found at addr 5, enumerating HUB
Initialise hub
Found 3 ports, multi_tt = 1
Setting interface 0
Enabling PORT POWER on port 1
Enabling PORT POWER on port 2
Enabling PORT POWER on port 3
Waiting for devices to respond to reset
Found device on port 1
Found highspeed device
Device found: type = Ethernet adapter, addr = 6
Trying booting from Ethernet device addr 6
Initialise ethernet with MAC b8:27:eb:bb:fb:59
Wait for Link up
Failed to initialise link
Trying booting from Ethernet device addr 6
Initialise ethernet with MAC b8:27:eb:bb:fb:59
Wait for Link up
Failed to initialise link
Trying booting from Ethernet device addr 6
Initialise ethernet with MAC b8:27:eb:bb:fb:59
Wait for Link up
Link up
Sending DHCP request
Waiting for dhcp_reply
Done ARP for 200.201.168.192 got 7c:8b:ca:00:a4:2e
Read File: config.txt, 745 (bytes)




Raspberry Pi Bootcode
Read File: config.txt, 745
Read File: start.elf, 3913252 (bytes)
Read File: fixup.dat, 9735 (bytes)
[ 0.000000] Booting Linux on physical CPU 0x0

Because I didn't saw anything on the uart during pxe boot and there was no message in the tftp logs that at least bootcode.bin was served my conclusion was that bootrom doesn't work properly and we need to prepare SD cards with latest bootcode.bin in order to boot. Which means that we can't fall back to the pxe boot during factory installation or firmware upgrade i.e. it is a stop problem.

Based on your replies I want to double check and make sure (with wireshark, tcpdump, etc) that dnsmasq tftp receives and servers the expected file requests (or not). It can be that I didn't pay attention to something and conclude wrong based on the fact that exactly same setup boots perfectly with the previous RPi3 i.e. I was expecting smooth transition after switching to the new RPi3B+

Atanas

[Janice357]

This is pretty much a show-stopper for implementing PiServer with the new RPi 3B+.

Cold boot works fine. Everything is up-to-date in the chroot, firmware, etc.

Tried some slightly older bootcode.bin files from repo with no success they just broke PXE altogether.

Move to a more appropriate section if this is not a firmware issue.

[palavrov]

1) BootROM executes on the chip, enumerates the USB devices and looks for a device matching the LAN9500 USB to Ethernet controller. It broadcasts a DHCP request and then gets a reply that includes the TFTP server address. Asks the TFTP server for bootcode.bin and then executes it.

I read somewhere during my research that problem could be because 10/100/1000 LAN7515 USB ID of the RPi3B+ doesn't match those of 10/100 LAN9544 which requires to load fixed bootloader.bin from sd card as workaround, could you please confirm that the USB ID is handled correctly in the bootrom?

I'll do a new test on our dev setup later today upgrading the bootcode.bin to the latest version on our tftp server - and really hope that this will solve the issue and all that is my miserable fault.

[mfa298]

During my research there was nothing into the dnsmasq/tftp logs even for bootcode.bin so my conclusion was that the bootrom doesn't load anything from tftp. Let'm double check that issue and will post here logs, wireshark traces, etc if it still occurs.

I bought a 3B+ shortly after they were released and netboot was one of the first things I tried, and yes I got it working with no sd card in sight.

First off it might be worth confirming it really is a 3B+ (metal cover over the WiFi and metal over the SoC).

The other potential issues I can think of which might stop you getting the initial dhcp and tftp requests are:

Some PoE splitters only pass two ethernet pairs through to the pi, with the Gigabit interface on the pi if it's plugged into a Gigabit switch that could stop the link coming up. Fixing the link speed might help although could lead to more issues.

Switch ports slow to allow packets; spanning tree, green Ethernet and various other things might mean the port doesn't pass packets for a few seconds, the bootrom will try doing it's dhcp and tftp requests much sooner than most other things try to talk to the network so it's possible the pi sends the requests but the switch is blocking them. Booting from the sd card initially might just put a slightly longer delay on before the requests are made (and may have better retry and backoff parameters)

[palavrov]

I bought a 3B+ shortly after they were released and netboot was one of the first things I tried, and yes I got it working with no sd card in sight.

Thanks!
The fact that it work for you gives me hope that will make it work for us too

Some PoE splitters only pass two ethernet pairs through to the pi, with the Gigabit interface on the pi if it's plugged into a Gigabit switch that could stop the link coming up. Fixing the link speed might help although could lead to more issues.

Indeed, we are using PoE and this could be a problem ... but with bootcode.bin on SD card PXE boots as expected so probably the splitters are OK.

Switch ports slow to allow packets; spanning tree, green Ethernet and various other things might mean the port doesn't pass packets for a few seconds, the bootrom will try doing it's dhcp and tftp requests much sooner than most other things try to talk to the network so it's possible the pi sends the requests but the switch is blocking them. Booting from the sd card initially might just put a slightly longer delay on before the requests are made (and may have better retry and backoff parameters)

That could be. With the old RPi3 we had too much troubles to get reliable pxe boot and drop the SD cards at all ... so the announce that RPi3B+ improved this (i.e. retrying instead of giving up) was giving us a hope to improve as well ... but instead we hit this problem that made me really (m/s)ad.

The boot process debug must be dramatically improved on the RPi to help developers catch whats wrong. Sometimes it can be something silly, sometime can be a bug - in all cases adding more visibility will improve the board. The fact that bootloader is closed source doesn't help too, I'm able to dig myself when hit such a problem but closed source just stops me - I don't want to steal Broadcom company secrets, just to do my job.

[palavrov]

Fixed!

RPi3B+ boots with dnsmasq running on my mac ... so probably I messed up something on the production setup and wrongly conclude that there is problem with the bootrom - my apologies guys.

[jerrm]

Fixed!

RPi3B+ boots with dnsmasq running on my mac ... so probably I messed up something on the production setup and wrongly conclude that there is problem with the bootrom - my apologies guys.

If no SD card is installed, fix the mmc rescan CPU utilization issue - see: viewtopic.php?f=28&t=211912

[palavrov]

...
If no SD card is installed, fix the mmc rescan CPU utilization issue - see: viewtopic.php?f=28&t=211912

That's pretty weird ... fortunately in our setup we fallback to booting from SD cards to avoid endless netboot because we need to boot 100+ boards and sometimes it could take hours to boot all of them.

But because preparing hundreds SD cards with our buildroot based firmware is time consuming we are inserting blank SD cards and then when the board boots for first time the firmware downloads new sd card image from the tftp server and writes it to the sd card so on next boot it will quickly boot from there. Firmware upgrade works the same way - we just download new sd card image and rewrite the sd card.