User avatar
rpdom
Posts: 16097
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: Raspberry Pi4 bootloader - network boot support - BETA

Thu Jan 16, 2020 7:45 am

dickon wrote:
Wed Jan 15, 2020 11:41 pm
The usual idiom is 'echo foo | sudo tee -a $file'.
I believe it's more common to recommend the

Code: Select all

sudo sh -c 'echo FIRMWARE_RELEASE_STATUS="beta" > /etc/default/rpi-eeprom-update'
format now.

I try to avoid sudo whenever I can. I'm old school and use su to get to a root account when needed and exit back to the normal user as soon as I am done. Even worse, when I first started with Unix we just logged in as root when needed :o

cleverca22
Posts: 280
Joined: Sat Aug 18, 2012 2:33 pm

Re: Raspberry Pi4 bootloader - network boot support - BETA

Wed Jan 22, 2020 4:41 am

ive been using the network boot support to greatly speed up my test cycles with custom `start4.elf` files, works wonders for that

but now that i'm dumping GPIO state from that firmware, i notice that the SPI firmware didnt fully clean up the gpio modes after using the network hw

Code: Select all

GPIO00   IN HIGH |  LOW   IN GPIO32
GPIO01   IN HIGH |  LOW   IN GPIO33
GPIO02   IN HIGH | HIGH   IN GPIO34
GPIO03   IN HIGH | HIGH   IN GPIO35
GPIO04   IN HIGH | HIGH   IN GPIO36
GPIO05   IN HIGH |  LOW   IN GPIO37
GPIO06   IN HIGH |  LOW   IN GPIO38
GPIO07   IN HIGH |  LOW   IN GPIO39
GPIO08   IN HIGH | HIGH   IN GPIO40
GPIO09   IN  LOW | HIGH   IN GPIO41
GPIO10   IN  LOW | HIGH  OUT GPIO42
GPIO11   IN  LOW | HIGH   IN GPIO43
GPIO12   IN  LOW | HIGH   IN GPIO44
GPIO13   IN  LOW | HIGH   IN GPIO45
GPIO14 ALT0 HIGH |  LOW   IN GPIO46
GPIO15 ALT0 HIGH |  LOW   IN GPIO47
GPIO16   IN  LOW |  LOW   IN GPIO48
GPIO17   IN  LOW |  LOW   IN GPIO49
GPIO18   IN  LOW |  LOW   IN GPIO50
GPIO19   IN  LOW |  LOW   IN GPIO51
GPIO20   IN  LOW |  LOW   IN GPIO52
GPIO21   IN  LOW |  LOW   IN GPIO53
GPIO22   IN  LOW
GPIO23   IN  LOW
GPIO24   IN  LOW
GPIO25   IN  LOW
GPIO26   IN  LOW
GPIO27   IN  LOW
GPIO28 ALT5 HIGH
GPIO29 ALT5  LOW
GPIO30   IN  LOW
GPIO31   IN  LOW 
nearly all of the gpio are set to input, which is a sane default

gpio 14/15 i can understand being in alt0, i enabled uart debug in `bootconf.txt` and my firmware would have set that mode anyways on boot

gpio42 being output is also due to my firmware, so i can blink the green LED

but gpio 28/29 being in alt5, is not my doing, https://elinux.org/RPi_BCM2711_GPIOs says that is RGMII_MDIO/RGMII_MDC

this makes me think, that the current beta firmware, is leaving some of the network hw partially enabled when it executes start4.elf, which might be considered a bug

uncarvedblock78
Posts: 17
Joined: Mon Oct 30, 2017 3:30 pm

Re: Raspberry Pi4 bootloader - network boot support - BETA

Wed Jan 22, 2020 2:57 pm

After successfully network booting a pi4 tftpboot/nfsroot, I'm currently trying tftpboot/iscsi lun. and getting some odd behavior.

I've configured a fresh install of Raspbian (firmware/software updated) on a sd card and synced it to the tftp and lun. Booting from tftp, the boot process stalls at start4.elf, failing to load the initramfs img. Booting the same configuration from the sd card, the initramfs appears to load fine (and properly mounts the lun).

I don't know if it's worth mentioning, but I did create the initramfs image from the 64bit kernel. I'm not sure where to begin troubleshooting this, any help would be appreciated. Thanks!

ezalpar
Posts: 3
Joined: Fri Nov 14, 2014 2:18 pm

Re: Raspberry Pi4 bootloader - network boot support - BETA

Mon Jan 27, 2020 10:16 am

Hello,


I am facing a strange problem: my raspberry is booting from the network, but I have no output on both hdmi ports.
can you

please guide me a bit as i keep searching on the internet but i didn't find anything

dickon
Posts: 708
Joined: Sun Dec 09, 2012 3:54 pm
Location: Home, just outside Reading

Re: Raspberry Pi4 bootloader - network boot support - BETA

Mon Jan 27, 2020 5:56 pm

cleverca22 wrote:
Wed Jan 22, 2020 4:41 am
this makes me think, that the current beta firmware, is leaving some of the network hw partially enabled when it executes start4.elf, which might be considered a bug
Well, given that start4.elf needs to load quite a few other bits from the network, I'd consider it sensible. Also, since AIUI the RPF don't support user-written code running before the kernel, calling it a bug is a bit much. For all you know, start4.elf relies on it.

cleverca22
Posts: 280
Joined: Sat Aug 18, 2012 2:33 pm

Re: Raspberry Pi4 bootloader - network boot support - BETA

Tue Jan 28, 2020 12:41 am

i'm comparing it against how the rpi3 behaves, when that mask rom loads bootcode.bin over the network, it reverts all gpio pins back to input mode

so the rpi4 is behaving differently, which might be a bug

hancin
Posts: 1
Joined: Wed Feb 05, 2020 6:32 pm

Re: Raspberry Pi4 bootloader - network boot support - BETA

Wed Feb 05, 2020 6:38 pm

Has anyone tried to run the Pi4 in a 'boot from network first', then 'boot from sd card' setup?

We're trying to do that, so we can occasionally reflash our pi to some known good image, but while the network booting part has had great success the boot from SD card part leads to very weird kernel panics / crashes/ memory dumps.

I'm a bit stumped on where to go next from here. Does the firmware on the sd card need to be a specific version for this scenario to work?

sjevtic
Posts: 3
Joined: Mon Feb 17, 2020 3:39 pm

Re: Raspberry Pi4 bootloader - network boot support - BETA

Mon Feb 17, 2020 9:38 pm

Hello Raspberry Pi Team,

I have been testing the network boot feature in your latest bootloader (pieeprom-2020-01-17.bin) with some success, though the experience has been far from smooth. Of course, all of this is expected since it is beta functionality, and accordingly, please find below my feedback.
beldzhang wrote:
Tue Jan 14, 2020 6:46 am
1) in DHCP requests:
other than the MAC address, there is no any field to identify a RPi client,

I noticed this as well, and it is not especially convenient. Raspberry Pi devices emit the same DHCP option 60 payload as common x86 PXE PCs, preventing the use of vendor-specific options in the manner which they were intended: to service a particular class of clients. MAC address tagging appears to be the only way at the moment to readily identify Raspberry Pi devices, and it is worth noting that not all DHCP environments make this functionality readily accessible.

There is a related, yet still distinct matter: configuring individual Raspberry Pi devices with unique parameters. The existing TFTP_PREFIX and TFTP_PREFIX_STR bootloader options achieve this in a rudimentary fashion (i.e., at the expense of maintaining a collection of board-specific files on the server), but are still less convenient than network bootloader environments like U-Boot and iPXE where configuration elements such as kernel command lines can be assembled at runtime from various data elements.

I realize that U-Boot or iPXE will likely never find their way into the Raspberry Pi 4 boot EEPROM based on size alone, but hopefully some day soon this will be easier, either because of enhancements in the Raspberry Pi 4 bootloader, or through chain loading of U-Boot and/or iPXE. As of this writing, network functionality for the Raspberry Pi 4 in both U-Boot and iPXE appears to be incomplete.
beldzhang wrote:
Tue Jan 14, 2020 6:46 am
4) TFTP download files
blocksize is 1024, could be larger?
is there any future plan to implement windowsize ? since start.elf / kernel / initrd are big, set windowsize > 1 to speed up downloading.

I have also found the TFTP performance of the Raspberry Pi 4 bootloader to be rather slow, at least in comparison to my reference environment (U-Boot 2018.07 on i.MX6) which claims 10.3 MiB/s for both kernel and initramfs downloads. Part of this is definitely related to block size: Raspberry Pi uses blocksize=1024 whereas my reference device uses blocksize=1468.

In addition I have observed two other anomalies related to the Raspberry Pi 4 TFTP download implementation.

First, my TCP Dump captures show numerous unacknowledged TFTP data packets from the server (Synology's clearly patched "TFTP Server SinglePort Version 1.66"). I do not observe any such unacknowledged packets TFTP data packets with the reference device.

Secondly, the Raspberry Pi occasionally fails to boot from the network, hanging either after "Starting start4.elf" or just before mounting the root filesystem. In the latter case a console message similar to the following is displayed:

Code: Select all

[    7.548926] RAMDISK: gzip image found at block 0
[    7.674065] RAMDISK: incomplete write (19060 != 32768)

I see no boot issues with the reference device, and no issues with network connectivity under Linux on either. Can you offer any suggestions as to what might be going wrong?

Thanks.

beldzhang
Posts: 6
Joined: Tue Jan 14, 2020 5:53 am

Re: Raspberry Pi4 bootloader - network boot support - BETA

Fri Feb 21, 2020 2:52 am

sjevtic wrote:
Mon Feb 17, 2020 9:38 pm
I noticed this as well, and it is not especially convenient. Raspberry Pi devices emit the same DHCP option 60 payload as common x86 PXE PCs, preventing the use of vendor-specific options in the manner which they were intended: to service a particular class of clients. MAC address tagging appears to be the only way at the moment to readily identify Raspberry Pi devices, and it is worth noting that not all DHCP environments make this functionality readily accessible.

is there any future plan to implement windowsize ? since start.elf / kernel / initrd are big, set windowsize > 1 to speed up downloadin

I have also found the TFTP performance of the Raspberry Pi 4 bootloader to be rather slow, at least in comparison to my reference environment (U-Boot 2018.07 on i.MX6) which claims 10.3 MiB/s for both kernel and initramfs downloads. Part of this is definitely related to block size: Raspberry Pi uses blocksize=1024 whereas my reference device uses blocksize=1468.
me and timg236 has a lot of discussion about this, you may have a look
https://github.com/raspberrypi/rpi-eeprom/issues/82
https://github.com/raspberrypi/rpi-eeprom/issues/87
sjevtic wrote:
Mon Feb 17, 2020 9:38 pm
First, my TCP Dump captures show numerous unacknowledged TFTP data packets from the server (Synology's clearly patched "TFTP Server SinglePort Version 1.66"). I do not observe any such unacknowledged packets TFTP data packets with the reference device.
Secondly, the Raspberry Pi occasionally fails to boot from the network, hanging either after "Starting start4.elf" or just before mounting the root filesystem. In the latter case a console message similar to the following is displayed:

Code: Select all

[    7.548926] RAMDISK: gzip image found at block 0
[    7.674065] RAMDISK: incomplete write (19060 != 32768)
in my environments(TFTP server was written by myself), I didn't notice any unusual TFTP packets, I will do a capture to see is there have.
also as my testing, never meet any RAMDISK issues,
in config.txt, I always set

Code: Select all

initramfs rpi4-initrd.gz followkernel.
and initrd was compressed by lzma

hopefully this is useful for you.

dickon
Posts: 708
Joined: Sun Dec 09, 2012 3:54 pm
Location: Home, just outside Reading

Re: Raspberry Pi4 bootloader - network boot support - BETA

Fri Feb 21, 2020 10:06 am

sjevtic wrote:
Mon Feb 17, 2020 9:38 pm
Secondly, the Raspberry Pi occasionally fails to boot from the network, hanging either after "Starting start4.elf" or just before mounting the root filesystem. In the latter case a console message similar to the following is displayed:

Code: Select all

[    7.548926] RAMDISK: gzip image found at block 0
[    7.674065] RAMDISK: incomplete write (19060 != 32768)
I've seen the same thing on occasion. It isn't all that common and a yank-the-power reboot solves it, but it can be irritating when it happens. Always the initrd for me; I haven't noticed any other phase breaking. I haven't kept an eye on the offsets where it breaks, but my (unreliable) memory suggests they're more or less random, and not constant. Pi 4 only, I think; I haven't seen it on the older devices.

sjevtic
Posts: 3
Joined: Mon Feb 17, 2020 3:39 pm

Re: Raspberry Pi4 bootloader - network boot support - BETA

Fri Feb 21, 2020 2:48 pm

dickon wrote:
Fri Feb 21, 2020 10:06 am
sjevtic wrote:
Mon Feb 17, 2020 9:38 pm
Secondly, the Raspberry Pi occasionally fails to boot from the network, hanging either after "Starting start4.elf" or just before mounting the root filesystem. In the latter case a console message similar to the following is displayed:

Code: Select all

[    7.548926] RAMDISK: gzip image found at block 0
[    7.674065] RAMDISK: incomplete write (19060 != 32768)

I've seen the same thing on occasion. It isn't all that common and a yank-the-power reboot solves it, but it can be irritating when it happens. Always the initrd for me; I haven't noticed any other phase breaking. I haven't kept an eye on the offsets where it breaks, but my (unreliable) memory suggests they're more or less random, and not constant. Pi 4 only, I think; I haven't seen it on the older devices.

I have not collected any stats or done any other sort of deep dive on this, but I would estimate the failure rate at 5%. But these 'berries are going to be used in a headless, autonomous application so even that is not acceptable.

I am really hoping someone from Raspberry Pi will comment on this. As I noted earlier, my tcpdump suggests that the TFTP interaction is not quite right. I am not even sure right now where the source for the bootloader code stored in EEPROM lives; the rpi-eeprom repository seems to be binary-only.

Incidentally, I was also previously having a lot of problems with warm boot scenarios (or for that matter, any time when the board had not been unpowered for at least a minute or so before booting again) hanging after "Starting start4.elf"; the failure rate was approaching 100%. I was able to address this by building and adding armstub8-gic.bin from the tools repository to the boot folder on my TFTP server along with the following in config.txt:

Code: Select all

arm_64bit=1
enable_gic=1
armstub=armstub8-gic.bin

It is clear that this code initializes a variety of registers; presumably the power-on defaults for a cold board are sufficiently sane that it is possible to get away without armstubs in this scenario. It is less clear though why the SD card Raspbian image does not include armstubs, and why it does not have this issue when warm booting.

sjevtic
Posts: 3
Joined: Mon Feb 17, 2020 3:39 pm

Re: Raspberry Pi4 bootloader - network boot support - BETA

Fri Feb 21, 2020 3:36 pm

beldzhang wrote:
Fri Feb 21, 2020 2:52 am
sjevtic wrote:
Mon Feb 17, 2020 9:38 pm
I noticed this as well, and it is not especially convenient. Raspberry Pi devices emit the same DHCP option 60 payload as common x86 PXE PCs, preventing the use of vendor-specific options in the manner which they were intended: to service a particular class of clients. MAC address tagging appears to be the only way at the moment to readily identify Raspberry Pi devices, and it is worth noting that not all DHCP environments make this functionality readily accessible.

is there any future plan to implement windowsize ? since start.elf / kernel / initrd are big, set windowsize > 1 to speed up downloadin

I have also found the TFTP performance of the Raspberry Pi 4 bootloader to be rather slow, at least in comparison to my reference environment (U-Boot 2018.07 on i.MX6) which claims 10.3 MiB/s for both kernel and initramfs downloads. Part of this is definitely related to block size: Raspberry Pi uses blocksize=1024 whereas my reference device uses blocksize=1468.
me and timg236 has a lot of discussion about this, you may have a look
https://github.com/raspberrypi/rpi-eeprom/issues/82
https://github.com/raspberrypi/rpi-eeprom/issues/87

Thanks for sharing that. It was an interesting read; your expertise in network boot is clearly an asset here, and I am glad you dealt with this ahead of me. :)

Sadly, it sounds like significant TFTP performance improvements are a long way out, though I certainly hope to get the core functionality working in short order. I truly hope U-Boot's network support for the '2711 will be stable soon since its TFTP implementation is mature and performs well; downloading a <1 MB U-Boot image with the Raspberry Pi bootloader would be far preferable to downloading ~14 MB of kernel + initramfs with the same. It will also make managing the boot environment MUCH simpler because it will avoid the need to maintain serial number/MAC address-specific directories to generate unique kernel command lines for each board; on the existing reference boards I generate the command line on the fly using U-Boot's scripting engine to uniquely call out the iSCSI LUNs to be used as the rootfs for each board. It works great.
beldzhang wrote:
Fri Feb 21, 2020 2:52 am
sjevtic wrote:
Mon Feb 17, 2020 9:38 pm
First, my TCP Dump captures show numerous unacknowledged TFTP data packets from the server (Synology's clearly patched "TFTP Server SinglePort Version 1.66"). I do not observe any such unacknowledged packets TFTP data packets with the reference device.
Secondly, the Raspberry Pi occasionally fails to boot from the network, hanging either after "Starting start4.elf" or just before mounting the root filesystem. In the latter case a console message similar to the following is displayed:

Code: Select all

[    7.548926] RAMDISK: gzip image found at block 0
[    7.674065] RAMDISK: incomplete write (19060 != 32768)
in my environments(TFTP server was written by myself), I didn't notice any unusual TFTP packets, I will do a capture to see is there have.

The behavior does not make a lot of sense, and seems like it could actually be an issue in the TFTP server itself. After all, why should the TFTP server keep on sending packets when it has not received and acknowledgement for the previous one? It is clear here that the windowsize is 1. I suspect these could be artifacts of the existence checks that the Raspberry Pi bootloader performs.
beldzhang wrote:
Fri Feb 21, 2020 2:52 am
also as my testing, never meet any RAMDISK issues,
in config.txt, I always set

Code: Select all

initramfs rpi4-initrd.gz followkernel.
and initrd was compressed by lzma

hopefully this is useful for you.

I have the same config.txt statement. ~95% of the time, there are no issues.

Thank you for your very detailed response.

timg236
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 292
Joined: Thu Jun 21, 2018 4:30 pm

Re: Raspberry Pi4 bootloader - network boot support - BETA

Fri Feb 21, 2020 4:00 pm

If you have a UART cable then setting BOOT_UART=1 in the EEPROM configuration might give some useful statistics. On successful boots you could try "vcdbg log msg" to see if there is any suspicious information there.

The DHCP option60 and option97 changes are on a dev branch, hopefully they can be released before too long.

cleverca22
Posts: 280
Joined: Sat Aug 18, 2012 2:33 pm

Re: Raspberry Pi4 bootloader - network boot support - BETA

Fri Feb 21, 2020 4:03 pm

on my rpi3, i have found that any tftp timeout (due to temporary packet loss) silently gets treated as file-not-found (which can cause it to load the wrong kernel or claim start.elf doesnt exist)

if the rpi4 bootcode.bin is using the same codebase, it might have similar bugs?

beldzhang
Posts: 6
Joined: Tue Jan 14, 2020 5:53 am

Re: Raspberry Pi4 bootloader - network boot support - BETA

Fri Feb 21, 2020 5:59 pm

sjevtic wrote:
Fri Feb 21, 2020 3:36 pm

The behavior does not make a lot of sense, and seems like it could actually be an issue in the TFTP server itself. After all, why should the TFTP server keep on sending packets when it has not received and acknowledgement for the previous one? It is clear here that the windowsize is 1. I suspect these could be artifacts of the existence checks that the Raspberry Pi bootloader performs.
do a capture of tftp transmission, no mistake found.
maybe this tftp server think ack pkt was lot and re-transmit? normally tftp timeout is 3 seconds.

I was complete my development boot server and put in here:
https://github.com/beldzhang/netboot/re ... v0.96.4669
you may have a try if you like to.

timg236
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 292
Joined: Thu Jun 21, 2018 4:30 pm

Re: Raspberry Pi4 bootloader - network boot support - BETA

Fri Feb 21, 2020 6:21 pm

The start.elf network stack is largely unchanged from Pi3 and IIRC the timeout for an individual packet was about 1-second which is probably a bit small. It doesn't appear to be configurable but it may be reasonable to increase it.

The Pi 4 bootloader as a new network stack with configurable timeouts at the file level rather than a packet and should be more resilient to dropped packets.

One option is to replace the start.elf with the bootloader TFTP code although that's a fairly large amount of work. Increasing the timeouts or inheriting some options from the bootloader EEPROM config might be more expedient.

Return to “Advanced users”