beta-tester
Posts: 1249
Joined: Fri Jan 04, 2013 1:57 pm
Location: de_DE

[solved] Raspbian Stretch: RPi3 Network booting, server broken

Sat Aug 19, 2017 11:38 pm

hi,
with Raspbian Stretch the server for RPi3 network booting is broken.
with Raspbian Jessie the server for RPi3 network booting is working well.

the tutorial Network Boot Your Raspberry Pi is not working anymore, when you use Raspbian Stretch.

i tried to only use /etc/dhcpcd.conf instead of /etc/network/interfaces, but same issue.
i tried to change the network adapter naming back to old style, but still same issue.

the issue is, the booting RPi3 is not able to mount the nfsroot.
on the booting RPi3 stucks and runs into kernel panic.

Code: Select all

IP-Config: Got DHCP answer from 192.168.0.1, my address is 192.168.0.210
IP-Config: Complete:
     device=eth0, hwaddr=xx:xx:xx:xx:xx:xx, ipaddr=192.168.0.210, mask=255.255.255.0, gw=192.168.0.1
     host=192.168.0.210, domain=, nis-domain=(none)
     bootserver=192.168.0.1, rootserver=192.168.0.1, rootpath=     nameserver0=192.168.0.1
CPU1: stopping ...
CPU2: stopping ...
CPU3: stopping ...
CPU4: stopping ...
---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(2,0)
the cmdline.txt file:

Code: Select all

dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=/dev/nfs nfsroot=192.168.0.1:/srv/nfs/rpi-12345678-root rw rootwait ip=dhcp elevator=deadline
on the Raspbian Stretch server i get following log:

Code: Select all

dnsmasq-dhcp[500]: 653460281 available DHCP range: 192.168.0.200 -- 192.168.0.249
dnsmasq-dhcp[500]: 653460281 vendor class: PXEClient:Arch:00000:UNDI:002001
dnsmasq-dhcp[500]: 653460281 DHCPDISCOVER(eth0) xx:xx:xx:xx:xx:xx
dnsmasq-dhcp[500]: 653460281 tags: ARM_RPI3, UUID_RPI3, ARCH_0, eth0
dnsmasq-dhcp[500]: 653460281 DHCPOFFER(eth0) 192.168.0.210 xx:xx:xx:xx:xx:xx
dnsmasq-dhcp[500]: 653460281 requested options: 43:vendor-encap, 60:vendor-class, 67:bootfile-name,
dnsmasq-dhcp[500]: 653460281 requested options: 128, 129, 130, 131, 132, 133, 134, 135, 66:tftp-server
dnsmasq-dhcp[500]: 653460281 next server: 192.168.0.1
dnsmasq-dhcp[500]: 653460281 sent size:  1 option: 53 message-type  2
dnsmasq-dhcp[500]: 653460281 sent size:  4 option: 54 server-identifier  192.168.0.1
dnsmasq-dhcp[500]: 653460281 sent size:  4 option: 51 lease-time  1d
dnsmasq-dhcp[500]: 653460281 sent size: 13 option: 67 bootfile-name  bootcode.bin
dnsmasq-dhcp[500]: 653460281 sent size:  4 option: 58 T1  12h
dnsmasq-dhcp[500]: 653460281 sent size:  4 option: 59 T2  21h
dnsmasq-dhcp[500]: 653460281 sent size:  4 option:  1 netmask  255.255.255.0
dnsmasq-dhcp[500]: 653460281 sent size:  4 option: 28 broadcast  192.168.0.255
dnsmasq-dhcp[500]: 653460281 sent size:  9 option: 60 vendor-class  50:58:45:43:6c:69:65:6e:74
dnsmasq-dhcp[500]: 653460281 sent size: 17 option: 97 client-machine-id  00:44:44:44:44:44:44:44:44:44:44:44:44:44...
dnsmasq-dhcp[500]: 653460281 sent size: 44 option: 43 vendor-encap  06:01:03:0a:04:00:50:58:45:08:07:80:00:01...
dnsmasq-tftp[500]: file /srv/tftp/bootsig.bin not found
dnsmasq-tftp[500]: sent /srv/tftp/bootcode.bin to 192.168.0.210
dnsmasq-tftp[500]: file /srv/tftp/12345678/autoboot.txt not found
dnsmasq-tftp[500]: error 0 Early terminate received from 192.168.0.210
dnsmasq-tftp[500]: failed sending /srv/tftp/12345678/start.elf to 192.168.0.210
dnsmasq-tftp[500]: sent /srv/tftp/12345678/config.txt to 192.168.0.210
dnsmasq-tftp[500]: file /srv/tftp/12345678/recovery.elf not found
dnsmasq-tftp[500]: sent /srv/tftp/12345678/start.elf to 192.168.0.210
dnsmasq-tftp[500]: sent /srv/tftp/12345678/fixup.dat to 192.168.0.210
dnsmasq-tftp[500]: file /srv/tftp/12345678/recovery.elf not found
dnsmasq-tftp[500]: sent /srv/tftp/12345678/config.txt to 192.168.0.210
dnsmasq-tftp[500]: file /srv/tftp/12345678/dt-blob.bin not found
dnsmasq-tftp[500]: file /srv/tftp/12345678/recovery.elf not found
dnsmasq-tftp[500]: sent /srv/tftp/12345678/config.txt to 192.168.0.210
dnsmasq-tftp[500]: file /srv/tftp/12345678/bootcfg.txt not found
dnsmasq-tftp[500]: sent /srv/tftp/12345678/cmdline.txt to 192.168.0.210
dnsmasq-tftp[500]: file /srv/tftp/12345678/recovery8.img not found
dnsmasq-tftp[500]: file /srv/tftp/12345678/recovery8-32.img not found
dnsmasq-tftp[500]: file /srv/tftp/12345678/recovery7.img not found
dnsmasq-tftp[500]: file /srv/tftp/12345678/recovery.img not found
dnsmasq-tftp[500]: file /srv/tftp/12345678/kernel8.img not found
dnsmasq-tftp[500]: file /srv/tftp/12345678/kernel8-32.img not found
dnsmasq-tftp[500]: file /srv/tftp/12345678/armstub8.bin not found
dnsmasq-tftp[500]: error 0 Early terminate received from 192.168.0.210
dnsmasq-tftp[500]: failed sending /srv/tftp/12345678/kernel7.img to 192.168.0.210
dnsmasq-tftp[500]: file /srv/tftp/12345678/armstub8-32.bin not found
dnsmasq-tftp[500]: file /srv/tftp/12345678/armstub7.bin not found
dnsmasq-tftp[500]: file /srv/tftp/12345678/armstub.bin not found
dnsmasq-tftp[500]: sent /srv/tftp/12345678/kernel7.img to 192.168.0.210
dnsmasq-tftp[500]: sent /srv/tftp/12345678/bcm2710-rpi-3-b.dtb to 192.168.0.210
dnsmasq-tftp[500]: sent /srv/tftp/12345678/config.txt to 192.168.0.210
dnsmasq-dhcp[500]: 398005002 available DHCP range: 192.168.0.200 -- 192.168.0.249
dnsmasq-dhcp[500]: 398005002 DHCPDISCOVER(eth0) xx:xx:xx:xx:xx:xx
dnsmasq-dhcp[500]: 398005002 tags: eth0
dnsmasq-dhcp[500]: 398005002 DHCPOFFER(eth0) 192.168.0.210 xx:xx:xx:xx:xx:xx
dnsmasq-dhcp[500]: 398005002 requested options: 1:netmask, 3:router, 6:dns-server, 12:hostname,
dnsmasq-dhcp[500]: 398005002 requested options: 15:domain-name, 17:root-path, 26:mtu, 40:nis-domain
dnsmasq-dhcp[500]: 398005002 next server: 192.168.0.1
dnsmasq-dhcp[500]: 398005002 sent size:  1 option: 53 message-type  2
dnsmasq-dhcp[500]: 398005002 sent size:  4 option: 54 server-identifier  192.168.0.1
dnsmasq-dhcp[500]: 398005002 sent size:  4 option: 51 lease-time  1d
dnsmasq-dhcp[500]: 398005002 sent size:  4 option: 58 T1  12h
dnsmasq-dhcp[500]: 398005002 sent size:  4 option: 59 T2  21h
dnsmasq-dhcp[500]: 398005002 sent size:  4 option:  1 netmask  255.255.255.0
dnsmasq-dhcp[500]: 398005002 sent size:  4 option: 28 broadcast  192.168.0.255
dnsmasq-dhcp[500]: 398005002 sent size:  4 option:  3 router  192.168.0.1
dnsmasq-dhcp[500]: 398005002 sent size:  4 option:  6 dns-server  192.168.0.1
dnsmasq-dhcp[500]: 398005002 available DHCP range: 192.168.0.200 -- 192.168.0.249
dnsmasq-dhcp[500]: 398005002 DHCPREQUEST(eth0) 192.168.0.210 xx:xx:xx:xx:xx:xx
dnsmasq-dhcp[500]: 398005002 tags: eth0
dnsmasq-dhcp[500]: 398005002 DHCPACK(eth0) 192.168.0.210 xx:xx:xx:xx:xx:xx
dnsmasq-dhcp[500]: 398005002 requested options: 1:netmask, 3:router, 6:dns-server, 12:hostname,
dnsmasq-dhcp[500]: 398005002 requested options: 15:domain-name, 17:root-path, 26:mtu, 40:nis-domain
dnsmasq-dhcp[500]: 398005002 next server: 192.168.0.1
dnsmasq-dhcp[500]: 398005002 sent size:  1 option: 53 message-type  5
dnsmasq-dhcp[500]: 398005002 sent size:  4 option: 54 server-identifier  192.168.0.1
dnsmasq-dhcp[500]: 398005002 sent size:  4 option: 51 lease-time  1d
dnsmasq-dhcp[500]: 398005002 sent size:  4 option: 58 T1  12h
dnsmasq-dhcp[500]: 398005002 sent size:  4 option: 59 T2  21h
dnsmasq-dhcp[500]: 398005002 sent size:  4 option:  1 netmask  255.255.255.0
dnsmasq-dhcp[500]: 398005002 sent size:  4 option: 28 broadcast  192.168.0.255
dnsmasq-dhcp[500]: 398005002 sent size:  4 option:  3 router  192.168.0.1
dnsmasq-dhcp[500]: 398005002 sent size:  4 option:  6 dns-server  192.168.0.1
rpc.mountd[437]: authenticated mount request from 192.168.0.210:812 for /srv/nfs/rpi-12345678-root (/srv/nfs/rpi-12345678-root)
rpc.mountd[437]: authenticated mount request from 192.168.0.210:678 for /srv/nfs/rpi-12345678-root (/srv/nfs/rpi-12345678-root)
rpc.mountd[437]: authenticated mount request from 192.168.0.210:735 for /srv/nfs/rpi-12345678-root (/srv/nfs/rpi-12345678-root)
rpc.mountd[437]: authenticated mount request from 192.168.0.210:862 for /srv/nfs/rpi-12345678-root (/srv/nfs/rpi-12345678-root)
rpc.mountd[437]: authenticated mount request from 192.168.0.210:756 for /srv/nfs/rpi-12345678-root (/srv/nfs/rpi-12345678-root)
rpc.mountd[437]: authenticated mount request from 192.168.0.210:1004 for /srv/nfs/rpi-12345678-root (/srv/nfs/rpi-12345678-root)


i can sucessfully mount 192.168.0.1:/srv/nfs/rpi-12345678-root of the Raspbian Stretch server by hand, from the RPi3 to the server, when the RPi3 was booted with a local installed OS on the sd card.

i can sucessfully network boot the Rpi3, when the server is running Raspbian Jessie - event, when the RPi3 gets Raspbian Stretch as booting system from the Raspbian Jessie server.
Last edited by beta-tester on Sat Aug 26, 2017 4:33 pm, edited 1 time in total.
{ I only give negative feedback }
RPi B (rev1, 256MB), B (rev2, 512MB), B+, 2B, 3B, 3B+, ZeroW, 4B (4GB), ...

beta-tester
Posts: 1249
Joined: Fri Jan 04, 2013 1:57 pm
Location: de_DE

Re: Raspbian Stretch: RPi3 Network booting, server broken

Sun Aug 20, 2017 1:16 pm

i still struggling around with Raspbian Stretch as server for RPi3 network booting... :x
{ I only give negative feedback }
RPi B (rev1, 256MB), B (rev2, 512MB), B+, 2B, 3B, 3B+, ZeroW, 4B (4GB), ...

leempi
Posts: 4
Joined: Sat Aug 26, 2017 12:23 am

Re: Raspbian Stretch: RPi3 Network booting, server broken

Sat Aug 26, 2017 1:01 am

It worked for me but I have a few differences.

I started with a fresh copy of 2017-08-16-raspbian-stretch-lite.img. Touched ssh/tweaked fstab and cmdline.txt after it came up, mounted its new home over NFS and rsync'd the lot there.

The NFS server is a small x86_64 Ubuntu Xenial box that handles DNS/DHCP/NFS/Plex/routing/etc. for the house.

I'm using a static IP vs. DHCP when NFSROOT'd. This also had the side effect of converting the nic name back to eth0 from the lovely consistent device name it had booting the same image locally. I'm not sure vers=3 as a NFS option is required in fstab anymore but it crept forward from an older system.

Code: Select all

[email protected] ~ $ cat /boot/cmdline.txt
dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 ip=192.168.1.85:192.168.1.3:192.168.1.3:255.255.255.0:pi:eth0:off root=/dev/nfs nfsroot=192.168.1.3:/exports/labpi rw elevator=deadline fsck.repair=yes rootwait

[email protected] ~ $ cat /etc/fstab
192.168.1.3:/exports/labpi	/	nfs	rw,vers=3	0	0
PARTUUID=2b43694f-01  /boot           vfat    defaults          0       2
proc            /proc           proc    defaults          0       0
#PARTUUID=2b43694f-02  /               ext4    defaults,noatime  0       1
# a swapfile is not a swap partition, no line here
#   use  dphys-swapfile swap[on|off]  for that
192.168.1.3:/mnt/scratch        /mnt/scratch    nfs     _netdev,rw,exec 0 0
192.168.1.3:/exports/pihome     /home/leem      nfs     _netdev,rw,exec 0 0

[email protected] ~ $ df -h
Filesystem                   Size  Used Avail Use% Mounted on
192.168.1.3:/exports/labpi   9.8G  1.1G  8.2G  12% /
devtmpfs                     460M     0  460M   0% /dev
tmpfs                        464M     0  464M   0% /dev/shm
tmpfs                        464M   12M  452M   3% /run
tmpfs                        5.0M  4.0K  5.0M   1% /run/lock
tmpfs                        464M     0  464M   0% /sys/fs/cgroup
/dev/mmcblk0p1                42M   21M   21M  51% /boot
192.168.1.3:/exports/pihome  3.9G  870M  2.9G  24% /home/leem
192.168.1.3:/mnt/scratch      59G  8.7G   48G  16% /mnt/scratch
tmpfs                         93M     0   93M   0% /run/user/1000
[email protected] ~ $ 

Cheers!

leempi
Posts: 4
Joined: Sat Aug 26, 2017 12:23 am

Re: Raspbian Stretch: RPi3 Network booting, server broken

Sat Aug 26, 2017 5:32 am

Re-read. dist-upgraded a pi that had a hard drive to stretch (a chore on its own).

Verified your exact issue. I network traced the connection for that IP, I see NFS mount request come in, get accepted, then the client wants to do some NFS stuff. Well, while the server ACCEPTED the mount on NFS V2 it starts rejecting RPC alls with:

Accept State: remote can't support version # (2)
Minimum: 3 Maximum 4

Hmm. Added a couple parms to /etc/default/nfs-kernel-server - --no-nfs-version 2 -d all

it got better, I no longer see the mount requests in syslog. I went back to /boot/cmdline on the client and added a NFS option vers=3 after the mount point. it works!

So it's kind of a 2 level bug. rcp.mounted on stretch doesn't seem to really support NFS v2 tho it advertises it and/or the nfs code in the /boot partitions kernel is too dumb to negotiate up.

The server:

Code: Select all

[email protected]:~# cat /etc/debian_version 
9.1
[email protected]:~# grep newpi /etc/exports
/exports/newpi	192.168.1.102(rw,no_root_squash,async,insecure,no_subtree_check,mp)
[email protected]:~# cat /etc/dhcp/dhcpd.conf 
default-lease-time 600;
max-lease-time 7200;
option subnet-mask 255.255.255.0;
option broadcast-address 192.168.1.255;
option routers 192.168.1.1;
option domain-name-servers 192.168.1.3;
option domain-name "silicon.heaven";
subnet 192.168.1.0 netmask 255.255.255.0 {
}
host newpi {
  hardware ethernet b8:27:eb:32:eb:03;
  fixed-address 192.168.1.102;
}

[email protected]:~# 
The client:

Code: Select all

[email protected] ~ $ cat /boot/cmdline.txt 
dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 ip=dhcp root=/dev/nfs nfsroot=192.168.1.82:/exports/newpi,vers=3 rw elevator=deadline fsck.repair=yes rootwait
[email protected] ~ $ df -h /
Filesystem                   Size  Used Avail Use% Mounted on
192.168.1.82:/exports/newpi  7.9G  1.1G  6.4G  15% /
[email protected] ~ $ 

Cheers!

Lee

beta-tester
Posts: 1249
Joined: Fri Jan 04, 2013 1:57 pm
Location: de_DE

Re: Raspbian Stretch: RPi3 Network booting, server broken

Sat Aug 26, 2017 4:33 pm

leempi wrote:
Sat Aug 26, 2017 5:32 am
...
Verified your exact issue. I network traced the connection for that IP, I see NFS mount request come in, get accepted, then the client wants to do some NFS stuff. Well, while the server ACCEPTED the mount on NFS V2 it starts rejecting RPC alls with:

Accept State: remote can't support version # (2)
Minimum: 3 Maximum 4

Hmm. Added a couple parms to /etc/default/nfs-kernel-server - --no-nfs-version 2 -d all

it got better, I no longer see the mount requests in syslog. I went back to /boot/cmdline on the client and added a NFS option vers=3 after the mount point. it works!

So it's kind of a 2 level bug. rcp.mounted on stretch doesn't seem to really support NFS v2 tho it advertises it and/or the nfs code in the /boot partitions kernel is too dumb to negotiate up.

...

The client:

Code: Select all

[email protected] ~ $ cat /boot/cmdline.txt 
dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 ip=dhcp root=/dev/nfs nfsroot=192.168.1.82:/exports/newpi,vers=3 rw elevator=deadline fsck.repair=yes rootwait
Cheers!
Lee
thank you very much, you got it working!!!
simply adding ",vers=3 " at the end of the nfsroot mount parameter fixed the issue i had.

again, thank you very much.

PS.: i will report, that you found a solution how to fix it.
https://github.com/raspberrypi/documentation/issues/719
{ I only give negative feedback }
RPi B (rev1, 256MB), B (rev2, 512MB), B+, 2B, 3B, 3B+, ZeroW, 4B (4GB), ...

beta-tester
Posts: 1249
Joined: Fri Jan 04, 2013 1:57 pm
Location: de_DE

Re: [solved] Raspbian Stretch: RPi3 Network booting, server broken

Mon Aug 28, 2017 9:00 am

PS.: it is interesting, that only RPi3 has that nfs mount problem.
the same server (RPi2 + Raspbian Stretch) is acting also as PXE server for x86 PC's, and all the available OS's can pxe boot and mount the nfsroot without problems and without the ",vers=3" addition.
ubuntu-16.04.3, ubuntu-17.04, ubuntu-12-04.4, debian-live-9.1.0, raspberry pi jessie for x08 (2017-06-22), ...

so i think it must be a bug/a limitation in bootcode.bin or start.efi of raspbian, that the booting RPi3 is loading at network booting.
https://github.com/raspberrypi/firmware/issues/863
{ I only give negative feedback }
RPi B (rev1, 256MB), B (rev2, 512MB), B+, 2B, 3B, 3B+, ZeroW, 4B (4GB), ...

leempi
Posts: 4
Joined: Sat Aug 26, 2017 12:23 am

Re: [solved] Raspbian Stretch: RPi3 Network booting, server broken

Thu Aug 31, 2017 12:41 am

I was using image 2017-08-16-raspbian-stretch-lite.img on the pi3 client. The server was dist-upgraded from Wheezy and is a rPi B model if that helps.

Lee

Return to “Troubleshooting”