User avatar
dickon
Posts: 1824
Joined: Sun Dec 09, 2012 3:54 pm
Location: Home, just outside Reading

Re: Raspberry Pi Zero W boot with NFS root filesystem

Sat May 16, 2020 9:29 pm

Are you still using TCP? I'd use UDP with a 1k block size if I were you. Likely to be more reliable: in the main, TCP throughput drops through the floor with a ~2% packet loss -- which is *easy* over wifi -- and drops to more or less zero at ~5% -- which is *not hard* over wifi. You may well find UDP works better, if you constrain the block size to something sane.

JovianPyx
Posts: 132
Joined: Fri Nov 20, 2015 9:34 pm

Re: Raspberry Pi Zero W boot with NFS root filesystem

Sun May 31, 2020 1:13 am

This is solved for this project. I've implemented a small RAMdisk and use rc.local to copy files that will be used during the applications actions. This works quite well. Once lighttpd loads, it remains RAM resident so during operation, the only thing WiFi is used for is sending the camera image to a browser.

Another problem has popped up though. There was a recent update that replaced the kernel. This update broke the system because now the kernel that is booting is not the same as used to build initrd.img-* for /boot. Because of this, initramfs is never created and all of the processes that cause NFS connection don't run. Since the updates done never touched the root filesystem on the SD card, it was not possible to just switch to running from SD card, regenerate the initrd.img file and reboot. I had to grab a new image for raspbian-lite and re-do all the steps to use an NFS root filesystem. Another solution I thought of was to copy the NFS filesystem (since it's up to date except for the next update) to the SD card, then run off the SD card to regenerate the initrd.img file. Seems like a lot of work for what is in my case a turnkey system.

ejolson
Posts: 6042
Joined: Tue Mar 18, 2014 11:47 am

Re: Raspberry Pi Zero W boot with NFS root filesystem

Sun May 31, 2020 1:37 am

JovianPyx wrote:
Sun May 31, 2020 1:13 am
This is solved for this project. I've implemented a small RAMdisk and use rc.local to copy files that will be used during the applications actions. This works quite well. Once lighttpd loads, it remains RAM resident so during operation, the only thing WiFi is used for is sending the camera image to a browser.

Another problem has popped up though. There was a recent update that replaced the kernel. This update broke the system because now the kernel that is booting is not the same as used to build initrd.img-* for /boot. Because of this, initramfs is never created and all of the processes that cause NFS connection don't run. Since the updates done never touched the root filesystem on the SD card, it was not possible to just switch to running from SD card, regenerate the initrd.img file and reboot. I had to grab a new image for raspbian-lite and re-do all the steps to use an NFS root filesystem. Another solution I thought of was to copy the NFS filesystem (since it's up to date except for the next update) to the SD card, then run off the SD card to regenerate the initrd.img file. Seems like a lot of work for what is in my case a turnkey system.
I think many have had this problem. One solution is to pin the current kernel so that it doesn't update. Since very few security patches affect the kernel, this is not so bad as it sounds. It would still be reasonable to update the kernel once in a while and great if the replacement for Raspbian was more considerate when booting from an initial RAM filesystem.
Last edited by ejolson on Sun May 31, 2020 5:15 am, edited 1 time in total.

JovianPyx
Posts: 132
Joined: Fri Nov 20, 2015 9:34 pm

Re: Raspberry Pi Zero W boot with NFS root filesystem

Sun May 31, 2020 3:36 am

Good suggestion which is in my case easy to implement (not updating the kernel).

I use a script called 'update' to do updates. In the new version of this, I commented out the

Code: Select all

apt full-upgrade
I agree that not much will happen by not upgrading the kernel. Unless there are problems that can be corrected by an update, I will probably just let it run as is and do no updates at all.

DarkElvenAngel
Posts: 1080
Joined: Tue Mar 20, 2018 9:53 pm

Re: Raspberry Pi Zero W boot with NFS root filesystem

Sun May 31, 2020 4:03 am

JovianPyx wrote:
Sun May 31, 2020 3:36 am
Good suggestion which is in my case easy to implement (not updating the kernel).

I use a script called 'update' to do updates. In the new version of this, I commented out the

Code: Select all

apt full-upgrade
I agree that not much will happen by not upgrading the kernel. Unless there are problems that can be corrected by an update, I will probably just let it run as is and do no updates at all.
You can use apt-mark so packages will not be upgraded. That way you can still run apt full-upgrade and it will just skip over the marked packages

https://www.computerhope.com/unix/apt-mark.htm

wintersteiger
Posts: 18
Joined: Sat May 09, 2020 9:02 pm

Re: Raspberry Pi Zero W boot with NFS root filesystem

Thu Nov 05, 2020 8:26 pm

Gave this a good go today with two RPI4s as I was just sick of re-imaging SD cards every other day. The server side is easy to do and the client isn't too hard to figure out if you're on ethernet, but the client wifi requires a lot of cussing. It's crucial not to do this on a headless system or with only a serial console as many error messages will go unnoticed.

I wasted a bunch of time on trying to figure out why the certificate for the wifi regulatory DB couldn't be loaded, e.g.:

Code: Select all

cfg80211: Loading compiled-in X.509 certificates for regulatory database
...
cfg80211: Problem loading in-kernel X.509 certificate (-22)
...
cfg80211: loaded regulatory.db is malformed or signature is missing/invalid
I still don't know why it does that, but despite the scary looking stack trace dumps, this isn't actually required to get a wifi connection. Possibly suboptimal performance if it has to fall back to safe regulatory settings, I guess.

The hardest thing to figure out was why it never even got to the point where it would load the script in /etc/initramfs-tools/scripts/nfs-top/wifi. The reason was that the kernel command line setting ip=dhcp has the effect that it will wait for dhcp on eth0, forever. I had to change my command line to

Code: Select all

ip=::::myhostname:wlan0:dhcp:::
to make sure it doesn't wait for eth0.

With that, things started to work sometimes, but it was very unreliable. For instance, wpa_supplicant sometimes starts a bit too "early" and complains that various driver bits aren't available yet. At other times, it would get DHCP, but the NFS connection failed (maybe default routes not set up yet?), resulting in kernel panics. I ended up with this in my /etc/initramfs-tools/scripts/nfs-top/wifi:

Code: Select all

#!/bin/sh
PREREQ="udev"

prereqs()
{
        echo "$PREREQ"
}

case $1 in
prereqs)
        prereqs
        exit 0
        ;;
esac

sleep 5

/sbin/wpa_supplicant -B -Dwext -iwlan0 -c /etc/wpa_supplicant/wpa_supplicant.conf

counter=0
while ! ip route | grep -o 'default via .* dev wlan0'; do
  echo "Waiting for wlan0..."
  sleep 2
  ipconfig wlan0
  ip route
  counter=$((counter+1))
  if [ "$counter" -gt 10 ]; then
    echo "Expired; help please"
    sh -i
    break
  fi
done

ipconfig wlan0

echo DONE

and the following in /etc/initramfs-tools/hooks/nfsroot:

Code: Select all

#!/bin/sh
PREREQ=""

prereqs()
{
  echo "$PREREQ"
}

case $1 in
prereqs)
  prereqs
  exit 0
  ;;
esac

. /usr/share/initramfs-tools/hook-functions

copy_exec /bin/grep /bin
copy_exec /bin/ps /bin
copy_exec /sbin/lsmod /sbin
copy_exec /sbin/ifconfig /sbin
copy_exec /sbin/ifup /sbin
copy_exec /sbin/ip /sbin
copy_exec /sbin/wpa_supplicant /sbin

mkdir -p ${DESTDIR}/etc/wpa_supplicant
cp /etc/initramfs-tools/wpa_supplicant.boot.conf ${DESTDIR}/etc/wpa_supplicant/wpa_supplicant.conf
mkdir -p ${DESTDIR}/var/run/wpa_supplicant

cp -r /lib/firmware ${DESTDIR}/lib
Note that wpa_supplicant doesn't seem to be able to use the normal control interface while in the initramfs, so I removed

Code: Select all

ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
as well as update_config and country, in my custom wpa_supplicant.conf to get rid of some of those error messages. I'm not 100% sure this was absolutely required to make it boot, but I suspect that this has some connection to the fact that the client now won't reboot anymore. Every time I try to reboot the pi, it runs all the scripts down to the last line (Reached target Reboot.) and then just hangs and I have to power cycle it.

And for those wondering: I get an advertised 150 Mb/s in iwconfig, but it's much slower than an SD card. I guess it's the increased latency. I'll have to look into NFS options and ramdisks to see whether it has any more performance in it.

Return to “Advanced users”