JovianPyx
Posts: 132
Joined: Fri Nov 20, 2015 9:34 pm

Raspberry Pi Zero W boot with NFS root filesystem

Fri Apr 03, 2020 2:15 am

I found this:
http://retinal.dehy.de/docs/doku.php?id ... rryrootnfs
Which is a fairly step by step flow for configuring a Pi Zero W to boot with an NFS mounted root filesystem.

I went through the entire document and did all of the steps. All of the steps worked as described except for the running the update-initramfs script. The code to launch it did not work.

update-initramfs -c -k all

Which I changed to

update-initramfs -c -k $(uname -r)

and then the script worked.

The Pi Zero W I used had already been running Jessie, so I worked with that.

Booting the pi zero, it prints messages about USB enumeration and stops at about 30 seconds waits until about 470 seconds and then says:

VFS: Unable to mount root fs via NFS, trying floppy.

NFS is installed on the Pi Zero W, it can mount the NFS servers exports.

That's telling me that part of this is working.

I tested mounting the root file system copied to the NFS server and that worked.

I can see what appears to be messages about the ramfs which is being called "partitions" such as ram0, ram1, etc.

At this point, I'm not sure what to troubleshoot.

Any help greatly appreciated. I remember reading that the only way to do what I'm doing is to use initramfs which this process does. I've checked the contents of the files, I've made sure that the two scripts I created have the x permission to allow execution.

DarkElvenAngel
Posts: 1085
Joined: Tue Mar 20, 2018 9:53 pm

Re: Raspberry Pi Zero W boot with NFS root filesystem

Fri Apr 03, 2020 2:45 am

JovianPyx,

I asked about this some time ago but didn't get to far.
https://www.raspberrypi.org/forums/view ... 6&t=265941

You've gotten farther than I did. But I see something sticking out your Pi is booting from Jessie? What is the NFS image based off of? In order for it to work the kernel version must match. And there are bugs in the older version of initramfs-tools mostly I encountered them doing updates.

I'm hoping you can sort it out but you should really use Buster.

ejolson
Posts: 6045
Joined: Tue Mar 18, 2014 11:47 am

Re: Raspberry Pi Zero W boot with NFS root filesystem

Fri Apr 03, 2020 3:05 am

The thread on this super cheap cluster

https://www.raspberrypi.org/forums/view ... 9&t=199994

also employs root mounted over NFS for each of the Pi Zero computers. In this case the networking is done over USB.
Last edited by ejolson on Fri Apr 03, 2020 3:25 am, edited 1 time in total.

JovianPyx
Posts: 132
Joined: Fri Nov 20, 2015 9:34 pm

Re: Raspberry Pi Zero W boot with NFS root filesystem

Fri Apr 03, 2020 3:13 am

The copy of the OS I put on the NFS server came from the SD card of the Pi Zero W.

The OS version used by the article author is older than Jessie, but if nothing else, I may try to upgrade to Buster and then do this again.

The problem is that many things could cause the root file system not to mount over NFS. I've tried to check what I can, for example, when I put the SD card back to normal, I can mount the volume I would use as root filesystem on the Pi Zero W so I know it _can_ mount meaning that the export is working and NFS client is working on the Pi Zero W.

It "feels" like something is wrong with cmdline.txt because that's where the setup is described for using NFS. I've had experience using NFS for filesystem root on pi2 and pi3, but those didn't need to use initramfs, so were simpler. I understand what needs to happen, just stymied over why it won't mount the filesystem over NFS.
Last edited by JovianPyx on Fri Apr 03, 2020 3:27 am, edited 1 time in total.

ejolson
Posts: 6045
Joined: Tue Mar 18, 2014 11:47 am

Re: Raspberry Pi Zero W boot with NFS root filesystem

Fri Apr 03, 2020 3:26 am

JovianPyx wrote:
Fri Apr 03, 2020 3:13 am
The copy of the OS I put on the NFS server came from the SD card of the Pi Zero W.

The OS version used by the article author is older than Jessie, but if nothing else, I may try to upgrade to Buster and then do this again.

The problem is that many things could cause the root file system not to mount over NFS. I've tried to check what I can, for example, when I put the SD card back to normal, I can mount the volume I would use as root filesystem on the Pi Zero W so I know it _can_ mount meaning that the export is working and NFS client is working on the Pi Zero W.

It "feels" like something is wrong with cmdline.txt because that's where the setup is described for using NFS. I've had experience using NFS for filesystem root, but on pi2 and pi3, but those didn't need to use initramfs, so were simpler. I understand what needs to happen, just stymied over why it won't mount the filesystem over NFS.
What kind of network are you using?

JovianPyx
Posts: 132
Joined: Fri Nov 20, 2015 9:34 pm

Re: Raspberry Pi Zero W boot with NFS root filesystem

Fri Apr 03, 2020 3:29 am

Ethernet with WiFi, that is, most of the system is wired ethernet, but the Pi Zero W uses wifi to connect. It is all a single class C network.

DarkElvenAngel
Posts: 1085
Joined: Tue Mar 20, 2018 9:53 pm

Re: Raspberry Pi Zero W boot with NFS root filesystem

Fri Apr 03, 2020 4:13 am

JovianPyx wrote:
Fri Apr 03, 2020 3:29 am
Ethernet with WiFi, that is, most of the system is wired ethernet, but the Pi Zero W uses wifi to connect. It is all a single class C network.
Can you verify you're connected to the wifi when trying to boot, are you getting an IP address from the network.

I don't see anything about dhcpcd so are you using a static IP. Can you ping the Pi while it's stuck in NFS boot?


Sorry if I missed something in the guide I skimmed over it.

ejolson
Posts: 6045
Joined: Tue Mar 18, 2014 11:47 am

Re: Raspberry Pi Zero W boot with NFS root filesystem

Fri Apr 03, 2020 4:40 am

JovianPyx wrote:
Fri Apr 03, 2020 3:29 am
Ethernet with WiFi, that is, most of the system is wired ethernet, but the Pi Zero W uses wifi to connect. It is all a single class C network.
I think NFS root over WiFi is uncommon enough that the standard initial RAM filesystem may need to be modified. I would suggest unpacking the one you are trying to use and checking whether wpa_supplicant is even included on it.

An alternative approach would be to boot from an SD card as usual and then use a systemd chroot environment to switch to an NFS mount. While I'm pretty sure this could be done, I unfortunately do not know enough about systemd to say how. If a chroot environment sounds like how you want to proceed, maybe indicate here and someone else could point out how.

DarkElvenAngel
Posts: 1085
Joined: Tue Mar 20, 2018 9:53 pm

Re: Raspberry Pi Zero W boot with NFS root filesystem

Fri Apr 03, 2020 5:13 am

ejolson wrote:
Fri Apr 03, 2020 4:40 am
JovianPyx wrote:
Fri Apr 03, 2020 3:29 am
Ethernet with WiFi, that is, most of the system is wired ethernet, but the Pi Zero W uses wifi to connect. It is all a single class C network.
I think NFS root over WiFi is uncommon enough that the standard initial RAM filesystem may need to be modified. I would suggest unpacking the one you are trying to use and checking whether wpa_supplicant is even included on it.

An alternative approach would be to boot from an SD card as usual and then use a systemd chroot environment to switch to an NFS mount. While I'm pretty sure this could be done, I unfortunately do not know enough about systemd to say how. If a chroot environment sounds like how you want to proceed, maybe indicate here and someone else could point out how.
The link posted has instructions on how to add wpa_supplicant I was going to mention that as well. I believe that there are steps missing from the guide however and it points to outdated information.

User avatar
dickon
Posts: 1824
Joined: Sun Dec 09, 2012 3:54 pm
Location: Home, just outside Reading

Re: Raspberry Pi Zero W boot with NFS root filesystem

Fri Apr 03, 2020 8:01 am

JovianPyx wrote:
Fri Apr 03, 2020 2:15 am
Booting the pi zero, it prints messages about USB enumeration and stops at about 30 seconds waits until about 470 seconds and then says:
[...]
If you have 'quiet' on the kernel commandline, remove it now. It's a hopeless default and makes debugging this sort of thing very difficult.

JovianPyx
Posts: 132
Joined: Fri Nov 20, 2015 9:34 pm

Re: Raspberry Pi Zero W boot with NFS root filesystem

Fri Apr 03, 2020 12:32 pm

I have tested using dhcpcd.conf to select static and I've tested using dhcp served IP addresses. When I boot the SD card normally, it connects to the network regardless.

I don't fully understand the initramfs processes described in the article. All the instructions tell to do is to run the script against a file in /boot. The script creates a file that is used during boot because of a line in config.txt. The instructions describe no modification of source, no compiles. What I get from the article is that this worked for the author.

One thing I did notice is that when I boot the SD card (not attempting NFS root filesystem), it takes a very long time before I can ping the Pi Zero W.

Ping does not respond while it is waiting to mount the NFS volume. My assumption is that it is not getting an IP address from DHCP.

I do not have a 'quiet' parameter in cmdline.txt

I don't know exactly what is going on in initramfs, but my understand (which could be flawed) is that initramfs uses the wpa_supplicant that is on the SDcard.

One of the scripts has two sleep commands in it, I'm wondering if the sleeps are too short.

JovianPyx
Posts: 132
Joined: Fri Nov 20, 2015 9:34 pm

Re: Raspberry Pi Zero W boot with NFS root filesystem

Fri Apr 03, 2020 1:47 pm

I've just noticed a glaring error in the article's wifi script source code. The script attempts to execute:

ipconfig wlan0 &

ipconfig is a Windows command.

I changed ipconfig to ifconfig, but the results were the same, it still won't mount the NFS root filesystem and it's address never becomes pingable (which I believe means that it doesn't have an IP address).

JovianPyx
Posts: 132
Joined: Fri Nov 20, 2015 9:34 pm

Re: Raspberry Pi Zero W boot with NFS root filesystem

Fri Apr 03, 2020 2:06 pm

What would be most helpful is to know how to set up initramfs so that the machine boots to a prompt with just the ramfs. Perhaps then I can try to troubleshoot why the NFS volume won't mount.

DarkElvenAngel
Posts: 1085
Joined: Tue Mar 20, 2018 9:53 pm

Re: Raspberry Pi Zero W boot with NFS root filesystem

Fri Apr 03, 2020 3:03 pm

To answer one big question you have first is how the initramfs works.

To put it very simply it's a very small version of your rootfs with just enough tools to boot your computer to the full rootfs. The initramfs is compressed into a file, on boot it's decompressed to ram and runs. So the idea is that when you turn on the pi it will load the kernel then unpack the initramfs to memory and that should initialize the wireless module fetch an IP address or have one assigned statically, mount the NFS as the rootfs and the switch_root to that filesystem and finally complete boot.

The problem I see is that the initramfs is getting hung up somewhere in these steps and I suspect it's in the wireless accessing the network.

I have looked at the guide you posted and it has you unpack and install some wpa_supplicant did you do that step??

You can access the console in the initramfs by adding break to the /boot/cmdline.txt

This will let you see if you have an ip address and if the network is reachable. If you need more help with stuffing tools into your initramfs I can help you with that. Just remember that the rescue terminal is very limited and if you find something you need is missing write that down.
Last edited by DarkElvenAngel on Fri Apr 03, 2020 3:04 pm, edited 1 time in total.

User avatar
dickon
Posts: 1824
Joined: Sun Dec 09, 2012 3:54 pm
Location: Home, just outside Reading

Re: Raspberry Pi Zero W boot with NFS root filesystem

Fri Apr 03, 2020 3:03 pm

I would change all the '/bin/sh' script headers to '/bin/sh -x' (note the space), which prints every command the shell attempts to execute to console. That will show you where it is failing. You can also pepper the things with 'busybox -i', which will drop it to an interactive shell that should allow you to check the state each time.

DarkElvenAngel
Posts: 1085
Joined: Tue Mar 20, 2018 9:53 pm

Re: Raspberry Pi Zero W boot with NFS root filesystem

Fri Apr 03, 2020 3:07 pm

dickon wrote:
Fri Apr 03, 2020 3:03 pm
I would change all the '/bin/sh' script headers to '/bin/sh -x' (note the space), which prints every command the shell attempts to execute to console. That will show you where it is failing. You can also pepper the things with 'busybox -i', which will drop it to an interactive shell that should allow you to check the state each time.
In order for this suggestion to work in case you are not aware you must rebuild the initramfs with

Code: Select all

update-initramfs -u -k $(uname -r)

JovianPyx
Posts: 132
Joined: Fri Nov 20, 2015 9:34 pm

Re: Raspberry Pi Zero W boot with NFS root filesystem

Fri Apr 03, 2020 3:23 pm

Thank you DarkElvenAngel and dickon, this is what I need, a way to spit out information about what is happening.

I will work with this and post whatever I find.

JovianPyx
Posts: 132
Joined: Fri Nov 20, 2015 9:34 pm

Re: Raspberry Pi Zero W boot with NFS root filesystem

Fri Apr 03, 2020 3:27 pm

DarkElvenAngel wrote:
Fri Apr 03, 2020 3:07 pm
dickon wrote:
Fri Apr 03, 2020 3:03 pm
I would change all the '/bin/sh' script headers to '/bin/sh -x' (note the space), which prints every command the shell attempts to execute to console. That will show you where it is failing. You can also pepper the things with 'busybox -i', which will drop it to an interactive shell that should allow you to check the state each time.
In order for this suggestion to work in case you are not aware you must rebuild the initramfs with

Code: Select all

update-initramfs -u -k $(uname -r)
I see. So any time I change those scripts (which I've been doing a lot of...) I need to run the update-initramfs script. Which means that I've wasted a lot of time - because I did not know this - and thank you VERY much for pointing that out.

User avatar
dickon
Posts: 1824
Joined: Sun Dec 09, 2012 3:54 pm
Location: Home, just outside Reading

Re: Raspberry Pi Zero W boot with NFS root filesystem

Fri Apr 03, 2020 3:36 pm

Oh. Oops. I'd've said so, but I assumed you knew that.

JovianPyx
Posts: 132
Joined: Fri Nov 20, 2015 9:34 pm

Re: Raspberry Pi Zero W boot with NFS root filesystem

Fri Apr 03, 2020 4:12 pm

Getting more lost it seems.

I tried to use the update command you gave, but it failed with a zillion messages that went screaming up. So I tried to use the create (-c instead of -u) and it complained that the target file existed. I deleted the file and tried again, but now it says the volume is full (/boot). However, I've looked in /boot and see that yes, it's very close to full, but I can't see anything to delete. The backup file (if it exists) should end in bak, but no such file exists.

So I'm clear - I'm doing the update-initramfs on the Pi Zero W with the SD card set to boot normally.

The first time I did it, the create (-c) worked and created the required file in /boot

Any clue as to what I should look for to stop this error?

User avatar
dickon
Posts: 1824
Joined: Sun Dec 09, 2012 3:54 pm
Location: Home, just outside Reading

Re: Raspberry Pi Zero W boot with NFS root filesystem

Fri Apr 03, 2020 4:23 pm

Delete the file first and recreate it? You might also have some luck changing the compression algorithm; if you're using gzip, try xz or the like.

DarkElvenAngel
Posts: 1085
Joined: Tue Mar 20, 2018 9:53 pm

Re: Raspberry Pi Zero W boot with NFS root filesystem

Fri Apr 03, 2020 4:33 pm

JovianPyx wrote: Getting more lost it seems.

I tried to use the update command you gave, but it failed with a zillion messages that went screaming up. So I tried to use the create (-c instead of -u) and it complained that the target file existed. I deleted the file and tried again, but now it says the volume is full (/boot). However, I've looked in /boot and see that yes, it's very close to full, but I can't see anything to delete. The backup file (if it exists) should end in bak, but no such file exists.

So I'm clear - I'm doing the update-initramfs on the Pi Zero W with the SD card set to boot normally.

The first time I did it, the create (-c) worked and created the required file in /boot

Any clue as to what I should look for to stop this error?
This is an issue with the older versions of Raspbian the boot partition is small you could delete the kernel images for the pi 3.

JovianPyx
Posts: 132
Joined: Fri Nov 20, 2015 9:34 pm

Re: Raspberry Pi Zero W boot with NFS root filesystem

Fri Apr 03, 2020 4:44 pm

heh - the problem turned out to be no space in /var/tmp, but I still get all kinds of errors ending with

update-initramfs: failed for /boot/initrd.img-4.9.35+ with 127

JovianPyx
Posts: 132
Joined: Fri Nov 20, 2015 9:34 pm

Re: Raspberry Pi Zero W boot with NFS root filesystem

Fri Apr 03, 2020 4:50 pm

The device full problem was that /var/tmp (tmpfs) was full. Deleted the files there (all related to the run of update-initramfs

It has now created the file.

ejolson
Posts: 6045
Joined: Tue Mar 18, 2014 11:47 am

Re: Raspberry Pi Zero W boot with NFS root filesystem

Fri Apr 03, 2020 4:57 pm

JovianPyx wrote:
Fri Apr 03, 2020 1:47 pm
I've just noticed a glaring error in the article's wifi script source code. The script attempts to execute:

ipconfig wlan0 &

ipconfig is a Windows command.

I changed ipconfig to ifconfig, but the results were the same, it still won't mount the NFS root filesystem and it's address never becomes pingable (which I believe means that it doesn't have an IP address).
You could be right. Be careful because the Debian initial RAM filesystem (as well as Raspbian when configured) uses busybox and some other weird stuff for system commands because glibc can no longer be linked as a static library.
Last edited by ejolson on Fri Apr 03, 2020 5:11 pm, edited 2 times in total.

Return to “Advanced users”