Robi_G1
Posts: 18
Joined: Wed May 15, 2019 7:23 am

Problems exporting an NFS share

Thu Jul 18, 2019 8:26 am

I've got a project where several pi's run NFS servers and export a directory (one each).
Another pi then mounts these directories, and exports them on its own NFS server.
A laptop then mounts these over direct wifi connection to the 'head' pi. The idea being that the laptop only has to make one connection to access files on all the 'sub' pi's.
To keep things simple to start with, I've only connected one 'sub' pi to the head pi.
The head pi mounts this share fine, and both boot without issue.
When I first install and start the NFS server on the head pi, it successfully exports the slave's shared directory, and everything's accessible via laptop.
But having done this, when I restart the pi's (unplug the power, the pi's are networked over a USB hub, and powering off the head pi powers off everything), the head pi won't boot again.

I assume that the NFS server on the head pi is attempting to export the sub-pi's directory before the sub-pi has booted, and getting stuck.
Is there a way to either re-order how things start at boot, or to get the head pi to wait for the sub-pi's export to become available before it starts its nfs server?
Failing that, is there a way to get some sort of diagnostic output so I can be absolutely sure of what's going on?

Many thanks,
Rob

epoch1970
Posts: 3555
Joined: Thu May 05, 2016 9:33 am
Location: Paris, France

Re: Problems exporting an NFS share

Thu Jul 18, 2019 9:19 am

Reexporting NFS is a bad idea. Known to cause loss of hair and data.
NFS was designed for the LAN. The network is expected to be reliable (and fast), the server is expected to be reliable. The only volatile part the design supports is clients.

In complex NFS environments, clients would use an automounter to keep their individual configuration tidy and generic.
Systemd comes with a sort of automount feature, the real deal is called autofs5 on Linux.
"S'il n'y a pas de solution, c'est qu'il n'y a pas de problème." Les Shadoks, J. Rouxel

Robi_G1
Posts: 18
Joined: Wed May 15, 2019 7:23 am

Re: Problems exporting an NFS share

Thu Jul 18, 2019 3:59 pm

After much faffing, I ended up with a working but incredibly unreliable solution that would (and did) fail more than 1/2 the time the system was reset.
Workarounds involved changes to /etc/exports (completely futile), changes to /etc/fstab to try to tell systemd to wait (each attempt just changed the way the system failed), and changes to /lib/systemd/system/nfs-server.service to make it require the 'home-pi-slave.mount' and a network connection (which had the most success, but the first time I got this setup working these settings were unchanged).
Occasionally the thing would start working, but then on a reboot would break somewhere in the chain, either a share would 'mount', but contain no files, the master or laptop would hang on boot, or connections would time out. Not a happy time.

Have settled for the NFS shares to the head pi (which are reliable), and am then using paramiko's sftp module in python to access the head pi (was previously running lovely 'cp' commands). I have an extra 1/4s lag in transmissions (from 1s to 1.25s) which is a shame, but at least this way works every time.

swampdog
Posts: 230
Joined: Fri Dec 04, 2015 11:22 am

Re: Problems exporting an NFS share

Sat Jul 20, 2019 5:59 pm

Have you considered iscsi for the "NFS server" pi's? Each presents a disk to the "head" pi and only NFS export from the "head".

It would be a pain to attempt. The obvious caveat is the (now) "ISCSI target" (aka "NFS server") pi's would have to be guaranteed to be up and each would have to be up before the "head" pi comes up - and go back down in reverse order.

Fwiw, I've done similar in the non-pi world but for samba as a stop-gap for lack of disk space. Rough description so you can decide if it's worth the bother..

Pair of HP (centos 6) servers, raid6, "sdhp0" and "sdhp1". There's a dns CNAME "smb" for "sdhp1" which is the primary samba server. Ran out of space and no time to put bigger disks into "smb". Both these machines use LVM (logical volume manager). Cheap and dirty solution was to buy a pair of esata caddies and plug one each into sdhp0 and sdhp1. On each machine, create new VG (volume group) "vgES" and on each a LV (logical volume) "T2L1SMB" (target 2, lun 1, smb). Have each of sdhp0,sdhp1 present these as iscsi targets.

Create a second samba server under KVM (kernel virtual machine) as a VM (virtual machine) called "smb1". Have smb1 initiate those two luns which it can then format with a filesystem accordingly. In my case I presented them both a a single mirrored LV "lvsmb" in a VG (volume group) "vgSMB". At this point the samba server on smb1 would be replaced by your NFS exports and if you're not bothered about mirroring just add each disk as another PV (physical volume).

It, "smb1", is considerably slower than "smb" because the data has to go flying about across the network multiple times. You will also need static IP addresses, obviously. There's a DNS/DHCP server I have full control over so I can force fixed IP onto machines by mac address.

I did use sdhp0 to present an iscsi device to a pi and got it to boot off of it (just /boot on its sdcard) but that wasn't worth the effort. Firstly, had to build a custom kernel. It was quite a while ago so maybe that's changed now. The second problem was much worse. When the pi booted it got an IP off DHCP (the iscsi device) then later on in the boot it got another when it brought up the networking proper. The second IP would re-lease accordingly but the first would get dropped by DHCP after the default period. End result the "iscsi" IP would get picked up by another client the pi would be borked. If you're just presenting a data iscsi disk and can bring it up after networking this shouldn't be a problem.

Simples! (ahem, where's my coat?) :-)

Robi_G1
Posts: 18
Joined: Wed May 15, 2019 7:23 am

Re: Problems exporting an NFS share

Mon Jul 22, 2019 9:25 am

swampdog wrote:
Sat Jul 20, 2019 5:59 pm
Have you considered iscsi for the "NFS server" pi's? Each presents a disk to the "head" pi and only NFS export from the "head".
Haven't played with iscsi before, and while what you've suggested sounds just like the kind of simultaneously elegant and messy thing I love, it's impossible to guarantee that the slave pi's would be up before the head pi. If anything, the head is almost guaranteed to boot first, because it powering on is what triggers the power on a usb hub that allows the slaves to boot.

With some messing about I might be able to fudge something to make the head pi wait in some way, but there's other, higher priority things on this project, and I have something that's currently working well enough. If there's time left and I get around to trying it out, I'll report back with successes / failures.

epoch1970
Posts: 3555
Joined: Thu May 05, 2016 9:33 am
Location: Paris, France

Re: Problems exporting an NFS share

Mon Jul 22, 2019 10:08 am

iSCSI demands perfect network and endpoints reliability. Much worse than NFS from this standpoint.
"S'il n'y a pas de solution, c'est qu'il n'y a pas de problème." Les Shadoks, J. Rouxel

Return to “Advanced users”