NFS on Buster RPi4

[Daddyfix]

Hello fellow Raspians

I am having trouble getting an NFS server working on my new RPi4 (buster) Debian Lite.

Hardware:
/boot -> SD card
rootfs -> USB Flash drive

How I installed NFS Server. This installs packages nfs-common and rpcbind

Code: Select all

sudo apt-get install nfs-kernel-server

During the install I get a DEPANDANCY message but I can find a solution

Code: Select all

apt install nfs-kernel-server -y
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  nfs-common rpcbind
Suggested packages:
  open-iscsi watchdog
The following NEW packages will be installed:
  nfs-common nfs-kernel-server rpcbind
0 upgraded, 3 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/351 kB of archives.
After this operation, 1,017 kB of additional disk space will be used.
Selecting previously unselected package rpcbind.
(Reading database ... 46793 files and directories currently installed.)
Preparing to unpack .../rpcbind_1.2.5-0.3_armhf.deb ...
Unpacking rpcbind (1.2.5-0.3) ...
Selecting previously unselected package nfs-common.
Preparing to unpack .../nfs-common_1%3a1.3.4-2.5_armhf.deb ...
Unpacking nfs-common (1:1.3.4-2.5) ...
Selecting previously unselected package nfs-kernel-server.
Preparing to unpack .../nfs-kernel-server_1%3a1.3.4-2.5_armhf.deb ...
Unpacking nfs-kernel-server (1:1.3.4-2.5) ...
Setting up rpcbind (1.2.5-0.3) ...
Created symlink /etc/systemd/system/portmap → /etc/systemd/system/rpcbind.service.
Created symlink /etc/systemd/system/sysinit.target.wants/rpcbind.service → /etc/systemd/system/rpcbind.service.
Created symlink /etc/systemd/system/sockets.target.wants/rpcbind.socket → /lib/systemd/system/rpcbind.socket.
Setting up nfs-common (1:1.3.4-2.5) ...

Creating config file /etc/idmapd.conf with new version
Created symlink /etc/systemd/system/multi-user.target.wants/nfs-client.target → /lib/systemd/system/nfs-client.target.
Created symlink /etc/systemd/system/remote-fs.target.wants/nfs-client.target → /lib/systemd/system/nfs-client.target.
nfs-utils.service is a disabled or a static unit, not starting it.
Setting up nfs-kernel-server (1:1.3.4-2.5) ...
Created symlink /etc/systemd/system/multi-user.target.wants/nfs-server.service → /lib/systemd/system/nfs-server.service.
A dependency job for nfs-server.service failed. See 'journalctl -xe' for details.

Creating config file /etc/exports with new version

Creating config file /etc/default/nfs-kernel-server with new version
A dependency job for nfs-server.service failed. See 'journalctl -xe' for details.
invoke-rc.d: initscript nfs-kernel-server, action "start" failed.
● nfs-server.service - NFS server and services
   Loaded: loaded (/lib/systemd/system/nfs-server.service; enabled; vendor preset: enabled)
   Active: inactive (dead)

Jul 21 20:46:10 pimedia systemd[1]: Dependency failed for NFS server and services.
Jul 21 20:46:10 pimedia systemd[1]: nfs-server.service: Job nfs-server.service/start failed with result 'dependency'.
Jul 21 20:46:14 pimedia systemd[1]: Dependency failed for NFS server and services.
Jul 21 20:46:14 pimedia systemd[1]: nfs-server.service: Job nfs-server.service/start failed with result 'dependency'.
Failed to start nfs-kernel-server, ignoring.
Processing triggers for man-db (2.8.5-2) ...
Processing triggers for systemd (241-5+rpi1) ...

Restarting the kernel server

Code: Select all

systemctl restart nfs-kernel-server.service 
A dependency job for nfs-server.service failed. See 'journalctl -xe' for details.
root@pimedia:~# journalctl -xe
-- A start job for unit nfs-mountd.service has finished with a failure.
-- 
-- The job identifier is 358 and the job result is dependency.
Jul 21 20:48:10 pimedia systemd[1]: nfs-mountd.service: Job nfs-mountd.service/start fai
Jul 21 20:48:10 pimedia systemd[1]: Dependency failed for NFSv4 ID-name mapping service.
-- Subject: A start job for unit nfs-idmapd.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- A start job for unit nfs-idmapd.service has finished with a failure.
-- 
-- The job identifier is 373 and the job result is dependency.
Jul 21 20:48:10 pimedia systemd[1]: nfs-idmapd.service: Job nfs-idmapd.service/start fai
Jul 21 20:48:10 pimedia systemd[1]: nfs-server.service: Job nfs-server.service/start fai
Jul 21 20:48:10 pimedia systemd[1]: nfs-config.service: Succeeded.
-- Subject: Unit succeeded
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- The unit nfs-config.service has successfully entered the 'dead' state.
Jul 21 20:48:10 pimedia systemd[1]: Started Preprocess NFS configuration.
-- Subject: A start job for unit nfs-config.service has finished successfully
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- A start job for unit nfs-config.service has finished successfully.
-- 
-- The job identifier is 371.
Jul 21 20:48:10 pimedia systemd[1]: Condition check resulted in RPC security service for
-- Subject: A start job for unit rpc-svcgssd.service has finished successfully
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- A start job for unit rpc-svcgssd.service has finished successfully.
-- 
-- The job identifier is 368.
Jul 21 20:48:10 pimedia systemd[1]: Condition check resulted in RPC security service for
-- Subject: A start job for unit rpc-gssd.service has finished successfully
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- A start job for unit rpc-gssd.service has finished successfully.
-- 
-- The job identifier is 372.

Any insight would be greatly appreciated.

I think the kernel does not have a working nfsd module yet?

Does anyone else have this issue?

[Milliways]

Works for me.

Have you created any exports?

[Daddyfix]

Yes I have? Hmmm....

Code: Select all

/mnt/usb/www 192.168.1.0/24(rw,no_root_squash,crossmnt,no_subtree_check)

Could I be having a problem cause im am using and SD for boot and Flash Drive for rootfs?

[Daddyfix]

This is what happens when I try and start nfs-kernel-server.service

Code: Select all

$ systemctl restart nfs-kernel-server.service 
A dependency job for nfs-server.service failed. See 'journalctl -xe' for details.

Status of fail

Code: Select all

root@pimedia:~# systemctl status nfs-kernel-server.service 
● nfs-server.service - NFS server and services
   Loaded: loaded (/lib/systemd/system/nfs-server.service; enabled; vendor preset: enabl
   Active: inactive (dead)

Jul 22 11:39:50 pimedia systemd[1]: Dependency failed for NFS server and services.
Jul 22 11:39:50 pimedia systemd[1]: nfs-server.service: Job nfs-server.service/start failed

Journalctl -xe reports a nfs-mountd.service fail with dependancy. I am very confused that there is no other hints to what might be going on

Code: Select all

root@pimedia:~# journalctl -xe
-- The job identifier is 2655 and the job result is dependency.
Jul 22 11:39:50 pimedia systemd[1]: Dependency failed for NFS Mount Daemon.
-- Subject: A start job for unit nfs-mountd.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- A start job for unit nfs-mountd.service has finished with a failure.
-- 
-- The job identifier is 2656 and the job result is dependency.

Jul 22 11:39:50 pimedia systemd[1]: nfs-mountd.service: Job nfs-mountd.service/start fai
Jul 22 11:39:50 pimedia systemd[1]: Dependency failed for NFSv4 ID-name mapping service.
-- Subject: A start job for unit nfs-idmapd.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- A start job for unit nfs-idmapd.service has finished with a failure.
-- 
-- The job identifier is 2671 and the job result is dependency.
Jul 22 11:39:50 pimedia systemd[1]: nfs-idmapd.service: Job nfs-idmapd.service/start fai
Jul 22 11:39:50 pimedia systemd[1]: nfs-server.service: Job nfs-server.service/start fai
Jul 22 11:39:50 pimedia systemd[1]: Condition check resulted in RPC security service for
-- Subject: A start job for unit rpc-gssd.service has finished successfully
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- A start job for unit rpc-gssd.service has finished successfully.
-- 
-- The job identifier is 2670.
Jul 22 11:39:50 pimedia systemd[1]: Condition check resulted in RPC security service for
-- Subject: A start job for unit rpc-svcgssd.service has finished successfully
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- A start job for unit rpc-svcgssd.service has finished successfully.
-- 
-- The job identifier is 2666.

[dickon]

There's your problem: idmapd isn't running. That's part of nfs-common, which should already be installed.

I'd check why idmapd isn't running -- it's probably not been configured -- or disable nfsv4 (it's overkill; nfsv3 is perfectly fine for most deployments). Don't ask me how, as you're using systemd, which gets replaced with sysvinit-core on every machine I have anything to do with. --purge, just to show my contempt for it.

[Daddyfix]

idmapd return the same dependancy issues. I keep getting the same responce from all the services.DEPENDANCY

I even remove all my /etc/exports entries.

I think it may be that kernel does not have the nfsd module.

Does ayone else have this working on RPi4 with the lastest Debian Buster Lite?

Hmmm. baffling!

[Daddyfix]

I just checked /lib/modules/uname-r/kernel/drivers

Code: Select all

$ find /lib/modules/ -name *nfs*
 /lib/modules/4.19.58+/kernel/fs/nfsd
 /lib/modules/4.19.58+/kernel/fs/nfsd/nfsd.ko
 /lib/modules/4.19.58+/kernel/fs/nfs
/lib/modules/4.19.58+/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko
/lib/modules/4.19.58-v7l+/kernel/fs/nfsd
/lib/modules/4.19.58-v7l+/kernel/fs/nfsd/nfsd.ko
/lib/modules/4.19.58-v7l+/kernel/fs/nfs
/lib/modules/4.19.58-v7l+/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko
/lib/modules/4.19.58-v7+/kernel/fs/nfsd
/lib/modules/4.19.58-v7+/kernel/fs/nfsd/nfsd.ko
/lib/modules/4.19.58-v7+/kernel/fs/nfs
/lib/modules/4.19.58-v7+/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko

I also added nfsd to /etc/modules-load.d/modules.conf

Still not working.... baffled

I think I might need Dougies help!

[Ernst]

You must be holding doing it wrong.

I will use a 400 line script to migrate my web/nfs/smb server to buster when the Pi4/2 arrives,
these two lines are used to install nfs:

Code: Select all

   sudo apt-get install nfs-common nfs-kernel-server -y
   echo /nfs          *\(rw,sync,no_subtree_check,no_root_squash\) | sudo tee >> /etc/exports

[dickon]

idmapd return the same dependancy issues. I keep getting the same responce from all the services.DEPENDANCY

Well, that's systemd for you...

I even remove all my /etc/exports entries.

Won't help.

I think it may be that kernel does not have the nfsd module.

Doubtful. Try

Code: Select all

lsmod | grep nfs

and you should see something like

Code: Select all

root@newdesktop:~# lsmod | grep nfs
nfsv3                  53248  2
nfsd                  430080  13
auth_rpcgss            73728  1 nfsd
nfs_acl                16384  2 nfsd,nfsv3
nfs                   311296  3 nfsv3
lockd                 118784  3 nfsd,nfsv3,nfs
grace                  16384  2 nfsd,lockd
fscache               389120  1 nfs
sunrpc                425984  30 nfsd,auth_rpcgss,lockd,nfsv3,nfs_acl,nfs

[Daddyfix]

Ah hah!! Were on to it dickon!

Code: Select all

root@pimedia:~# lsmod |grep nfs
root@pimedia:~# lsmod
Module                  Size  Used by
root@pimedia:~#

Im gonna try and reinstall raspian lite buster and see if i get the same results

[dickon]

Code: Select all

rpcinfo -p

is also your friend. You should see something like:

Code: Select all

root@rtr-dickonh:~# rpcinfo -p
   program vers proto   port  service
    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    2   tcp    111  portmapper
    100000    4   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    2   udp    111  portmapper
    100024    1   udp  42821  status
    100024    1   tcp  55185  status
    100003    3   tcp   2049  nfs
    100003    4   tcp   2049  nfs
    100227    3   tcp   2049
    100003    3   udp   2049  nfs
    100003    4   udp   2049  nfs
    100227    3   udp   2049
    100021    1   udp  40243  nlockmgr
    100021    3   udp  40243  nlockmgr
    100021    4   udp  40243  nlockmgr
    100021    1   tcp  33857  nlockmgr
    100021    3   tcp  33857  nlockmgr
    100021    4   tcp  33857  nlockmgr
    100005    1   udp  50052  mountd
    100005    1   tcp  42437  mountd
    100005    2   udp  33191  mountd
    100005    2   tcp  52775  mountd
    100005    3   udp  48352  mountd
    100005    3   tcp  40081  mountd

Try a

Code: Select all

modprobe nfsv3

and see what that gets you.

[Daddyfix]

No nfs in rpcinfo

Code: Select all

# rpcinfo -p
   program vers proto   port  service
    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    2   tcp    111  portmapper
    100000    4   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    2   udp    111  portmapper
    100024    1   udp  58215  status
    100024    1   tcp  39509  status
    100021    1   udp  40834  nlockmgr
    100021    3   udp  40834  nlockmgr
    100021    4   udp  40834  nlockmgr
    100021    1   tcp  43359  nlockmgr
    100021    3   tcp  43359  nlockmgr
    100021    4   tcp  43359  nlockmgr
    

Modprobe also....

Code: Select all

oot@pimedia:~# modprobe nfsv3
modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.19.50-v7l+/modules.dep.bin'
modprobe: FATAL: Module nfsv3 not found in directory /lib/modules/4.19.50-v7l+
root@pimedia:~# modprobe nfsv4
modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.19.50-v7l+/modules.dep.bin'
modprobe: FATAL: Module nfsv4 not found in directory /lib/modules/4.19.50-v7l+

But when I do....

Code: Select all

# grep -r 'nfsv3' /lib/modules/4.19.58-v7l+/
/lib/modules/4.19.58-v7l+/modules.builtin:kernel/fs/nfs/nfsv3.ko

????

[dickon]

4.19.50-v7l+ != 4.19.58-v7l+

You've got a mismatch between kernel version and installed kernel modules. I'd

Code: Select all

apt-get update; apt-get install --reinstall raspberrypi-kernel raspberrypi-kernel-headers

and reboot, if I were you. Don't bother with the headers if you're not going to install any out-of-tree modules; they just take an age to unpack.

[Daddyfix]

Dickon,

Firstly, thanks alot for your help on this!

Perhaps when I rsync'd from the MicroSd card to the USB Flash Drive, the modules didn't get copied.

[Daddyfix]

Same results after reboot. Mismatch on versions

Code: Select all

root@pimedia:~# modprobe nfsv4
modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.19.50-v7l+/modules.dep.bin'
modprobe: FATAL: Module nfsv4 not found in directory /lib/modules/4.19.50-v7l+

[Daddyfix]

I think there is a screw up with my SD and USB

Code: Select all

root@pimedia:~# cd /lib/modules
root@pimedia:/lib/modules# l
total 12
drwxr-xr-x 3 root root 4096 Jul 23 10:36 4.19.58-v7l+
drwxr-xr-x 3 root root 4096 Jul 23 10:36 4.19.58-v7+
drwxr-xr-x 3 root root 4096 Jul 23 10:36 4.19.58+

See PARTUUID for mmcblk0p2 and sda2....

Code: Select all

root@pimedia:/lib/modules# blkid
/dev/mmcblk0p1: LABEL_FATBOOT="boot" LABEL="boot" UUID="B6BB-0F0E" TYPE="vfat" PARTUUID="932d16d8-01"
/dev/mmcblk0p2: LABEL="rootfs" UUID="638417fb-7220-47b1-883c-e6fee02f51ac" TYPE="ext4" PARTUUID="932d16d8-02"
/dev/sda1: LABEL_FATBOOT="boot" LABEL="boot" UUID="B6BB-0F0E" TYPE="vfat" PARTUUID="932d16d8-01"
/dev/sda2: LABEL="rootfs" UUID="638417fb-7220-47b1-883c-e6fee02f51ac" TYPE="ext4" PARTUUID="932d16d8-02"
/dev/mmcblk0: PTUUID="932d16d8" PTTYPE="dos"
/dev/sdb1: LABEL="Optware" UUID="e680fa02-e8a6-4dae-8dbc-40deaca8c774" SEC_TYPE="ext2" TYPE="ext3" PARTUUID="000e5e4f-01"
/dev/sdb2: LABEL="Swapfile" UUID="d71e2253-1d91-4315-8fec-be3308ba8fef" TYPE="swap" PARTUUID="000e5e4f-02"
/dev/sdb3: LABEL="Data" UUID="b1dddce2-8b2c-446b-9092-5975d32a423c" TYPE="ext3" PARTUUID="000e5e4f-03"

Mounts

Code: Select all

/dev/mmcblk0p2 on / type ext4 (rw,noatime)
devtmpfs on /dev type devtmpfs (rw,relatime,size=886284k,nr_inodes=122119,mode=755)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,relatime)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/net_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=36,pgrp=1,timeout=0,minproto=5,maxproto=5,direct)
sunrpc on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
configfs on /sys/kernel/config type configfs (rw,relatime)

[Daddyfix]

/boot/cmdline.txt

Code: Select all

# cat /boot/cmdline.txt
dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=PARTUUID=932d16d8-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait

[dickon]

Hmm.

Code: Select all

root@pi4:~# dpkg -S /lib/modules/4.19.50-v7l+/kernel/fs/nfsd/nfsd.ko
raspberrypi-kernel: /lib/modules/4.19.50-v7l+/kernel/fs/nfsd/nfsd.ko

so a --reinstall really should have replaced those. Where did you get 4.19.58-v7l+ from? rpi-update?

[Daddyfix]

Yes. I did a rpi-update earlier this week in attempt to install the latest kernel (with modules) as I thought that might be the problem

Is there a way to reverse the kernel versions to an earlier state?

[Ernst]

Yes. I did a rpi-update earlier this week in attempt to install the latest kernel (with modules) as I thought that might be the problem

Is there a way to reverse the kernel versions to an earlier state?

FYI: I did not use rpi-update, only apt-upgrade, and nfs is working as expected on my web/smb/nfs server that I migrated to buster just now. The migration was done using the latest raspbian lite.

I suggest you take a fresh card and start again, that will be less effort.

[Daddyfix]

Dickon,

I agree totally.

But thanks again for all your help. I learned alot about kernel modules.

----

[dickon]

Oh.

Two different partitions with the same UUID. Yes, that might cause trouble.

Check which one is mounted as /, then mount the other one somewhere else, and check it's /lib/modules for the missing modules. If that's the problem, I leave you to sort out re-UUIDing the offending filesystem and rebooting. It's entirely possible that the kernel is choosing one at more or less random on boot, depending on which one starts up first, or the order the modules are linked into the kernel, or all sorts of random other race conditions, and you've managed to reinstall the kernel into the other one.

A --reinstall right now, on the 'broken' filesystem, might sort out the problem for now, but, equally, is a recipe for extremely annoying trouble down the line. This is what ZFS makes so, so much better...

[Daddyfix]

Dickon,

You were right. Once I reformatted both the Mirco SD and the USB. Copied the identical Raspian Buster 2019-07-?? image to both mediums and everything for NFS server worked like it should.

Thanks again for all you help on this.