ejolson
Posts: 5137
Joined: Tue Mar 18, 2014 11:47 am

NFSv4 with user-level Kerberos

Thu Jan 16, 2020 1:50 am

Has anyone here set up a Pi 4B as a NFSv4 file server using Kerberos for user level authentication? I'm about to embark upon such an adventure (maybe) and was wondering if anyone else has already documented what needs to be done and how or at least knows some reliable sources of information.

ejolson
Posts: 5137
Joined: Tue Mar 18, 2014 11:47 am

Re: NFSv4 with user-level Kerberos

Mon Jan 20, 2020 11:11 pm

ejolson wrote:
Thu Jan 16, 2020 1:50 am
Has anyone here set up a Pi 4B as a NFSv4 file server using Kerberos for user level authentication? I'm about to embark upon such an adventure (maybe) and was wondering if anyone else has already documented what needs to be done and how or at least knows some reliable sources of information.
I've made some progress. Everything appears easy to get wrong--I clearly had multiple errors in my configuration. I'm now trying to isolate which changes were necessary for things to work.

ejolson
Posts: 5137
Joined: Tue Mar 18, 2014 11:47 am

Re: NFSv4 with user-level Kerberos

Wed Jan 22, 2020 5:24 pm

While marveling at how the key-distribution center, the administration server, the network file system, the lightweight directory access protocol, the identity mapping server and servers implementing various versions of the generic security service all work (or fail to work) together, it occurred to me that PiServer

https://www.raspberrypi.org/blog/piserver/

might be using a fully kerberised version of NFSv4 to mount home directories.

Does anyone know?

incognitum
Posts: 476
Joined: Tue Oct 30, 2018 3:34 pm

Re: NFSv4 with user-level Kerberos

Wed Jan 22, 2020 9:15 pm

ejolson wrote:
Wed Jan 22, 2020 5:24 pm
https://www.raspberrypi.org/blog/piserver/

might be using a fully kerberised version of NFSv4 to mount home directories.

Does anyone know?
Negative.
It mounts home directories through sshfs (which uses ssh's sftp protocol).

Problem with kerberos is that it pretty much requires static IPs, forward and reverse DNS entries for all hosts including clients.
And that does not go along with Piserver's unique selling point that it can work without making any chances to your existing network infrastructure (can play nice with existing dhcp server, in proxy dhcp mode, even if systems get assigned different IP at some point, etc.)

ejolson
Posts: 5137
Joined: Tue Mar 18, 2014 11:47 am

Re: NFSv4 with user-level Kerberos

Thu Jan 23, 2020 4:44 am

incognitum wrote:
Wed Jan 22, 2020 9:15 pm
ejolson wrote:
Wed Jan 22, 2020 5:24 pm
https://www.raspberrypi.org/blog/piserver/

might be using a fully kerberised version of NFSv4 to mount home directories.

Does anyone know?
Negative.
It mounts home directories through sshfs (which uses ssh's sftp protocol).
I think PiNet also used sshfs for the user home directories. I've had troubles with this in the past because
  • Sshfs is slow.
  • It didn't implement filesystem locking well enough for Gnome or Firefox.
I wonder if things have improved. At any rate, I'm currently having some success with Kerberos and dynamic IP numbers, so NFSv4 may work after all.

incognitum
Posts: 476
Joined: Tue Oct 30, 2018 3:34 pm

Re: NFSv4 with user-level Kerberos

Thu Jan 23, 2020 12:21 pm

ejolson wrote:
Thu Jan 23, 2020 4:44 am
[*] Sshfs is slow.
Correct, it will not win benchmarks.
However the disk IO of the average student saving some Scratch project, some python code or some other document, is not that high.
So it can serve its purpose.

No experience with gnome though.
At any rate, I'm currently having some success with Kerberos and dynamic IP numbers, so NFSv4 may work after all.
Does it also no longer requires having correct forward and reverse DNS?
Recall that used to be actually the main problem. You could have dynamic IP, if all hosts had hostnames and DNS was updated correctly to reflect new IP (which is hard if you are not responsible for handing them out, and existing infrastructure does not provide DNS).

hortimech
Posts: 416
Joined: Wed Apr 08, 2015 5:52 pm

Re: NFSv4 with user-level Kerberos

Thu Jan 23, 2020 4:36 pm

Why not set up a Samba4 AD DC for the authentication ?
This will give you everything you need.

You can find a script to set NSFv4 here:
https://github.com/thctlo/samba4/blob/m ... h-nfsv4.sh

ejolson
Posts: 5137
Joined: Tue Mar 18, 2014 11:47 am

Re: NFSv4 with user-level Kerberos

Thu Jan 23, 2020 4:52 pm

incognitum wrote:
Thu Jan 23, 2020 12:21 pm
ejolson wrote:
Thu Jan 23, 2020 4:44 am
[*] Sshfs is slow.
Correct, it will not win benchmarks.
However the disk IO of the average student saving some Scratch project, some python code or some other document, is not that high.
So it can serve its purpose.

No experience with gnome though.
At any rate, I'm currently having some success with Kerberos and dynamic IP numbers, so NFSv4 may work after all.
Does it also no longer requires having correct forward and reverse DNS?
Recall that used to be actually the main problem. You could have dynamic IP, if all hosts had hostnames and DNS was updated correctly to reflect new IP (which is hard if you are not responsible for handing them out, and existing infrastructure does not provide DNS).
It seems possible to disable the reverse DNS in the configuration. Then the /etc/hosts files can be used to remind each system which principle they should be using in the Kerberos keytab.

It's not fully setup yet as I've run into a problem synchronising passwords between Kerberos and Samba, which is needed to access the home directories from Windows PCs. The difficulty is getting Samba to authenticate logins with the Kerberos database while using two separate identity mappers: One for Windows and the other for Unix to Unix. Judging from

http://kb.mit.edu/confluence/display/is ... +Directory

Even the experts maintain two separate password systems and then try to keep them in synch using add-hoc scripts. While active directory is much more than needed for Samba, it appears I've twisted my ankle on a hole dug by the same rabbit. More soon.

ejolson
Posts: 5137
Joined: Tue Mar 18, 2014 11:47 am

Re: NFSv4 with user-level Kerberos

Thu Jan 23, 2020 5:20 pm

hortimech wrote:
Thu Jan 23, 2020 4:36 pm
Why not set up a Samba4 AD DC for the authentication ?
This will give you everything you need.

You can find a script to set NSFv4 here:
https://github.com/thctlo/samba4/blob/m ... h-nfsv4.sh
That thought has also occurred to me. Having gotten NFSv4 to work with MIT Kerberos, finally, I've been reluctant to start over with a full active directory domain controller just to add Samba.

That script seems promising. I see there a blog just above it that explains how it works. I'll take a look.

hortimech
Posts: 416
Joined: Wed Apr 08, 2015 5:52 pm

Re: NFSv4 with user-level Kerberos

Fri Jan 24, 2020 12:10 pm

I wouldn't worry about the reverse dns, you cannot use IPs with kerberos, you must use hostnames.

You seem to be trying to re-invent the wheel. To use kerberos, you need somewhere to store your users, run a KDC and a DNS server, guess what a Samba AD DC does ?

ejolson
Posts: 5137
Joined: Tue Mar 18, 2014 11:47 am

Re: NFSv4 with user-level Kerberos

Fri Jan 24, 2020 4:45 pm

hortimech wrote:
Fri Jan 24, 2020 12:10 pm
I wouldn't worry about the reverse dns, you cannot use IPs with kerberos, you must use hostnames.

You seem to be trying to re-invent the wheel. To use kerberos, you need somewhere to store your users, run a KDC and a DNS server, guess what a Samba AD DC does ?
You might be right. When I started, I didn't know that active directory could be used to authenticate NFSv4 mounts. What's the alternative for pam_krb5?

hortimech
Posts: 416
Joined: Wed Apr 08, 2015 5:52 pm

Re: NFSv4 with user-level Kerberos

Fri Jan 24, 2020 7:24 pm

if by 'pam_krb5' you mean 'libpam-krb5', then believe it or not, it is 'libpam-krb5' :)

ejolson
Posts: 5137
Joined: Tue Mar 18, 2014 11:47 am

Re: NFSv4 with user-level Kerberos

Sat Jan 25, 2020 4:14 am

hortimech wrote:
Fri Jan 24, 2020 7:24 pm
if by 'pam_krb5' you mean 'libpam-krb5', then believe it or not, it is 'libpam-krb5' :)
Right now each computer boots diskless with a RAM-backed read-write overlay and gets a dynamic IP. It then fetches a Kerberos keytab using ssh with the newly generated princs

host/random.host.name
nfs/random.host.name

Originally the hostnames were based on MAC numbers, but it turned out more secure for the server to just randomly generate a new 16-byte hostname and matching keytab each time the client booted. This also prevents old keytabs from getting out of sync because lack of filesystem persistence.

At this point the diskless workstation can mount the NFSv4 export. Subsequently, users authenticated against the pam_krb5 plugin automatically have appropriate user-level access to their own home directories, but not any one else's. So far the approach looks promising and could also be a better way for Piserver.

I've looked at doing the same with a Samba-based active directory domain controller and it appears possible. Do I still need to add suitable nfs/hostname princs to active directory and copy the corresponding keytabs over to the user workstations? If there is already an active directory domain controller for a different domain on the same subnet, will they fight? Have you done this before?

hortimech
Posts: 416
Joined: Wed Apr 08, 2015 5:52 pm

Re: NFSv4 with user-level Kerberos

Sat Jan 25, 2020 9:07 am

ejolson wrote:
Sat Jan 25, 2020 4:14 am
I've looked at doing the same with a Samba-based active directory domain controller and it appears possible. Do I still need to add suitable nfs/hostname princs to active directory and copy the corresponding keytabs over to the user workstations? If there is already an active directory domain controller for a different domain on the same subnet, will they fight? Have you done this before?
In order:
Yes, it is
I believe so
No
Played with it, but my fellow team mate, Louis, uses it in production (it was his github page I pointed you to). He even produces Samba4 4.11 packages for raspbian Buster.

ejolson
Posts: 5137
Joined: Tue Mar 18, 2014 11:47 am

Re: NFSv4 with user-level Kerberos

Sat Jan 25, 2020 11:29 pm

hortimech wrote:
Sat Jan 25, 2020 9:07 am
ejolson wrote:
Sat Jan 25, 2020 4:14 am
I've looked at doing the same with a Samba-based active directory domain controller and it appears possible. Do I still need to add suitable nfs/hostname princs to active directory and copy the corresponding keytabs over to the user workstations? If there is already an active directory domain controller for a different domain on the same subnet, will they fight? Have you done this before?
In order:
Yes, it is
I believe so
No
Played with it, but my fellow team mate, Louis, uses it in production (it was his github page I pointed you to). He even produces Samba4 4.11 packages for raspbian Buster.
That's interesting. The scripts look well thought out. I've been reading and it seems the built-in DNS functions of a domain controller make it difficult to create a separate one for something like Piserver on a network that is already using active directory for other things. As far as I can tell, it is possible to create an independent Kerberos realm without affecting any existing infrastructure. So that's what I'm trying to do for now.

Is it possible to store a single password hash that can be used for both Samba and Kerberos on the LDAP server? I see people hinting at such configurations.

This weekend, I'm going to focus on getting diskless network boot with NFSv4-mounted home directories fully functional and worry about Samba later.

hortimech
Posts: 416
Joined: Wed Apr 08, 2015 5:52 pm

Re: NFSv4 with user-level Kerberos

Sun Jan 26, 2020 9:50 am

ejolson wrote:
Sat Jan 25, 2020 11:29 pm

That's interesting. The scripts look well thought out. I've been reading and it seems the built-in DNS functions of a domain controller make it difficult to create a separate one for something like Piserver on a network that is already using active directory for other things. As far as I can tell, it is possible to create an independent Kerberos realm without affecting any existing infrastructure. So that's what I'm trying to do for now.

Is it possible to store a single password hash that can be used for both Samba and Kerberos on the LDAP server? I see people hinting at such configurations.

This weekend, I'm going to focus on getting diskless network boot with NFSv4-mounted home directories fully functional and worry about Samba later.
Can you describe just how you are using Active Directory at the moment ?

First mention of piserver (something else that I never used, but it appears to be a version of LTSP, which I have used),, something that needs an update, preferably to 64bit for the rpi4, but this would entail the Rpi foundation accepting that Raspbian needs to be 64bit for the rpi4

To get Kerberos to work, you need a KDC, DNS and somewhere to store the users, which is usually LDAP. These are all components that Active Directory uses, all wrapped up in one package.

ejolson
Posts: 5137
Joined: Tue Mar 18, 2014 11:47 am

Re: NFSv4 with user-level Kerberos

Sun Jan 26, 2020 7:14 pm

hortimech wrote:
Sun Jan 26, 2020 9:50 am
Can you describe just how you are using Active Directory at the moment ?
Right now active directly is used to keep track of about 25000 users for access to Windows PCs in the computing labs and some other resources including email.

Configuring servers that authenticate through the existing AD infrastructure is simple. I can't do the same in this case because the NFSv4 clients on network-booted diskless computers apparently need dynamically generated host and nfs keytabs to participate in the user-level authentication. The security and performance needs of the existing active directory infrastructure implies that dynamically adding and deleting host entries for a bunch of network-booted student-controlled machines is out of the question. It's not practical to generate the princs ahead of time because the IP numbers and hostnames keep changing. Moreover, although there are only fifty users, there are potentially thousands of machines involved.

Thus, I've set up a separate Kerberos realm which only grants permission to mount home directories from a single NFS server to which those users can authenticate. The goal is to create something like Piserver, except using the POSIX compliant and much faster NFSv4 instead of sshfs to mount the home directories. The result so far has mostly been indigestion from a soup with too many ingredients.

hortimech
Posts: 416
Joined: Wed Apr 08, 2015 5:52 pm

Re: NFSv4 with user-level Kerberos

Sun Jan 26, 2020 9:10 pm

ejolson wrote:
Sun Jan 26, 2020 7:14 pm

Thus, I've set up a separate Kerberos realm which only grants permission to mount home directories from a single NFS server to which those users can authenticate. The goal is to create something like Piserver, except using the POSIX compliant and much faster NFSv4 instead of sshfs to mount the home directories. The result so far has mostly been indigestion from a soup with too many ingredients.
I feel that you can do this with Samba AD, it will take all the hard work out of setting up kerberos, dns, ldap etc. Just use a different dns domain and subnet from your existing Active directory domain.

ejolson
Posts: 5137
Joined: Tue Mar 18, 2014 11:47 am

Re: NFSv4 with user-level Kerberos

Mon Jan 27, 2020 5:19 am

hortimech wrote:
Sun Jan 26, 2020 9:10 pm
I feel that you can do this with Samba AD, it will take all the hard work out of setting up kerberos, dns, ldap etc. Just use a different dns domain and subnet from your existing Active directory domain.
You are probably right. I just don't know enough about Samba to make sure it doesn't start advertising its services as a domain controller and create a mess.

Speaking of messes, I now have what appears to be a working network boot image with Kerberos-authenticated NFSv4 mounts of the user's home directories.

Once I ignored Samba, everything would have been easy if systemd didn't insist on starting idmapd and gssd before the new keytab for the random hostname and IP address has been fetched.

Currently, I need flush the caches with

nfsidmap -c

at the end of system startup to fix the mess created by wrong sequencing.

So moving on to the mundane, does anyone know systemd well enough to add the few lines necessary to fetch the keytab before starting all the services related to the nfs client?

I have done similar things with BSD-style startup scripts, System V init, OpenRC and recently runit. However, my online web searches for systemd and nfs mostly return instructions for services that seem no longer to exist. I haven't understood the idea behind systemd well enough to develop any intuition how to accomplish what I want to do.

Anyway, things are working as is. The only thing left is to polish the systemd startup so it boots more smoothly.

ejolson
Posts: 5137
Joined: Tue Mar 18, 2014 11:47 am

Re: NFSv4 with user-level Kerberos

Mon Jan 27, 2020 11:03 pm

ejolson wrote:
Mon Jan 27, 2020 5:19 am
Anyway, things are working as is. The only thing left is to polish the systemd startup so it boots more smoothly.
The network boot image for NFSv4 with Kerberos authenticated home directories mostly survived the first classroom test today. There were about 3 incidents where a machine suddenly locked up within a few minutes which need to be tracked down, but most most were reliable. I'm working on getting the startup sequence right with systemd and found this amusing graph on the main webpage

Image

If the developers of the Raspberry Pi had initially chose NetBSD as the primary operating system, I wonder whether all those tutorials from 2013 would still work. In my opinion, the world would be a very different place if computer-related knowledge and skills didn't go out of date so quickly.

ejolson
Posts: 5137
Joined: Tue Mar 18, 2014 11:47 am

Re: NFSv4 with user-level Kerberos

Wed Jan 29, 2020 11:21 pm

Woohoo! After a bit of tuning the system survived an entire class without technical malfunctions. My opinion is that setting up NFSv4 with user-level Kerberos authentication was well worth the effort: One gets a maintained performant POSIX-compliant network filesystem that can by mounted on student-administered computers without the security problems associated with traditional host-based NFS authentication. For performance reasons this is a much better choice than sshfs for Piserver and pretty much the only other option since CIFS with Unix extensions has been deprecated.

If anyone has been following along and wants simple instructions for setting up NFSv4 to serve home directories to computers with dynamically assigned IP numbers and changing host names, please post here and I'll try to reply with details how I set things up.

Return to “Networking and servers”