andrum99
Posts: 1153
Joined: Fri Jul 20, 2012 2:41 pm

ZFS for Raspbian

Mon Nov 25, 2019 11:00 pm

Hi folks. I've created a couple of build scripts to enable relatively easy building of ZFS on Raspbian. There are a couple of knobs that will need twiddled each time you update your kernel, but apart from that it is mostly automated. Check out https://github.com/andrum99/zfs-for-pi.
Last edited by andrum99 on Wed Nov 27, 2019 1:11 pm, edited 2 times in total.

andrum99
Posts: 1153
Joined: Fri Jul 20, 2012 2:41 pm

Re: ZFS for Raspbian

Tue Nov 26, 2019 7:36 pm

I've now added automatic installation of the Debian zfsutils-linux package, which sets everything up on boot as you would expect. It also has the handy side-effect of adding the Debian buster-backports repository, allowing you to install further packages which I have also added, commented out, for ease of discovery.

RandyOo
Posts: 29
Joined: Sat Mar 12, 2016 8:16 am

Re: ZFS for Raspbian

Wed Nov 27, 2019 12:29 pm

Thanks for sharing this. I've already gotten ZFS compiled and running, but if I have to set it up again, I'll give these scripts a whirl. Certainly looking forward to the 64-bit userland, and the simplification of using Debian's packages directly!

Quick aside, I was curious: does using the "-j 6" option on the "make" command during compile actually yield any benefit over "-j 4" on this quad-core machine?

andrum99
Posts: 1153
Joined: Fri Jul 20, 2012 2:41 pm

Re: ZFS for Raspbian

Wed Nov 27, 2019 12:52 pm

RandyOo wrote:
Wed Nov 27, 2019 12:29 pm
Thanks for sharing this. I've already gotten ZFS compiled and running, but if I have to set it up again, I'll give these scripts a whirl. Certainly looking forward to the 64-bit userland, and the simplification of using Debian's packages directly!

Quick aside, I was curious: does using the "-j 6" option on the "make" command during compile actually yield any benefit over "-j 4" on this quad-core machine?
I saw a recomendation to use 1.5 times the number of CPU cores, hence 6 on the Pi 4B.

I'm in the process of moving my NAS to ZFS on a second Pi 4B. I had thought that I was getting faster speeds that using ext4, which I was, but only because I was using a faster hard disk. Using the same hard disk, the speeds are basically the same.

andrum99
Posts: 1153
Joined: Fri Jul 20, 2012 2:41 pm

Re: ZFS for Raspbian

Wed Nov 27, 2019 5:52 pm

My zpool survived its first test today - some sort of UAS problem:

Code: Select all

Nov 27 16:05:39 pi4b2 kernel: [71006.791609] sd 0:0:0:0: [sda] tag#2 uas_eh_abort_handler 0 uas-tag 1 inflight: IN
Nov 27 16:05:39 pi4b2 kernel: [71006.791630] sd 0:0:0:0: [sda] tag#2 CDB: opcode=0x28 28 00 08 7f 7e 88 00 01 00 00
Nov 27 16:05:39 pi4b2 kernel: [71006.807633] scsi host0: uas_eh_device_reset_handler start
Nov 27 16:05:39 pi4b2 kernel: [71006.945201] usb 2-2: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
Nov 27 16:05:39 pi4b2 kernel: [71006.980417] scsi host0: uas_eh_device_reset_handler success
Nov 27 16:07:11 pi4b2 kernel: [71098.953779] sd 0:0:0:0: [sda] tag#12 uas_eh_abort_handler 0 uas-tag 1 inflight: IN
Nov 27 16:07:11 pi4b2 kernel: [71098.953799] sd 0:0:0:0: [sda] tag#12 CDB: opcode=0x28 28 00 09 38 84 60 00 01 00 00
Nov 27 16:07:11 pi4b2 kernel: [71098.969780] scsi host0: uas_eh_device_reset_handler start
Nov 27 16:07:11 pi4b2 kernel: [71099.107289] usb 2-2: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
Nov 27 16:07:11 pi4b2 kernel: [71099.142398] scsi host0: uas_eh_device_reset_handler success
Nov 27 16:16:34 pi4b2 kernel: [71662.166932] sd 0:0:0:0: [sda] tag#17 uas_eh_abort_handler 0 uas-tag 1 inflight: IN
Nov 27 16:16:34 pi4b2 kernel: [71662.166952] sd 0:0:0:0: [sda] tag#17 CDB: opcode=0x28 28 00 10 ad 0c 60 00 01 00 00
Nov 27 16:16:34 pi4b2 kernel: [71662.182934] scsi host0: uas_eh_device_reset_handler start
Nov 27 16:16:34 pi4b2 kernel: [71662.320482] usb 2-2: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
Nov 27 16:16:34 pi4b2 kernel: [71662.355577] scsi host0: uas_eh_device_reset_handler success


I suspect if I was using ext4 instead this would have caused minor corruption. Instead ZFS simply registered 3 small delays:

Code: Select all

Nov 27 2019 16:05:39.324732559 ereport.fs.zfs.delay
Nov 27 2019 16:07:11.486567133 ereport.fs.zfs.delay
Nov 27 2019 16:16:34.693404049 ereport.fs.zfs.delay

satadru
Posts: 24
Joined: Thu Apr 18, 2013 5:18 pm

Re: ZFS for Raspbian

Wed Nov 27, 2019 6:08 pm

So word of advice here, please don't tell users to install zfs from the zfs master tree. There are zpool options here which enable by default which will make importing those volumes onto zfs installed on other systems, like debian or ubuntu, impossible. (Because those other distributions use zfs built from stable releases.)

You should install zfs from the 0.8 branch https://github.com/zfsonlinux/zfs/tree/zfs-0.8-release which you can do by specifying the banch when pulling zfs:

Code: Select all

git clone --depth=1 https://github.com/zfsonlinux/ -b zfs-0.8-release
Also there are instructions for zfs on ubuntu which can probably be adapted for debian/raspbian: https://github.com/zfsonlinux/zfs/wiki/ ... oot-on-ZFS

Note that just installing ```zfs-dkms``` on an ubuntu 19.10 arm64 system should give you zfs.

There is a version of that up on the debian site: https://packages.debian.org/buster-backports/zfs-dkms
Installing onto a zfs rootfs can be done, but it's a little more work, mainly requiring that you make sure that get the initrd and vmlinuz files onto the first partition of a fat sdcard.

Note also that ubuntu 20.04 will have really advanced their zfs install process with their zsys tool, which is in development, and that might make installing then really useful, since you have USERDATA pools broken out on a per user basis.

I say this as someone running with a zfs root pool on a (4gb) rpi4 with ubuntu 19.10:

Code: Select all

rpi4:~$ zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
bpool   480M  29.3M   451M        -         -     0%     6%  1.00x    ONLINE  -
rpool  58.5G  7.07G  51.4G        -         -    19%    12%  1.00x    ONLINE  -
rpi4:~$ uname -a
Linux rpi4 5.3.0-1014-raspi2 #16-Ubuntu SMP Tue Nov 26 11:18:23 UTC 2019 aarch64 aarch64 aarch64 GNU/Linux
rpi4:~$ cat /proc/device-tree/model 
Raspberry Pi 4 Model B Rev 1.1

jahboater
Posts: 5414
Joined: Wed Feb 04, 2015 6:38 pm
Location: West Dorset

Re: ZFS for Raspbian

Wed Nov 27, 2019 6:56 pm

RandyOo wrote:
Wed Nov 27, 2019 12:29 pm
Quick aside, I was curious: does using the "-j 6" option on the "make" command during compile actually yield any benefit over "-j 4" on this quad-core machine?
Another common recommendation is N cpu's + 1.
Anyway its good to have a job ready to run if one of the cores is held up waiting for I/O or something.

andrum99
Posts: 1153
Joined: Fri Jul 20, 2012 2:41 pm

Re: ZFS for Raspbian

Wed Nov 27, 2019 7:41 pm

satadru wrote:
Wed Nov 27, 2019 6:08 pm
So word of advice here, please don't tell users to install zfs from the zfs master tree. There are zpool options here which enable by default which will make importing those volumes onto zfs installed on other systems, like debian or ubuntu, impossible. (Because those other distributions use zfs built from stable releases.)
I am pulling the master tree, but then switching to the 0.8.2 release branch. This post is in the Advanced Users forum for a reason - I expect someone who wants to try this to be smart enough to realise there may be issues. The fact you've mentioned some of the potential pitfalls here is certainly welcome.

I suggest using Raspbian for ZFS, simply because Raspbian has had more testing in general than any other distro. Ubuntu will certainly work for ZFS.

My procedure was to work around the fact we have a 64-bit kernel, but a 32-bit userland, which makes building 64-bit kernel modules tricky. Another option would be cross-compilation. As I note in the README.md, this is simply a workaround until we have a 64-bit userland for Raspbian: at that point we can just use the Debian packages, including zfs-dkms.

andrum99
Posts: 1153
Joined: Fri Jul 20, 2012 2:41 pm

Re: ZFS for Raspbian

Thu Dec 05, 2019 10:41 am

I've been so impressed with how well ZFS is working on Raspbian with the 64-bit kernel I've now put it into use on my main Pi server. I have a single zpool consisting of two 3.5" 1TB hard disks in USB 3 enclosures attached to a 4GB Raspberry Pi 4B. It's working very well so far. I've relegated the two 2.5" 640GB drives to another zpool which contains archived data, such as older backup images from my laptop, and old backups of my various Pis.

The Pi is also running two Java apps (on OpenJDK) - Minim Server and MediaPlayer, which is an OpenHome music player. So far it's working great! The only thing that isn't working in the new version of Kodi in the official repo - it crashed on startup and I've not got round to figuring out why yet. It might just be because it doesn't like the 64-bit kernel.
Last edited by andrum99 on Tue Feb 18, 2020 3:47 am, edited 1 time in total.

fruitoftheloom
Posts: 22568
Joined: Tue Mar 25, 2014 12:40 pm
Location: Delightful Dorset

Re: ZFS for Raspbian

Fri Jan 10, 2020 9:48 am

andrum99 wrote:
Thu Dec 05, 2019 10:41 am
I've been so impressed with how well ZFS is working on Raspbian with the 64-bit kernel I've now put it into use on my main Pi server. I have a single zpool consisting of two 3.5" 1TB hard disks in USB 3 enclosures attached to a 4GB Raspberry Pi 4B. It's working very well so far. I've relegated the two 2.5" 64GB drives to another zpool which contains archived data, such as older backup images from my laptop, and old backups of my various Pis.

The Pi is also running two Java apps (on OpenJDK) - Minim Server and MediaPlayer, which is an OpenHome music player. So far it's working great! The only thing that isn't working in the new version of Kodi in the official repo - it crashed on startup and I've not got round to figuring out why yet. It might just be because it doesn't like the 64-bit kernel.

https://www.realworldtech.com/forum/?th ... tid=189841
Rather than negativity think outside the box !

Asus ChromeBox 3 Celeron is my other computer.

User avatar
dickon
Posts: 1168
Joined: Sun Dec 09, 2012 3:54 pm
Location: Home, just outside Reading

Re: ZFS for Raspbian

Fri Jan 10, 2020 1:04 pm

Yeah, there are issues with bundling the kernel with the binary code, but that doesn't apply to source. It's annoying he's got that attitude, frankly; there was a lot of nonsense spouted about 'layer violations' back in the day, which seems to have hardened his stance towards it.

Whilst he's right about Oracle -- I'm on record saying that if I were ever made a CTO of anything I'd make it a disciplinary offence to install anything by that company[0] -- the fact is, Sun open sourced it under the CDDL as part of Solaris before they were bought, and that code is free to use under the terms of that licence. There's enough ambiguity that Canonical clearly feel happy enough to ship it, even if Torvalds doesn't.


[0] Their sales teams have allegedly been known to sell software that their licensing teams then say you're not covered for. You only find out about this when they audit you, which you have to agree to before running anything. Their other odious habit of installing new, chargeable features on minor point releases, enabled by default, also catches people out in expensive ways. See also: recent changes to Java's licence terms. Yeah: avoid.

andrum99
Posts: 1153
Joined: Fri Jul 20, 2012 2:41 pm

Re: ZFS for Raspbian

Fri Jan 10, 2020 8:38 pm

fruitoftheloom wrote:
Fri Jan 10, 2020 9:48 am
andrum99 wrote:
Thu Dec 05, 2019 10:41 am
<snip>

https://www.realworldtech.com/forum/?th ... tid=189841
I don't care about licensing issues for my use case. Also, I'm not a business. Oracle will never sue me personally as I don't really have any money.

Some of the licensing issues that are often talked about are actually religious issues. Canonical feel safe enough to include ZFS in their product, so I'm inclined to think everyone is safe to do so.

andrum99
Posts: 1153
Joined: Fri Jul 20, 2012 2:41 pm

Re: ZFS for Raspbian

Mon Feb 03, 2020 3:50 am

I've updated my scripts to build ZFS for Linux version 0.8.3, which was recently released. Get it at the usual place - https://github.com/andrum99/zfs-for-pi

andrum99
Posts: 1153
Joined: Fri Jul 20, 2012 2:41 pm

Re: ZFS for Raspbian

Tue Feb 18, 2020 3:43 am

Readme.md updated with instructions for force loading ZFS modules at boot time to work around a problem with them not getting automatically loaded, resulting in zpools failing to mount at boot time. Usual address: https://github.com/andrum99/zfs-for-pi

andrum99
Posts: 1153
Joined: Fri Jul 20, 2012 2:41 pm

Re: ZFS for Raspbian

Tue Feb 18, 2020 6:22 pm

I've integrated a PR from procount which tweaks the build scripts to make changing to a newer kernel release/version a bit easier, and updated the readme to document this.

https://github.com/andrum99/zfs-for-pi

Return to “Advanced users”