Page 1 of 1

ZFS for Raspbian

Posted: Mon Nov 25, 2019 11:00 pm
by andrum99
Hi folks. I've created a couple of build scripts to enable relatively easy building of ZFS on Raspbian. There are a couple of knobs that will need twiddled each time you update your kernel, but apart from that it is mostly automated. Check out https://github.com/andrum99/zfs-for-pi.

Re: ZFS for Raspbian

Posted: Tue Nov 26, 2019 7:36 pm
by andrum99
I've now added automatic installation of the Debian zfsutils-linux package, which sets everything up on boot as you would expect. It also has the handy side-effect of adding the Debian buster-backports repository, allowing you to install further packages which I have also added, commented out, for ease of discovery.

Re: ZFS for Raspbian

Posted: Wed Nov 27, 2019 12:29 pm
by RandyOo
Thanks for sharing this. I've already gotten ZFS compiled and running, but if I have to set it up again, I'll give these scripts a whirl. Certainly looking forward to the 64-bit userland, and the simplification of using Debian's packages directly!

Quick aside, I was curious: does using the "-j 6" option on the "make" command during compile actually yield any benefit over "-j 4" on this quad-core machine?

Re: ZFS for Raspbian

Posted: Wed Nov 27, 2019 12:52 pm
by andrum99
RandyOo wrote:
Wed Nov 27, 2019 12:29 pm
Thanks for sharing this. I've already gotten ZFS compiled and running, but if I have to set it up again, I'll give these scripts a whirl. Certainly looking forward to the 64-bit userland, and the simplification of using Debian's packages directly!

Quick aside, I was curious: does using the "-j 6" option on the "make" command during compile actually yield any benefit over "-j 4" on this quad-core machine?
I saw a recomendation to use 1.5 times the number of CPU cores, hence 6 on the Pi 4B.

I'm in the process of moving my NAS to ZFS on a second Pi 4B. I had thought that I was getting faster speeds that using ext4, which I was, but only because I was using a faster hard disk. Using the same hard disk, the speeds are basically the same.

Re: ZFS for Raspbian

Posted: Wed Nov 27, 2019 5:52 pm
by andrum99
My zpool survived its first test today - some sort of UAS problem:

Code: Select all

Nov 27 16:05:39 pi4b2 kernel: [71006.791609] sd 0:0:0:0: [sda] tag#2 uas_eh_abort_handler 0 uas-tag 1 inflight: IN
Nov 27 16:05:39 pi4b2 kernel: [71006.791630] sd 0:0:0:0: [sda] tag#2 CDB: opcode=0x28 28 00 08 7f 7e 88 00 01 00 00
Nov 27 16:05:39 pi4b2 kernel: [71006.807633] scsi host0: uas_eh_device_reset_handler start
Nov 27 16:05:39 pi4b2 kernel: [71006.945201] usb 2-2: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
Nov 27 16:05:39 pi4b2 kernel: [71006.980417] scsi host0: uas_eh_device_reset_handler success
Nov 27 16:07:11 pi4b2 kernel: [71098.953779] sd 0:0:0:0: [sda] tag#12 uas_eh_abort_handler 0 uas-tag 1 inflight: IN
Nov 27 16:07:11 pi4b2 kernel: [71098.953799] sd 0:0:0:0: [sda] tag#12 CDB: opcode=0x28 28 00 09 38 84 60 00 01 00 00
Nov 27 16:07:11 pi4b2 kernel: [71098.969780] scsi host0: uas_eh_device_reset_handler start
Nov 27 16:07:11 pi4b2 kernel: [71099.107289] usb 2-2: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
Nov 27 16:07:11 pi4b2 kernel: [71099.142398] scsi host0: uas_eh_device_reset_handler success
Nov 27 16:16:34 pi4b2 kernel: [71662.166932] sd 0:0:0:0: [sda] tag#17 uas_eh_abort_handler 0 uas-tag 1 inflight: IN
Nov 27 16:16:34 pi4b2 kernel: [71662.166952] sd 0:0:0:0: [sda] tag#17 CDB: opcode=0x28 28 00 10 ad 0c 60 00 01 00 00
Nov 27 16:16:34 pi4b2 kernel: [71662.182934] scsi host0: uas_eh_device_reset_handler start
Nov 27 16:16:34 pi4b2 kernel: [71662.320482] usb 2-2: reset SuperSpeed Gen 1 USB device number 2 using xhci_hcd
Nov 27 16:16:34 pi4b2 kernel: [71662.355577] scsi host0: uas_eh_device_reset_handler success


I suspect if I was using ext4 instead this would have caused minor corruption. Instead ZFS simply registered 3 small delays:

Code: Select all

Nov 27 2019 16:05:39.324732559 ereport.fs.zfs.delay
Nov 27 2019 16:07:11.486567133 ereport.fs.zfs.delay
Nov 27 2019 16:16:34.693404049 ereport.fs.zfs.delay

Re: ZFS for Raspbian

Posted: Wed Nov 27, 2019 6:08 pm
by satadru
So word of advice here, please don't tell users to install zfs from the zfs master tree. There are zpool options here which enable by default which will make importing those volumes onto zfs installed on other systems, like debian or ubuntu, impossible. (Because those other distributions use zfs built from stable releases.)

You should install zfs from the 0.8 branch https://github.com/zfsonlinux/zfs/tree/zfs-0.8-release which you can do by specifying the banch when pulling zfs:

Code: Select all

git clone --depth=1 https://github.com/zfsonlinux/ -b zfs-0.8-release
Also there are instructions for zfs on ubuntu which can probably be adapted for debian/raspbian: https://github.com/zfsonlinux/zfs/wiki/ ... oot-on-ZFS

Note that just installing ```zfs-dkms``` on an ubuntu 19.10 arm64 system should give you zfs.

There is a version of that up on the debian site: https://packages.debian.org/buster-backports/zfs-dkms
Installing onto a zfs rootfs can be done, but it's a little more work, mainly requiring that you make sure that get the initrd and vmlinuz files onto the first partition of a fat sdcard.

Note also that ubuntu 20.04 will have really advanced their zfs install process with their zsys tool, which is in development, and that might make installing then really useful, since you have USERDATA pools broken out on a per user basis.

I say this as someone running with a zfs root pool on a (4gb) rpi4 with ubuntu 19.10:

Code: Select all

rpi4:~$ zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
bpool   480M  29.3M   451M        -         -     0%     6%  1.00x    ONLINE  -
rpool  58.5G  7.07G  51.4G        -         -    19%    12%  1.00x    ONLINE  -
rpi4:~$ uname -a
Linux rpi4 5.3.0-1014-raspi2 #16-Ubuntu SMP Tue Nov 26 11:18:23 UTC 2019 aarch64 aarch64 aarch64 GNU/Linux
rpi4:~$ cat /proc/device-tree/model 
Raspberry Pi 4 Model B Rev 1.1

Re: ZFS for Raspbian

Posted: Wed Nov 27, 2019 6:56 pm
by jahboater
RandyOo wrote:
Wed Nov 27, 2019 12:29 pm
Quick aside, I was curious: does using the "-j 6" option on the "make" command during compile actually yield any benefit over "-j 4" on this quad-core machine?
Another common recommendation is N cpu's + 1.
Anyway its good to have a job ready to run if one of the cores is held up waiting for I/O or something.

Re: ZFS for Raspbian

Posted: Wed Nov 27, 2019 7:41 pm
by andrum99
satadru wrote:
Wed Nov 27, 2019 6:08 pm
So word of advice here, please don't tell users to install zfs from the zfs master tree. There are zpool options here which enable by default which will make importing those volumes onto zfs installed on other systems, like debian or ubuntu, impossible. (Because those other distributions use zfs built from stable releases.)
I am pulling the master tree, but then switching to the 0.8.2 release branch. This post is in the Advanced Users forum for a reason - I expect someone who wants to try this to be smart enough to realise there may be issues. The fact you've mentioned some of the potential pitfalls here is certainly welcome.

I suggest using Raspbian for ZFS, simply because Raspbian has had more testing in general than any other distro. Ubuntu will certainly work for ZFS.

My procedure was to work around the fact we have a 64-bit kernel, but a 32-bit userland, which makes building 64-bit kernel modules tricky. Another option would be cross-compilation. As I note in the README.md, this is simply a workaround until we have a 64-bit userland for Raspbian: at that point we can just use the Debian packages, including zfs-dkms.

Re: ZFS for Raspbian

Posted: Thu Dec 05, 2019 10:41 am
by andrum99
I've been so impressed with how well ZFS is working on Raspbian with the 64-bit kernel I've now put it into use on my main Pi server. I have a single zpool consisting of two 3.5" 1TB hard disks in USB 3 enclosures attached to a 4GB Raspberry Pi 4B. It's working very well so far. I've relegated the two 2.5" 640GB drives to another zpool which contains archived data, such as older backup images from my laptop, and old backups of my various Pis.

The Pi is also running two Java apps (on OpenJDK) - Minim Server and MediaPlayer, which is an OpenHome music player. So far it's working great! The only thing that isn't working in the new version of Kodi in the official repo - it crashed on startup and I've not got round to figuring out why yet. It might just be because it doesn't like the 64-bit kernel.

Re: ZFS for Raspbian

Posted: Fri Jan 10, 2020 9:48 am
by fruitoftheloom
andrum99 wrote:
Thu Dec 05, 2019 10:41 am
I've been so impressed with how well ZFS is working on Raspbian with the 64-bit kernel I've now put it into use on my main Pi server. I have a single zpool consisting of two 3.5" 1TB hard disks in USB 3 enclosures attached to a 4GB Raspberry Pi 4B. It's working very well so far. I've relegated the two 2.5" 64GB drives to another zpool which contains archived data, such as older backup images from my laptop, and old backups of my various Pis.

The Pi is also running two Java apps (on OpenJDK) - Minim Server and MediaPlayer, which is an OpenHome music player. So far it's working great! The only thing that isn't working in the new version of Kodi in the official repo - it crashed on startup and I've not got round to figuring out why yet. It might just be because it doesn't like the 64-bit kernel.

https://www.realworldtech.com/forum/?th ... tid=189841

Re: ZFS for Raspbian

Posted: Fri Jan 10, 2020 1:04 pm
by dickon
Yeah, there are issues with bundling the kernel with the binary code, but that doesn't apply to source. It's annoying he's got that attitude, frankly; there was a lot of nonsense spouted about 'layer violations' back in the day, which seems to have hardened his stance towards it.

Whilst he's right about Oracle -- I'm on record saying that if I were ever made a CTO of anything I'd make it a disciplinary offence to install anything by that company[0] -- the fact is, Sun open sourced it under the CDDL as part of Solaris before they were bought, and that code is free to use under the terms of that licence. There's enough ambiguity that Canonical clearly feel happy enough to ship it, even if Torvalds doesn't.


[0] Their sales teams have allegedly been known to sell software that their licensing teams then say you're not covered for. You only find out about this when they audit you, which you have to agree to before running anything. Their other odious habit of installing new, chargeable features on minor point releases, enabled by default, also catches people out in expensive ways. See also: recent changes to Java's licence terms. Yeah: avoid.

Re: ZFS for Raspbian

Posted: Fri Jan 10, 2020 8:38 pm
by andrum99
fruitoftheloom wrote:
Fri Jan 10, 2020 9:48 am
andrum99 wrote:
Thu Dec 05, 2019 10:41 am
<snip>

https://www.realworldtech.com/forum/?th ... tid=189841
I don't care about licensing issues for my use case. Also, I'm not a business. Oracle will never sue me personally as I don't really have any money.

Some of the licensing issues that are often talked about are actually religious issues. Canonical feel safe enough to include ZFS in their product, so I'm inclined to think everyone is safe to do so.

Re: ZFS for Raspbian

Posted: Mon Feb 03, 2020 3:50 am
by andrum99
I've updated my scripts to build ZFS for Linux version 0.8.3, which was recently released. Get it at the usual place - https://github.com/andrum99/zfs-for-pi

Re: ZFS for Raspbian

Posted: Tue Feb 18, 2020 3:43 am
by andrum99
Readme.md updated with instructions for force loading ZFS modules at boot time to work around a problem with them not getting automatically loaded, resulting in zpools failing to mount at boot time. Usual address: https://github.com/andrum99/zfs-for-pi

Re: ZFS for Raspbian

Posted: Tue Feb 18, 2020 6:22 pm
by andrum99
I've integrated a PR from procount which tweaks the build scripts to make changing to a newer kernel release/version a bit easier, and updated the readme to document this.

https://github.com/andrum99/zfs-for-pi