Page 1 of 1

Anyone tried ZFS on Pi 4 or planning to do so?

Posted: Thu Aug 15, 2019 9:07 am
by JulianFP
As in the title, has anyone tried or is close to trying ZFS on a Pi 4? I know it's resource hungry but at least in my case I want to have a go once my pi 4 4GB arrives and with Ubuntu looking to offer it even as a root file system option ZFS does look to be taking a significant step forward in its reach.

I use my Pi as a Kodi box and also as a very low traffic NAS to act as one of quite a number of backup target for the backups from my main PC where the total volume of write traffic is usually less than 100MB a day (a few incremental changes on spreadsheets and perhaps a few document scans) and the total backup archive size is currently about 0.6TB so for me it's the bit scrubbing and other data integrity stuff that is most important.

Right now my Pi3B+ runs my NAS just fine using a pair of 2TB USB drives formatted as ext4 and mirrored using mdadm but I do hanker after the extra data integrity features that zfs would give me since it's sole use is as a backup target. For my use case I think I would technically even be within ZFS memory resource guidelines which I have read are recommended to be 1GB RAM + an extra 1GB per 1TB of storage so, pessimistically assuming 1GB for OS and other stuff, that shouldn't be presenting problems at least in available RAM terms for a 2TB array which is what I have and, as mentioned above, my performance requirements are extremely modest.

Re: Anyone tried ZFS on Pi 4 or planning to do so?

Posted: Sun Aug 18, 2019 12:29 pm
by MarkJ62
I run ZFS on a Debian x64 machine. The first issue will be that it requires 64 bit so that means aarch64 architecture. You should be fine with 2TB drives in a mirror using a 4GB Pi4. It only uses up to half the available memory when you start giving it a lot of work to do and it frees it again when its quiet. You won’t be able to turn on deduplication as that does require a lot of memory.

Re: Anyone tried ZFS on Pi 4 or planning to do so?

Posted: Fri Aug 23, 2019 8:05 am
by BeauSlim
Trying ZFS on the Pi4 is one of the reasons I grabbed the 4GB version. Mark is probably correct that it will require 64 bit.

There are still firmware issues (USB booting) and OS support (uasp bugs, 1GB RAM limitations, etc.) isn't there yet, so I'll probably wait a while before bothering.

EDIT: I gave a few 64-bit distros (gentoo, Ubuntu 64, Ubuntu Core18) a spin today. The gentoo build looks like it might be capable of running ZFS. It had no problem with my USB SATA devices and recognizes the full 4GB of the Pi4 (unlike Ubuntu 64).

A custom kernel (or at least a module) is required and I don't have time right now to learn gentoo's mid-90's style of build-from-source package admin, but if someone is interested, https://github.com/sakaki-/gentoo-on-rpi-64bit is probably the best place to start.

Re: Anyone tried ZFS on Pi 4 or planning to do so?

Posted: Wed Aug 28, 2019 4:23 am
by colettace
A few of us have been trying to compile it from scratch but we're running into issues with the makefiles and such. Here's the thread, scroll all the way down for the part about compiling on rpi4. Any help is appreciated!

Re: Anyone tried ZFS on Pi 4 or planning to do so?

Posted: Sat Sep 21, 2019 12:00 pm
by cyclic
I have zfs installed on pi4 and testing with 2 ssd mirror for 2 weeks.
It seems to work. Have had 200GB plus read/write.
There is a problem with reboot or cold start whereby the ssd come up both "FAULTED".
However next cold start may come up OK and online.
Does anyone have USB3 problems intermittently on start or restart?

Re: Anyone tried ZFS on Pi 4 or planning to do so?

Posted: Sun Sep 22, 2019 4:03 am
by BeauSlim
cyclic, I'd like to hear more details about your setup. Are you running Raspbian? 32 or 64-bit? What USB enclosures are you using for your SSDs?

I found setting up ZFS to be relatively easy on the 64-bit Ubuntu Server images for Pi4 from https://github.com/TheRemote/Ubuntu-Ser ... l/releases and with a couple hints from https://github.com/TheRemote/Ubuntu-Ser ... /issues/12

Using ZFS as a backing store for docker works great, but I'm finding local write speeds to a simple ZFS volume on a Samsung 750 EVO in a JMicron 567-based USB3 enclosure to be a little slow at around 75 MB/s. Remote NFS writes to this same share drop to about 50MB/s. Recompiling the kernel to support UASP doesn't seem to make much difference to ZFS performance.

Re: Anyone tried ZFS on Pi 4 or planning to do so?

Posted: Tue Oct 15, 2019 5:52 pm
by Thinkcat
To just run ZFS, I doubt 64 bits or enormous amount of memory is strictly necessary. I have a 10 year old Atom-based netbook that is i386 and had only 1 GB of memory when I installed lubuntu into it onto a ZFS root volume.

Atom laptops will get stuck while doing compiles and such unless one tunes a few kernel and ARC parameters before proceeding. One thing is the "vmalloc" kernel parameter. It should be maybe 512 MB in a 1 GB system. Another one is to set the maximum size of ARC clearly below that amount. To maybe 384 MB.

I haven't tried ZFS in a Pi yet. But I do plan to. Soon.

Re: Anyone tried ZFS on Pi 4 or planning to do so?

Posted: Sat Oct 26, 2019 12:41 pm
by Thinkcat
I have compiled the 0.8.2 version from Github according to these instructions, but now I don't know how to create an initramfs. There are packages for 0.7.6 that I can see, but I'd rather not uninstall 0.8.2 and go to an older version. Any pointers?

Otherwise it works perfectly. I just followed manually what's in that script in my link.

Re: Anyone tried ZFS on Pi 4 or planning to do so?

Posted: Thu Nov 07, 2019 9:25 pm
by RandyOo
Thinkcat wrote:
Sat Oct 26, 2019 12:41 pm
I have compiled the 0.8.2 version from Github according to these instructions, but now I don't know how to create an initramfs. There are packages for 0.7.6 that I can see, but I'd rather not uninstall 0.8.2 and go to an older version. Any pointers?

Otherwise it works perfectly. I just followed manually what's in that script in my link.
@Thinkcat , your link is giving a 404 error, unfortunately.

I did manage to get zfs compiled and running based on the official build instructions , followed by a

Code: Select all

make install
, but then encountered this error when trying to execute the zfs command:

Code: Select all

zfs: error while loading shared libraries: libnvpair.so.1: cannot open shared object file: No such file or directory
.

The solution was found here:

Code: Select all

ln -s /usr/local/lib/nvpair.a /usr/lib/nvpair.a
ln -s /usr/local/lib/libnvpair.la /usr/lib/libnvpair.a
ln -s /usr/local/lib/libnvpair.so /usr/lib/libnvpair.so
ln -s /usr/local/lib/libnvpair.so.1 /usr/lib/libnvpair.so.1
ln -s /usr/local/lib/libnvpair.so.1.0.1 /usr/lib/libnvpair.so.1.0.1
ln -s /usr/local/lib/libuutil.a /usr/lib/libuutil.a
ln -s /usr/local/lib/libuutil.la /usr/lib/libuutil.la
ln -s /usr/local/lib/libuutil.so /usr/lib/libuutil.so
ln -s /usr/local/lib/libuutil.so.1 /usr/lib/libuutil.so.1
ln -s /usr/local/lib/libuutil.so.1.0.1 /usr/lib/libuutil.so.1.0.1
ln -s /usr/local/lib/libzfs.a /usr/lib/libzfs.a
ln -s /usr/local/lib/libzfs_core.a /usr/lib/libzfs_core.a
ln -s /usr/local/lib/libzfs_core.la /usr/lib/libzfs_core.la
ln -s /usr/local/lib/libzfs_core.so /usr/lib/libzfs_core.so
ln -s /usr/local/lib/libzfs_core.so.1 /usr/lib/libzfs_core.so.1
ln -s /usr/local/lib/libzfs_core.so.1.0.0 /usr/lib/libzfs_core.so.1.0.0
ln -s /usr/local/lib/libzfs.la /usr/lib/libzfs.la
ln -s /usr/local/lib/libzfs.so /usr/lib/libzfs.so
ln -s /usr/local/lib/libzfs.so.2 /usr/lib/libzfs.so.2
ln -s /usr/local/lib/libzfs.so.2.0.0 /usr/lib/libzfs.so.2.0.0
It does appear to be functional, thus far.

Re: Anyone tried ZFS on Pi 4 or planning to do so?

Posted: Sat Nov 09, 2019 5:55 pm
by Thinkcat
RandyOo wrote:
Thu Nov 07, 2019 9:25 pm
@Thinkcat , your link is giving a 404 error, unfortunately.
Here now.

Re: Anyone tried ZFS on Pi 4 or planning to do so?

Posted: Thu Nov 14, 2019 10:50 pm
by RandyOo
I just posted in the Pi4 64-bit kernel thread, but suppose it merits sharing here, too:
https://www.raspberrypi.org/forums/view ... 5#p1566275

TLDR: ZFS sorta works on the default kernel, but repeatedly triggered memory errors on a moderate load. With the 64-bit kernel, not only are the errors gone, but performance is much better, to boot!

@Thinkcat, thanks for the link. I haven't looked into ZFS on root, but that's not a big priority for me.

Re: Anyone tried ZFS on Pi 4 or planning to do so?

Posted: Fri Nov 15, 2019 12:49 am
by andrum99
I tried building ZFS on Raspbian and I get the following error:

Code: Select all

Making all in module
In file included from /home/pi/zfs/include/spl/sys/kstat.h:33,
                 from /home/pi/zfs/module/spl/spl-kstat.c:28:
/home/pi/zfs/module/spl/spl-kstat.c: In function ‘proc_kstat_open’:
/home/pi/zfs/include/spl/sys/proc.h:31:22: error: implicit declaration of function ‘PDE’ [-Werror=implicit-function-declaration]
 #define PDE_DATA(x) (PDE(x)->data)
                      ^~~
/home/pi/zfs/module/spl/spl-kstat.c:484:15: note: in expansion of macro ‘PDE_DATA’
  f->private = PDE_DATA(inode);
               ^~~~~~~~
/home/pi/zfs/include/spl/sys/proc.h:31:28: error: invalid type argument of ‘->’ (have ‘int’)
 #define PDE_DATA(x) (PDE(x)->data)
                            ^~
/home/pi/zfs/module/spl/spl-kstat.c:484:15: note: in expansion of macro ‘PDE_DATA’
  f->private = PDE_DATA(inode);
               ^~~~~~~~
cc1: some warnings being treated as errors
make[5]: *** [scripts/Makefile.build:303: /home/pi/zfs/module/spl/spl-kstat.o] Error 1
make[5]: *** Waiting for unfinished jobs....
make[4]: *** [scripts/Makefile.build:544: /home/pi/zfs/module/spl] Error 2
make[4]: *** Waiting for unfinished jobs....
How do I fix this?

Re: Anyone tried ZFS on Pi 4 or planning to do so?

Posted: Sat Jun 06, 2020 8:47 am
by pmatuszy
this is how to compile the latest version of ZFS on raspberry pi - i've checked it on rpi v.3 and v.4 - works like a charm:


install some packages:
apt-get install git bc bison flex libssl-dev libncurses5-dev

install rpi-source - this is very important !!

sudo wget https://raw.githubusercontent.com/notro ... rpi-source -O /usr/local/bin/rpi-source && sudo chmod +x /usr/local/bin/rpi-source && /usr/local/bin/rpi-source -q --tag-update

rpi-source

create a file zfs-install.sh with the following content

vi zfs-install.sh
==== CUT HERE ===
#!/usr/bin/env sh

set -e
set -x

CUR_PWD="$(pwd)"
cd "$(dirname $0)"

if [ ! -d zfs ]; then
# https://github.com/zfsonlinux/zfs/wiki/Building-ZFS
sudo apt-get install -y build-essential autoconf automake libtool gawk alien fakeroot ksh
sudo apt-get install -y zlib1g-dev uuid-dev libattr1-dev libblkid-dev libselinux-dev libudev-dev
sudo apt-get install -y libacl1-dev libaio-dev libdevmapper-dev libssl-dev libelf-dev
sudo apt-get install -y python3 python3-dev python3-setuptools python3-cffi

#sudo apt-get install -y linux-headers-$(uname -r)
sudo apt-get install raspberrypi-kernel-headers

sudo apt-get install -y git
git clone https://github.com/zfsonlinux/zfs.git
fi

cd zfs
git checkout .
git checkout zfs-0.8-release
git pull
git branch

make clean || true
make distclean || true

./autogen.sh
autoreconf --install --force
./configure

make -s -j$(nproc)
sudo make install

sudo ldconfig

sudo depmod -a
sudo modprobe zfs
sudo zpool import -a
sudo zpool import -a -d /dev

cd "${CUR_DIR}"
==== CUT HERE ===

chmod 700 zfs-install.sh

./zfs-install.sh

job done

Re: Anyone tried ZFS on Pi 4 or planning to do so?

Posted: Sat Jun 06, 2020 2:32 pm
by nexusguy59
Thank you for this I was wondering how to do this. I did this on an ACER H320 that I turned in to an Ubuntu NAS with zfs. I was wanting to do it on the PI and now I can this is awesome!

Re: Anyone tried ZFS on Pi 4 or planning to do so?

Posted: Sat Jun 06, 2020 3:10 pm
by andrum99
I've got a simpler solution:

1. Install Ubuntu 20.04 arm64 on Pi
2. sudo apt install zfs-dkms

That works well for my simple use case.

Edit: note that you will lose Pi-specific graphics acceleration. Ubuntu for Pi is a server version and does not include that.

Re: Anyone tried ZFS on Pi 4 or planning to do so?

Posted: Sat Jun 06, 2020 3:11 pm
by dickon
Edit: What andrum99 said.

Re: Anyone tried ZFS on Pi 4 or planning to do so?

Posted: Thu Jun 25, 2020 8:09 pm
by Cpzengel
Project nasbeery.de

The ZFS Nas that fits in a beer can

Need
Rpi4
2x usb hdd/ssd
Ubuntu 20 lts
Internet via Wire

wget nasras.de
sh index.html

Wait 25 mins

Get
Share
Cockpit Webgui on Port 9090 with full ZFS
Comming
Keep Data on install
Use Windows Explorer to Restore
Active Directory

Cheers

Re: Anyone tried ZFS on Pi 4 or planning to do so?

Posted: Thu Jul 02, 2020 4:17 pm
by Cpzengel
Our http://nasbeery.de Website is now poulated with more Information
A Video in german is up to come to zfs.rocks

cheers
chriz

Re: Anyone tried ZFS on Pi 4 or planning to do so?

Posted: Mon Jul 20, 2020 9:03 am
by Peter-Pan
I plan to install ZFS on my Raspberry 4 which is running the 64bit Raspbian OS. But for testing I only have a 3 B+ available where I want to get it running and test it first.

I have installed a fresh Raspbian 64bit OS on the 3B+.
Currently there are 2 kernels installed:
4.19.118+ and 5.4.42-v8+
I have first tried the simplest method:
sudo apt install zfs-dkms zfsutils-linux

But I get this error message:
"Building initial module for 4.19.118+
configure: error:
*** Unable to build an empty module."

After that I tried several instructions from this thread where you build the modules yourself, but it always comes down to the same error.
Unable to build an empty module.

I have already installed the following things:
- sudo wget https://raw.githubusercontent.com/notro ... rpi-source -O /usr/local/bin/rpi-source && sudo chmod +x /usr/local/bin/rpi-source && /usr/local/bin/rpi-source -q --tag-update
- rpi-source
- apt reinstall raspberrypi-kernel-headers
- a lot of other packages for building, but dont think they are importend here

Anyone got any advice?

Re: Anyone tried ZFS on Pi 4 or planning to do so?

Posted: Tue Jul 21, 2020 5:50 am
by BeauSlim
I am currently running on a Pi4 4GB with Ubuntu64. As you say, a simple "sudo apt install zfs-dkms zfsutils-linux" and you're done.

I'm not saying Ubuntu is right for you. It has its own issues.

If you really want to run on raspbian, my first guess is make sure you aren't running out of RAM while compiling. I assume you are building on an external USB HD/SSD? If so, create a swap file and make sure it is active before compiling.

Or set up a cross-compile environment on a PC and do it there. It is fiddly to set up but 10x faster.

Second, I would try compiling just for the 5.4 kernel.

Re: Anyone tried ZFS on Pi 4 or planning to do so?

Posted: Tue Jul 21, 2020 10:08 am
by Cpzengel
nasbeery.de nas skript is running like a charm with ubuntu 20.04 lts server, thats certified for rpi4
i never ran out of ram, 4gb should be fine for about 2-4tb of disk mirror

by the way...

we added an raid status inidcator via buzzer or led on gpio and fixed the previous versions with windows explorer
all snapshots are now available in windows explorer!

Re: Anyone tried ZFS on Pi 4 or planning to do so?

Posted: Wed Jul 22, 2020 11:06 pm
by BeauSlim
Cpzengel wrote:
Tue Jul 21, 2020 10:08 am
we added an raid status inidcator via buzzer or led on gpio
That's a nice touch! It is a Pi after all.

Your script really should ask the user to enter a password.

Re: Anyone tried ZFS on Pi 4 or planning to do so?

Posted: Wed Aug 19, 2020 1:39 pm
by Cpzengel
i did not get a dialog working, we will do more improvements like keep data soon!