ZFS: From super to microcomputer


23 posts
by fdion » Tue Oct 02, 2012 8:03 pm
I'll cut to the chase, but I have ZFS working on the Raspberry Pi. If anybody swings by CHS tomorrow, I'll bring a Pi with it configured.

All the gory details are here:
http://raspberry-python.blogspot.com/2012/10/zfs-file-system-on-raspberry-pi.html

I think that is pretty cool, but it'll become more obvious why it is cool in some future posts. Still, the recipe is there and the demo.

Let me know what you think.
Francois
http://raspberry-python.blogspot.com - @f_dion
User avatar
Posts: 305
Joined: Sun Sep 16, 2012 2:33 pm
Location: North Carolina, USA
by fdion » Thu Oct 04, 2012 4:41 pm
I added comments, there is an issue to resolve with the zpool status, but the rest seems to work.
BTW, I see some activity today (early) from a referring url of slashdot.org/popular, but I dont see anything there about zfs posted in recent times. I was curious to see what they were talking about that might have direct linked to my url. Anybody saw anything there today?
Francois
http://raspberry-python.blogspot.com - @f_dion
User avatar
Posts: 305
Joined: Sun Sep 16, 2012 2:33 pm
Location: North Carolina, USA
by jackokring » Fri Oct 05, 2012 12:18 am
Is ZFS the copy on write filesystem or something else?
Pi=B256R0USB CL4SD8GB Raspbian Stock. https://sites.google.com/site/rubikcompression/strictly-long https://dl.dropboxusercontent.com/u/1615413/Own%20Work/Leptronics.pdf https://groups.google.com/forum/#!topic/comp.compression/t22ct_BKi9w
User avatar
Posts: 784
Joined: Tue Jul 31, 2012 8:27 am
Location: London, UK
by fdion » Sat Oct 06, 2012 3:09 am
Yes, i'm referring to the Copy On Write ZFS file system. To be precise it is a volume manager and file system in one, with really useful features like snapshots (free performance wise since it is a C-O-W filesystem) and clones, send / receive, compression per fs (multiple fs per pool are possible), mirroring, raid-z etc.
Francois
http://raspberry-python.blogspot.com - @f_dion
User avatar
Posts: 305
Joined: Sun Sep 16, 2012 2:33 pm
Location: North Carolina, USA
by marked » Thu Dec 27, 2012 9:08 pm
I am having a problem compiling this.
Following your instructions I get to point where I am running scons, and it fails the compilation with
Code: Select all
openssl/sha.h cannot be found.


I have openssl installed, openssl-dev and openssl-devel are not found with apt-get. Do I need to compile up a new openssl library?

thanks,
mark
Posts: 64
Joined: Fri Jul 29, 2011 4:25 pm
by marked » Sat Dec 29, 2012 11:56 pm
Right, well after leaving it overnight, locate openssl provided a location of /lib/include/openssl/*. One scons later and I have a working zfs.

I have no idea as to what happened in the intervening time unless an ldconfig was required.
Now to test with some USB disks. Thanks for the guide.

mark
Posts: 64
Joined: Fri Jul 29, 2011 4:25 pm
by chuchana » Thu Jan 31, 2013 6:34 pm
I also got this error when I ran scons:
Code: Select all
fatal error: openssl/sha.h: No such file or directory

The solution was installing libssl-dev¹:
Code: Select all
sudo apt-get install libssl-dev


¹ source: http://blog.tomg.co/post/21322413373/how-to-install-node-js-on-your-raspberry-pi
Posts: 3
Joined: Thu Jan 31, 2013 5:10 pm
by la_tristesse » Thu May 16, 2013 12:58 pm
How do I import a pool on a usb disk?
Posts: 16
Joined: Sun Jun 24, 2012 1:17 pm
by marked » Thu May 16, 2013 1:33 pm
sudo zpool import

and then hope that your zfs-fuse/zfsonlinux version can mount the pool.

marked
Posts: 64
Joined: Fri Jul 29, 2011 4:25 pm
by Nixzs » Fri May 17, 2013 1:31 am
Hi I am a kid and I really want to get xfs on the pi to work and I am not that great at the coding on it but I am learning. I am good with major os's and servers etc though so I will understand most of your terms and stuff. Anyways I have tried doing the "apt-get" etc on the command line and I have downloaded zfs and spl from zfsonlinux and when i did"apt-get" I did download a special archive made for whezzy zfs but I dont know what to do with all these files please help me :D
Thx,
Nils
Last edited by Nixzs on Tue May 21, 2013 10:02 pm, edited 1 time in total.
Posts: 4
Joined: Fri May 17, 2013 1:28 am
by marked » Fri May 17, 2013 10:55 am
what exactly are you trying to achieve? just to test is or to move zfs drives around different systems?

When I was doing this early this year I found:

use zfsonlinux over zfs-fuse (as above from fdion) up to 100x speed improvement, but generally 10x particularly on network writes. smbshare=on works on zfsonlinux. speeds generally around 2MB/s - up to 10x slower than using xfs/ext(2|3|4).
This was using raspbian wheezy 2012-12-08, and zfsonlinux 0.6.rc13. You will need to compile a kernel (12hrs on rpi, minutes with a cross-compile) then compile spl/zfs.

A disk formatted with zfs from the above could be viewed as being a zfs disk but could be not mounted on OpenIndiana/OpenSolaris/ Nexenta/FreeNAS etc. due to a mismatch on versioning. I didn't try the reverse, so was unable to determine exactly what was wrong.

Using an external 500GB usb disk I could get about 20GB on a dedup setup before hitting memory and 100% cpu usage problems with the spl on a 512MB rpi model B. I didn't try any comprehensive tuning to achieve better performance apart from setting a zfs_arc_max of 128MB.

(as an aside I also have btrfs working on rpi as well.) I have yet to try with the latest zfs/raspbian wheezy.

Basically interesting exercise in getting things working, and seeing how to get zfs working, but shouldn't be used in any anger.

TL;DR - Slow write speed of approx <2MB/s ; No zfs tuning data ; cannot move disks to other systems; performance dies horribly, with the zfs/spl processes taking 100%cpu.
Posts: 64
Joined: Fri Jul 29, 2011 4:25 pm
by Nixzs » Tue May 21, 2013 10:04 pm
While I want to run some sort of home ZFS kind of FTP server for me and my friend but nothing special I just wan tot see it run and work on the Raspberry Pi and not really store any thing on it because I only have a 16GB flash drive for external zfs drive and the 4GB SD Card so not really special just to have some fun and get it work ;)
Posts: 4
Joined: Fri May 17, 2013 1:28 am
by Heater » Tue May 21, 2013 10:16 pm
Is this going to make using SD cards on the Raspberry Pi any more reliable? Because just now I'm losing my temper with them. Floppy disks were more reliable back in the day.
Posts: 1235
Joined: Tue Jul 17, 2012 3:02 pm
by Nixzs » Tue May 21, 2013 11:08 pm
Lol maybe but I cant even get it to work so idk if it ever gonna happen could you help me maybe. If you could help me I will help you with Reliable SD Cards. Although ZFS is kinda not really bootable thats a great idea
----Nils
Posts: 4
Joined: Fri May 17, 2013 1:28 am
by mar.ste » Mon Sep 16, 2013 6:52 pm
ZFS should run on FreeBSD out of the box!...
https://wiki.freebsd.org/FreeBSD/arm/Raspberry%20Pi
Posts: 6
Joined: Tue Oct 16, 2012 3:54 pm
by rayb0rn » Wed Oct 16, 2013 7:17 pm
Hello, I am running Raspbian GNU/Linux 7 on my Pi.

I have read through all of the replies, please can someone clarify which distribution package I should be installing from http://zfsonlinux.org/ instead of going down the zfs-fuse route inorder to setup ZFS.

Thank you
Posts: 2
Joined: Fri Aug 16, 2013 12:29 pm
by yee379 » Sun Jan 12, 2014 5:36 am
i have been running zfs since the early zfs-fuse days (now on native) on my ubuntu server for years without issue.

i have recently set up a mac with zfs as a simple remote backup solution using (zfs send/recv and snapshots - see https://github.com/adaugherity/zfs-backup), however the kernel kext/module is rather alpha quality at the moment (lots of kernel panics). in addition, buying a new mac isn't exactly cheap so i figured why not try on a raspberry pi!

could someone comment as to the 64-bit requirements of zfs and that of the arm6 being 32-bit? as i would want to configure it as a backup sink, speed is not really an issue (especially since it will be on the opposite side of the world), but stability is (although due to the COW of zfs, i doubt i would ever loose data anyway)

there's an interesting product at http://cubox-i.com/table/ - basically a quad core 64-bit arm with 2gb ram (of course it's basically the cost of 4 raspberry pi's tho...) which might actually make zfs on a small box useable.
Posts: 4
Joined: Sat Aug 25, 2012 7:47 am
by andrum99 » Sun Jan 12, 2014 11:12 am
rayb0rn wrote:Hello, I am running Raspbian GNU/Linux 7 on my Pi.

I have read through all of the replies, please can someone clarify which distribution package I should be installing from http://zfsonlinux.org/ instead of going down the zfs-fuse route inorder to setup ZFS.

Thank you

None of them, since they are not compiled for ARM. You would need to download the source and build it yourself, either on the Pi or in a suitable build environment set up for cross-compiling. (Or perhaps in an emulated Raspberry Pi on a PC).

(Or just live with zfs-fuse from the official Raspbian repository).
My blog, with lots of Raspberry Pi goodness: andrum99.blogspot.co.uk
Posts: 215
Joined: Fri Jul 20, 2012 2:41 pm
by marked » Tue Jan 14, 2014 9:48 pm
yee379 wrote:could someone comment as to the 64-bit requirements of zfs and that of the arm6 being 32-bit? as i would want to configure it as a backup sink, speed is not really an issue (especially since it will be on the opposite side of the world), but stability is (although due to the COW of zfs, i doubt i would ever loose data anyway)

I have run ZFS using both zfs-fuse and zfsonlinux, on an external USB WD Elements SE 500GB portable hard drive.
I only ever managed to get 2MB/s write (ZFSoL) on a CIFS networked share (ZFS smbshare=on), and around 20MB/s directly on the pi. I haven't tried to do any zfs send/recv, or NFS mounts, so can't comment on this. However you can see from this that it is highly crippled in speed due to the 32bt-64bit and 64-32bit conversions. The only way that I can see this being successful at higher speeds (bearing in mind limitations of the USB chain) is to offload into the GPU the computations that require 64/128bit manipulation.

As this requires the videocore iv I can't see it happening much before https://github.com/hermanhermitage/videocoreiv gets further along. Essentially watch for GPU acceleration of bi-directional streaming (e.g.crypto (openssl/cryptoluks/truecrypt)).

As for stability I generally got about 15-30mins use before cpu hit 100% when the filesystem was being read/written. When just mounted and idle it seemed stable (ie. didn't do anything until a reboot).


there's an interesting product at http://cubox-i.com/table/ - basically a quad core 64-bit arm with 2gb ram (of course it's basically the cost of 4 raspberry pi's tho...) which might actually make zfs on a small box useable.

You got me excited - but it is a quad-core 32bit ARM (a quad cortex A9), that has some 64 and 128bit extensions. I don't think 64bit chips are in the wild yet - I thought later this year, Q2? I know there was talk of them being put on massively parallel problem sets in servers, but with the recent collapse in calexeda, I am not sure who is in the running.

However the cubox does have integrated SATA and Gigabit ethernet, though I couldn't find what speeds the actual throughput is. Of course with ZFS, depending on pool usage and whether dedup is used, the more memory the better, so would be a better option all round.

so in essence, zfs on a pi is pretty much a toy, until acceleration can be used, unless you really want to delve into the zfs internals. There is also very little information in managing zfs on such limited memory. (pointers would be appreciated), but ZRAM may also be useful.
Posts: 64
Joined: Fri Jul 29, 2011 4:25 pm
by yee379 » Sun Jan 26, 2014 7:18 am
so i finally got around to trying to compile zfs on the raspberry. i cloned the spl repo and did a autogen.sh. however, on ./configure i get:

Code: Select all
checking kernel source version... Not found
configure: error: *** Cannot find UTS_RELEASE definition.


my system:

$ uname -a
Linux raspberrypi 3.10.25+ #622 PREEMPT Fri Jan 3 18:41:00 GMT 2014 armv6l GNU/Linux

$ sudo apt-cache search linux-headers
linux-headers-3.10-3-all - All header files for Linux 3.10 (meta-package)
linux-headers-3.10-3-all-armhf - All header files for Linux 3.10 (meta-package)
linux-headers-3.10-3-common - Common header files for Linux 3.10-3
linux-headers-3.10-3-rpi - Header files for Linux 3.10-3-rpi
linux-headers-3.2.0-4-all - All header files for Linux 3.2 (meta-package)
linux-headers-3.2.0-4-all-armhf - All header files for Linux 3.2 (meta-package)
linux-headers-3.2.0-4-common - Common header files for Linux 3.2.0-4
linux-headers-3.2.0-4-rpi - Header files for Linux 3.2.0-4-rpi
linux-headers-3.6-trunk-all - All header files for Linux 3.6 (meta-package)

none of the packaged linux-headers seem to be picked up by the configure script. (25+?)

any help is appreciated!
Posts: 4
Joined: Sat Aug 25, 2012 7:47 am
by marked » Sun Jan 26, 2014 9:05 am
You need the kernel headers for 3.10.25+. If you follow the guide at http://elinux.org/RPi_Kernel_Compilation you should get most of the way there. I think 3.10.25+ is the current wheezy version, 2014-01-07. (my Raspberry Pi is running the 2013-12-? version with 3.10.23+).
I think you just need 3.10.y, but you may be able to specify the explicit release version. Of course this would all be unnecessary if kernel-headers-* would be properly packaged and released.

I would suggest symlinking /usr/src/linux to a directory on an external disk/usb flash drive. Check to ensure that the modules are built against the kernel_version+. The + is important, I have had occasions where it tries to compile against kernel_version, and that fails compilation, with everything not found.

Expect to spend some significant time waiting for it to compile on the order of several hours for kernel + spl + zfs on the rPi unless you have a cross compiler setup. Overnight would probably be best, assuming no errors.

mark
Posts: 64
Joined: Fri Jul 29, 2011 4:25 pm
by yee379 » Mon Jan 27, 2014 9:24 am
marked wrote: Of course this would all be unnecessary if kernel-headers-* would be properly packaged and released.


yeah.. i didn't fancy building a new kernel from scratch, set up an external drive/nfs etc., so i just booted the thing on a kernel with headers... :)

Code: Select all
# /boot/config.txt
kernel=vmlinuz-3.10-3-rpi
initramfs initrd.img-3.10-3-rpi followkernel


but of course i hit another issue now! For both SPL and ZFS i get:

Code: Select all
$ make deb-utils deb-kmod
Installing spl-0.6.2-23_g4c99541.src.rpm
error: Architecture is not included: armv6l


so i add it to the rpm spec file (rpm/generic/spl.spec) and then

Code: Select all
make[1]: Leaving directory `/home/pi/zfs/spl'
name=spl; \
        version=0.6.2-23_g4c99541; \
        arch=`rpm -qp ${name}-${version}.src.rpm --qf %{arch} | tail -1`; \
        pkg1=${name}-${version}.${arch}.rpm; \
        fakeroot alien --bump=0 --scripts --to-deb $pkg1; \
        rm -f $pkg1
spl-0.6.2-23_g4c99541.armv6l.rpm is for architecture armv6l ; the package cannot be built on this system


any ideas?
Posts: 4
Joined: Sat Aug 25, 2012 7:47 am
by marked » Mon Jan 27, 2014 10:10 am
yee379 wrote:.. snip package build errors /snip
any ideas?

Unfortunately package build is not one of my areas, so I just bypass it with IIRC
Code: Select all
make install modules

then to make sure
Code: Select all
modprobe zfs

grepping modules for zfs and dependencies (lzo/etc..)
Code: Select all
lsmod | grep zfs

and you should find that zfs is enabled. so
Code: Select all
sudo zpool status

should exit with no pools found.

Package removal is simply reflashing the card :D

mark
Posts: 64
Joined: Fri Jul 29, 2011 4:25 pm