piman314
Posts: 15
Joined: Tue Feb 09, 2016 11:57 am

64-bit EXT4: breaking the 16TB limit

Sun Sep 16, 2018 6:02 am

Morning folks, I have some old pis hanging around and the need to create an occasional backup system for relatively non-vital backups, and my old 24TB RAID array now going spare and unused. So I thought I'd get one of the Pis to do backups over rsync to the array. It would be slow, limited mostly by network speed than USB, and it seems to work OK-ish with 40MBytes/s write speeds on RAID6, better than I'd expected and higher than my 10MBytes/s network speed limit, all perfectly adequate for this particular use case.

When creating a RAID array on the Pi via USB and mdadm, I was unable to mount a new EXT4 partition greater than 16TB, presumably because Raspbian is using 32-bit filesystems despite creating it on a new disc that was about 24TB in size. The version of the EXT4 toolset seems to be 1.43.4 which is higher than that needed to create and use 64-bit filesystems. The manual pages for tune2fs, resize2fs and mkfs.ext4 don't seem to mention the 64-bit options. I'm using the system at the moment with a 16TB filesystem successfully but would prefer to grow that if possible rather than fuzt around with multiple partitions (this is supposed to be a blisteringly simple backup system).

Has anyone successfully created a larger than 16Tbyte file system using ext4 linux on a pi? I did originally create this on a 32-bit Pi 2 so maybe I should re-build the system with a Pi 3 (it's now running on a Pi 3B+ but using the SD card image created on a Pi 2).

ejolson
Posts: 2163
Joined: Tue Mar 18, 2014 11:47 am

Re: 64-bit EXT4: breaking the 16TB limit

Sun Sep 16, 2018 6:19 pm

piman314 wrote:
Sun Sep 16, 2018 6:02 am
Has anyone successfully created a larger than 16Tbyte file system using ext4 linux on a pi?
Are you mounting your 24TB array as an iSCSI device?

According to this blog, it will take more than 3GB RAM to run fsck on such a large ext4 filesystem. That blog also lists the options necessary for creating a greater-than-16TB ext4 filesystem. Note there are additional overheads with the resulting 48-bit indexing. Be sure to check the comments afterwards to make sure you have an adequate number of inodes.

I used a Pi 3B+ with an 8TB external USB drive formatted with BTRFS. It worked well, however, if I wanted to rebalance or dedupe the filesystem I would first remount it on a PC with sufficient memory. The same could be done before running fsck on an ext4-formatted 24TB RAID array.

One idea is to use LVM to create four 6TB volumes and format each of those with separate ext4 filesystems. You could also experiment with thinly provisioned volumes and create five, six, seven or more 6TB filesystems.

If it were my decision, I'd format the whole 24TB using BTRFS. I find the live snapshots and subvolumes to be convenient when storing backups. See this thread for more information.

piman314
Posts: 15
Joined: Tue Feb 09, 2016 11:57 am

Re: 64-bit EXT4: breaking the 16TB limit

Mon Sep 17, 2018 7:45 am

ejolson wrote:
Sun Sep 16, 2018 6:19 pm
Are you mounting your 24TB array as an iSCSI device?
No, it's hung off the back in two USB JBOD containers, one of the tests I'm doing is to see if the Pi's USB is solid enough to keep the array together, for a Pi 2 the answer is no, but the Pi 3 has written 6TBytes and only broke up the array once. As the backups will be incremental after the initial backup is done, it's not going to be backing up more than a few gig in future so the Pi 3 should easily be good enough. If the backups were vital of course I wouldn't be doing it this way.

I created the 24TB array then tried to format EXT4 but it failed with a "short read" when writing the superblock, it looked like the tools were creating the filesystem properly but getting some failures from further down the software stack, this week when the backed-up main fileserver isn't busy I'll be recreating the array and experimenting some more. The versions of the toolchain that I'm using ought to be creating 64bit when they see the size of the device, and seemed to be doing so, but coughed halfway through the process. I'll see if I can create the array on a full-fat system then bring it back to the pi, and various other combinations. Failing that I'll blow the dust off some unused intel gear and find out something else to do with the pi.
ejolson wrote:
Sun Sep 16, 2018 6:19 pm
If it were my decision, I'd format the whole 24TB using BTRFS. I find the live snapshots and subvolumes to be convenient when storing backups.
That's what I do for my main backups using different hardware and then pass the backups across to an identical system off-site over a VPN, I expire snapshots after a month if they're not a Saturday snapshot, and expire Saturday snapshots after a year. It's saved my bacon a few times. This pi-based system is just for non-vital backups using cobbled-together hardware I already own.

ejolson
Posts: 2163
Joined: Tue Mar 18, 2014 11:47 am

Re: 64-bit EXT4: breaking the 16TB limit

Mon Sep 17, 2018 2:29 pm

piman314 wrote:
Mon Sep 17, 2018 7:45 am
No, it's hung off the back in two USB JBOD containers
How many disks? Is there any hardware RAID built into the JBOD containers?

Assuming 24MB/sec sync speed (simulataneous reading and writing over USB) it would seem to take about twelve days for a Pi to synchronise a bunch of disks using Linux software RAID.

If no hardware RAID is available, I would recommend using the two JBOD containers to create a mirror-redundant disk pool for BTRFS. Use a command such as

# mkfs.btrfs -L Backup -m raid1 -d raid1 /dev/sda1 /dev/sdb1 ...

where the ... are replaced by the remaining disks available. Note that the advantage of integrating RAID functionality into the filesystem is that only allocated sectors need to be mirrored--a big saving when working with 24TB arrays.

Depending on how many disks you have and their perceived reliability, it might be tempting to try out the experimental RAID6 functionality. Unfortunately RAID5 and RAID6 have known bugs even with the most recent kernels. More information is available here.

Development of BTRFS has been slow, but continuous, in recent years. The filesystem is complicated and apparently requires significant technical skill to develope. At the same time people have been porting Sun ZFS to work on Linux.

I do not have the technical understanding to fully characterise the design tradeoffs between BTRFS and ZFS. In the case of Raspberry Pi, ZFS deduplication requires more memory than is available, whereas BTRFS has batch duplication which could be performed by plugging the array into a capable PC once a month. For this reason BTRFS seems a better choice.

Compared to Linux software RAID, BTRFS seems less likely to accidentally remove a disk from the array. If not all drives are online, BTRFS won't mount the remaining ones unless a force option is specified. Linux RAID, on the other hand, immediately removes the missing drives and then mounts the array in degraded mode. I think the behaviour of BTRFS is preferable, especially when the disks are connected through USB.

Since your primary backup is already using BTRFS, were you particularly wanting to do something different with the older array?

piman314
Posts: 15
Joined: Tue Feb 09, 2016 11:57 am

Re: 64-bit EXT4: breaking the 16TB limit

Mon Sep 17, 2018 4:28 pm

Sorry but this is getting a little off the original topic (I've done BTRFS RAID in the past, not going there again), I'm looking to see if anyone has any experience with firing up large EXT4 filesystems on a Pi, it looks like a "no" at the moment, I'll play around with it a little more and post back any results I get once I find out where the "short read" problem is creeping in. It works fine with a 16TB partition but I prefer not to partition my arrays at all if I can get away with it.

ejolson
Posts: 2163
Joined: Tue Mar 18, 2014 11:47 am

Re: 64-bit EXT4: breaking the 16TB limit

Mon Sep 17, 2018 7:04 pm

piman314 wrote:
Mon Sep 17, 2018 4:28 pm
Sorry but this is getting a little off the original topic (I've done BTRFS RAID in the past, not going there again), I'm looking to see if anyone has any experience with firing up large EXT4 filesystems on a Pi, it looks like a "no" at the moment, I'll play around with it a little more and post back any results I get once I find out where the "short read" problem is creeping in. It works fine with a 16TB partition but I prefer not to partition my arrays at all if I can get away with it.
Since you are posting on the Raspbian forum, it would appear you are using a 32-bit kernel. Note, however, that the main reason Raspbian uses a 32-bit kernel on the otherwise 64-bit Pi 3B and 3B+ is for compatibility with earlier versions of the Pi and the Zero. Such compatibility is useful in educational environments where SD cards may be swapped between Pi computers of different types and vintage.

To use a 24 terabyte RAID6 array, it might be expedient to boot a fully 64-bit operating system. I would start by experimenting with this 64-bit Gentoo image, which has had some pretty positive reviews.

piman314
Posts: 15
Joined: Tue Feb 09, 2016 11:57 am

Re: 64-bit EXT4: breaking the 16TB limit

Mon Sep 17, 2018 7:21 pm

ejolson wrote:
Mon Sep 17, 2018 7:04 pm
To use a 24 terabyte RAID6 array, it might be expedient to boot a fully 64-bit operating system. I would start by experimenting with this 64-bit Gentoo image, which has had some pretty positive reviews.
Ah Gentoo, haven't used that for probably 10 years, that will be interesting to have a poke at again ;-)

It'll be worth a try so thanks for the suggestion and link, it'll only need the basics and RAID so no X etc so with any luck will work.

ejolson
Posts: 2163
Joined: Tue Mar 18, 2014 11:47 am

Re: 64-bit EXT4: breaking the 16TB limit

Tue Sep 18, 2018 12:15 am

piman314 wrote:
Mon Sep 17, 2018 7:21 pm
It'll be worth a try so thanks for the suggestion and link
You're welcome. There are reports that the Raspberry Pi hardware is only able to deal with 6 high-speed USB2 devices at a time. Assuming your RAID6 24TB array is in the form of five 8TB drives or six 6TB drives things should be fine. However eight 4TB drives may not work out so well. I'm not sure if the presence of full and low speed USB keyboards, mice, printers, scanners and web cameras would affect the HD limit or not. At any rate, this is something to be aware of if random USB drives keep going offline during use.

piman314
Posts: 15
Joined: Tue Feb 09, 2016 11:57 am

Re: 64-bit EXT4: breaking the 16TB limit

Tue Sep 18, 2018 5:31 am

ejolson wrote:
Tue Sep 18, 2018 12:15 am
There are reports that the Raspberry Pi hardware is only able to deal with 6 high-speed USB2 devices at a time. Assuming your RAID6 24TB array is in the form of five 8TB drives or six 6TB drives things should be fine. However eight 4TB drives may not work out so well. I'm not sure if the presence of full and low speed USB keyboards, mice, printers, scanners and web cameras would affect the HD limit or not. At any rate, this is something to be aware of if random USB drives keep going offline during use.
On the Pi 3 B+, it's been running 8 drives connected via two USB to 4-disc SATA bridges rather than individual USB-to-SATA bridges plugged into USB hubs, so lsusb -v --tree only shows three devices, the ethernet interface and the two SATA bridges, it's now done 7 TBytes in RAID 6 so fairly solid.

ejolson
Posts: 2163
Joined: Tue Mar 18, 2014 11:47 am

Re: 64-bit EXT4: breaking the 16TB limit

Tue Sep 18, 2018 9:04 pm

piman314 wrote:
Tue Sep 18, 2018 5:31 am
On the Pi 3 B+, it's been running 8 drives connected via two USB to 4-disc SATA bridges rather than individual USB-to-SATA bridges plugged into USB hubs, so lsusb -v --tree only shows three devices, the ethernet interface and the two SATA bridges, it's now done 7 TBytes in RAID 6 so fairly solid.
That's good news. Those USB enclosures sound interesting. Would you mind telling the manufacturer and model number? Was there any reason you chose them over similar ones?

piman314
Posts: 15
Joined: Tue Feb 09, 2016 11:57 am

Re: 64-bit EXT4: breaking the 16TB limit

Wed Sep 19, 2018 8:27 am

ejolson wrote:
Tue Sep 18, 2018 9:04 pm
That's good news. Those USB enclosures sound interesting. Would you mind telling the manufacturer and model number? Was there any reason you chose them over similar ones?
Just some very old StarTech units, they're so old they only have USB2, but they also have E-SATA multipliers which is why I bought them as I used to have them connected via E-SATA although that turned out to not be very quick due to hardware limitations on the available cards, but they were quick enough at the time. I don't use them any more for anything serious.

Return to “Raspbian”