ext2+nilfs2 on sd-card

[filesystem]

Dear forum,

for those who are interested in filesystem performance,
particular flash memory (usb flash drive, sd card):

i've set up my Raspberry with an ext2 filesystem for root ("/"),
and an large nilfs2 filesystem for /usr, /var etc. via symbolic links (setup done on other linux pc)

Here my personal benchmark results using appended filesystem.c file, which creates (and deletes) files with geometrically distributed size - similar to size distribution of my pc's file size distribution. Every 10 MB the performance in MB/s is written.

Ext2/noatime
10485.760000 MB/s
20971.520000 MB/s
31457.280000 MB/s
13.976355 MB/s
10.483663 MB/s
12.580396 MB/s
4.893028 MB/s
5.592033 MB/s
5.550958 MB/s
4.992981 MB/s
4.436112 MB/s
4.839395 MB/s
2.900255 MB/s
3.123351 MB/s
2.912657 MB/s
2.541964 MB/s
2.097127 MB/s
2.220488 MB/s
2.289967 MB/s
as you can see, perfomance gets worse over time, with ext2.

now:
nilfs2/noatime
10485.760000 MB/s
10.480520 MB/s
10.482266 MB/s
13.976355 MB/s
10.483663 MB/s
10.484013 MB/s
9.173893 MB/s
10.484449 MB/s
10.484595 MB/s
10.484712 MB/s
10.484807 MB/s
10.484886 MB/s
9.736082 MB/s
9.786057 MB/s
9.829786 MB/s
9.868370 MB/s
9.902668 MB/s

... superb - an that with approx. 50% of files being as large as 4096 bytes(!)

hope, that helps deciding wich filesystem to use

PS: dont forget to install nilfs2-tools and to move var/swapfile to root ("/") and to create an ext2/3/4 filesystem for this swapfile because nilfs2 is not able to support swap files.

[filesystem]

Dear reader,

I've found some this thread: "What's The Fastest Linux Filesystem On Cheap Flash Media?" on linuxplanet and was courious about the recommendation "ext4".

Now, I've did my tests again using the same C programm and an old, rarely used "sandisk cruzer micro 4.0gb". In the beginning ext4 had a very good performance. Asymptotically towards approx. 7 MB/s. But later - after approx 197*10MB written - it dropped to 3 MB/s.

Did this with nilfs2 - constant performance of approx. 6 MB/s.

f2fs seems not very interesting to me, too less development activity in linux kernel, still experimental.

[filesystem]

here some comments:
(tested with 3.13.0-46-generic #79-Ubuntu SMP)
- ext2 and ext4 (w/o journal) lose are dropping performance near 1570 gb
- before dropping perf., ext4 seems approximating to 7 MB/s from top
- before doppring perf., ext2 seems approximating to 5 MB/s from bottom
- forget about btrfs, very volatile performance
- f2fs: volatile performance, but steady above nilfs2 (+0.5 MB/s in the long run)
- nilfs2 performance looks like a line. My personal recommendation.

Note: perhaps performance drop of ext2+4 has to do more with the usb stick than with their filesystem design. I do not have tested dozens of flash drives.

[jwatte]

FWIW: I'm a fan of nilfs2. I lost a full disk a few years back to this FS, which was a bummer, but the developer has found and fixed that bug, and I've been meaning to set one up again. I now have the perfect case!

My RPi controls a robot, which logs telemetry to files in /var/tmp, as well as recording the camera in a low-resolution mp4 file into the same partition.

I have purchased a 64 GB USB stick which shows up as /dev/sda.
I'f used fdisk to create a new partition table with a single partition /dev/sda1 of type 83 on /dev/sda.
After installing nilfs-tools (not nilfs2-tools!) I can sudo mkfs -t nilfs2 /dev/sda1
I can then mount -t nilfs2 /dev/sda1 /var/tmp
I must then chmod 777 /var/tmp/. -- very important!
Then I can add this line to /etc/fstab:
/dev/sda1 /var/tmp nilfs2 defaults,noatime 0 2

Now, each time I boot, /var/tmp gets mounted on this USB stick, using nilfs2, with good performance, and I don't have to write to the internal sdcard anymore!

/var/log is still being written by various system services, so if saving sdcard write wear is of crucial importance, copying all of /var onto the file system, and mounting it over /var, would be even better. I don't need to do that, though.

Hope this helps someone else!

[jahboater]

I assume you are using the same SSD for all the experiments, otherwise the results are meaningless.

Ext4 has delayed allocation/extents so most files will be contiguous and there should be less fragmentation. Filefrag -v <myfile> tells all.

You can put these filesystems in virtual memory with:

Code: Select all

tmpfs /tmp  tmpfs defaults,noatime 0 0
tmpfs /var/log  tmpfs defaults,noatime,size=64m 0 0

[kwag]

@jwatte,

That's exactly what I did on my Pi.
I mounted the whole /var as nilfs2 on an external 32GB thumb drive, and there I run a full blown forum (FUDForum) with lightpd web server and SQLite database, and all works via two reverse tunnels back home from two public VPS's, for redundancy.
Works like a charm!

-Karl

[filesystem]

Oh, well...

I've got at least 4 times problems with flash storage the last stagionis (think vivaldi), 3x with tran*cend flash cards/usb pen drives and with one cor*air voyager usb pen drive, all of them returning wrong data blocks without raising an error, which lead to mysterious filesystem and/or application error messages first.

So the conclusion to be safe here is:
If in doubt, use btrfs, it has at least checksums on all data+metadata.

On my sandisk microsdhc with my raspi2 I'm still moderate optimistic and use f2fs. I did not have any problems with kingston, sandisk and samsung sdhc-flash storage so far.

PS: The claimed checksum of nilfs2 is not verified on every read, perhaps on mount only.

[filesystem]

PS: swap file on nilfs2 may be possible by

losetup -f /dir/swapfile
swapon /dev/loopX