ShiftPlusOne
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6229
Joined: Fri Jul 29, 2011 5:36 pm
Location: The unfashionable end of the western spiral arm of the Galaxy

F2FS image available for testing

Fri May 08, 2020 7:24 pm

Hello all,

We've been experimenting a bit with making Raspbian a little more resilient to power losses and unreliable sd cards. Logging filesystems like F2FS are a bit more resilient than EXT4, so it's one of the options we're looking at.

If you would like to try an F2FS image, particularly if you've been suffering from random data corruption, please give it a go:
https://drive.google.com/file/d/1ThtPWu ... oFvTR0Hui/

Any feedback and suggestions are welcome.

Bear in mind a couple of important differences:
  • Initial boot will take a lot longer, so users running their pi headless should wait a few minutes longer. This is because F2FS resizing takes longer and can't be done in the background.
  • Every boot will take about 30 seconds longer. The reliability comes at the cost of longer fsck and mount times.
  • Do not use in production.

cjan
Posts: 841
Joined: Sun May 06, 2012 12:00 am

Re: F2FS image available for testing

Tue May 12, 2020 10:14 am

how to set in fstab?

ShiftPlusOne
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6229
Joined: Fri Jul 29, 2011 5:36 pm
Location: The unfashionable end of the western spiral arm of the Galaxy

Re: F2FS image available for testing

Tue May 12, 2020 11:04 am

All the relevant changes are already made in the image, there's nothing to change in fstab.

cjan
Posts: 841
Joined: Sun May 06, 2012 12:00 am

Re: F2FS image available for testing

Wed May 13, 2020 3:42 am

dphys-swapfile.service not happy with f2fs?

ShiftPlusOne
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6229
Joined: Fri Jul 29, 2011 5:36 pm
Location: The unfashionable end of the western spiral arm of the Galaxy

Re: F2FS image available for testing

Wed May 13, 2020 5:16 am

What does it say?

There is an issue where dphys-swapfile tries to use ftruncate to create a file, which ends up being sparse on an F2FS filesystem. Since swapfiles can't be sparse, the service fails. However, that image shouldn't have that problem.

The swap file should already exist and not need to be created. And if it needs to be adjusted, I patched dphys-swapfile to use dd instead of ftruncate. The only way I can see this going wrong is if you reinstalled it, losing the modification and deleted the swapfile. But of course it's possible I've missed something.

cjan
Posts: 841
Joined: Sun May 06, 2012 12:00 am

Re: F2FS image available for testing

Wed May 13, 2020 12:07 pm

defualts 100M is fine, but after adjust size then broken, can you test it?

ShiftPlusOne
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6229
Joined: Fri Jul 29, 2011 5:36 pm
Location: The unfashionable end of the western spiral arm of the Galaxy

Re: F2FS image available for testing

Wed May 13, 2020 1:24 pm

Ok, I see the problem. The original script has 'set -e', which will exit if any command returns non-zero. Then later they try to check a return code using $?, which of course it doesn't get to if it's non-zero. The change I made would always return non-zero to make sure that the 'dd' path is followed.

I'll fix it in the next version with a proper check.

Thanks for reporting the issue.

cjan
Posts: 841
Joined: Sun May 06, 2012 12:00 am

Re: F2FS image available for testing

Wed May 13, 2020 9:46 pm

ShiftPlusOne wrote:
Wed May 13, 2020 1:24 pm
I'll fix it in the next version with a proper check.
the next version, base(mini).img, pls.

cjan
Posts: 841
Joined: Sun May 06, 2012 12:00 am

Re: F2FS image available for testing

Wed May 13, 2020 9:57 pm

cjan wrote:
Wed May 13, 2020 9:46 pm
ShiftPlusOne wrote:
Wed May 13, 2020 1:24 pm
I'll fix it in the next version with a proper check.
the next version, base(mini).img, pls.
btw, in that case for test, maybe kernel-5.4 ?

RonR
Posts: 1199
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: F2FS image available for testing

Sun May 17, 2020 1:17 am

cvt2f2fs will convert an existing SD card running Raspbian from ext4 to f2fs without the need for a new image (or anything else other than the temporary use of a USB flash drive).

cvt2f2fs also patches /sbin/dphys-swapfile to get along with f2fs.

busywait
Posts: 63
Joined: Sat May 09, 2020 10:48 pm
Location: Southampton, UK

Re: F2FS image available for testing

Thu May 21, 2020 10:10 am

cjan wrote:
Wed May 13, 2020 9:57 pm
btw, in that case for test, maybe kernel-5.4 ?
You can install the 5.4 test kernel after booting with 'sudo rpi-update'. I don't know if this has any other side-effects at the moment, but I have noticed that Sonic Pi (audio programming environment) fails to start with an error after I run rpi-update.

I would not want kernel-5.4 in this image because I assume that is why I can't run Sonic-Pi.

busywait
Posts: 63
Joined: Sat May 09, 2020 10:48 pm
Location: Southampton, UK

Re: F2FS image available for testing

Thu May 21, 2020 1:06 pm

ShiftPlusOne wrote:
Fri May 08, 2020 7:24 pm
We've been experimenting a bit with making Raspbian a little more resilient to power losses and unreliable sd cards. Logging filesystems like F2FS are a bit more resilient than EXT4, so it's one of the options we're looking at.

If you would like to try an F2FS image...
...
Bear in mind a couple of important differences:
  • Initial boot will take a lot longer, so users running their pi headless should wait a few minutes longer. This is because F2FS resizing takes longer and can't be done in the background.
  • Every boot will take about 30 seconds longer. The reliability comes at the cost of longer fsck and mount times.
  • Do not use in production.
I put the image on to a couple of different 64GB SD cards, a Samsung Pro (class U1, a couple of years old), and a new SanDisk Extreme (class A2, a month old).

I don't have any load or hardware that has ever caused a corruption, but I was curious to see what IO speed changes there might be, with just the agnostics utility from the Raspian repo.

Code: Select all

sudo update
sudo apt install agnostics
sh /usr/share/agnostics/sdtest.sh >> pi4-SanDiskExt-f2fs.txt
Observations and opinions:
- On first boot (file system resize) I felt a need for a progress bar while resizing 64GB f2fs to reassure me it was still working properly
- Yes, the reboot time was longer, a pity, but I'm used to bigger computers, so I can live with it
- The Samsung Pro U1 card seems to have had a performance hit in random writes moving from ext4 to f2fs, but my test run on ext4 was a month ago, and only a single run recorded, so maybe not fair to compare?
- I think that f2fs might help the newer card (SanDisk Extreme, class *A2*), raising the random write speeds from max 748 IOPS to 1073 IOPS. The sequential writes range widely across runs with this card, on ext4 between 19MB/sec to 37MB/sec, and on f2fs from 6MB/sec[!!!] to 37MB/sec
- There was no stand-out difference between kernel-4.19, kernel-5.4, or kernel-5.4 64-bit with the card I tested (SanDisk Extreme A2 64GB).

Measurement ranges for different cards, file systems and kernels

Samsung Pro U1 64GB, ext4, kernel 4.19 32bit armv7l, 1 run a month ago after install + upgrade + reboot
Sequential write speed 31507 KB/sec
Random write speed 713 IOPS
Random read speed 3628 IOPS

Samsung Pro U1 64GB, f2fs, kernel 4.19 32bit armv7l, 2 runs, after first boot before and after sudo apt full-upgrade
Sequential write speed 19913-23813 KB/sec
Random write speed 550-611 IOPS
Random read speed 3566-3666 IOPS

Sandisk Extreme A2 64GB, ext4, kernel-4.19 32bit armv7l, 2 runs, after first boot before and after sudo apt full-upgrade
Sequential write speed 16688-37513KB/sec
Random write speed 567-756 IOPS
Random read speed 2533-2847 IOPS

Sandisk Extreme A2 64GB, f2fs, kernel-4.19 32bit armv7l, 6 runs before and after sudo apt full-upgrade
Sequential write speed 5409-36880 KB/sec
Random write speed 759-1073 IOPS
Random read speed 2440-2741 IOPS

Sandisk Extreme A2 64GB, f2fs, kernel-5.4 32-bit armv7l, 2 runs after rpi-update + reboot
Sequential write speed 25157-37321 KB/sec
Random write speed 1013-1055 IOPS
Random read speed 2714-2739 IOPS

Sandisk Extreme A2 64GB, f2fs, kernel-5.4 64-bit aarch64, 2 runs after adding arm_64bit=1 to /boot/config/txt + reboot
Sequential write speed 32653-32899 KB/sec
Random write speed 835-1059 IOPS
Random read speed 2720-2753 IOPS

UPDATE:
I'm going to backtrack on the A2 card speed increase that I thought I saw with f2fs. I've done a couple of extra runs back on ext4 again also hit random writes at >1000 IOPS, so this card doesn't seem to *suffer* with f2fs, and/or my setup was to busy for benchmarks, and/or ext4 perks up after a rest :)

I'm going to put the f2fs image back on and run with it now - this is just a little toy machine behind a TV, but it gets a little use most days.
Last edited by busywait on Thu May 21, 2020 4:38 pm, edited 1 time in total.

ShiftPlusOne
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6229
Joined: Fri Jul 29, 2011 5:36 pm
Location: The unfashionable end of the western spiral arm of the Galaxy

Re: F2FS image available for testing

Thu May 21, 2020 3:18 pm

Fixed version of dphys-swapfile should be in apt now, so manual patching shouldn't be required.
busywait wrote: - Yes, the reboot time was longer, a pity, but I'm used to bigger computers, so I can live with it
That may be a bug rather than a feature. I'm yet to look into why it's doing a full scan on each boot.

busywait wrote: - On first boot (file system resize) I felt a need for a progress bar while resizing 64GB f2fs to reassure me it was still working properly
I'll take a look, but I'm not sure it's possible. As an alternative, maybe scrolling the last few lines of the resize command could help show that something is still happening.

chrisy
Posts: 21
Joined: Sat May 25, 2013 7:34 pm

Re: F2FS image available for testing

Thu May 28, 2020 11:03 pm

I've been using f2fs on my Pis for years - it's pretty much the first thing I do . Access/boot times always seem to be quicker. I had to set up a new Pi recently but hadn't noticed this test image or I would have given it a try - would have saved me having to migrate manually.

busywait
Posts: 63
Joined: Sat May 09, 2020 10:48 pm
Location: Southampton, UK

Re: F2FS image available for testing

Mon Jun 08, 2020 3:31 pm

The F2FS image seems to "just work" apart from the issues already noted. Is there anything to look out for in particular, or other feedback that would be helpful?
ShiftPlusOne wrote:
Thu May 21, 2020 3:18 pm
busywait wrote: - Yes, the reboot time was longer, a pity, but I'm used to bigger computers, so I can live with it
That may be a bug rather than a feature. I'm yet to look into why it's doing a full scan on each boot.
This turned out to be an annoying inconvenience this morning - I had to wait for the Pi to boot from a 64GB card while my daughter was waiting to do some schoolwork - it seemed longer with someone else waiting :/ Not sure why I'd powered it down anyway.

Is it an fsck? Is there a safe config change that avoids the scan?
ShiftPlusOne wrote:
Thu May 21, 2020 3:18 pm
busywait wrote: - On first boot (file system resize) I felt a need for a progress bar while resizing 64GB f2fs to reassure me it was still working properly
I'll take a look, but I'm not sure it's possible. As an alternative, maybe scrolling the last few lines of the resize command could help show that something is still happening.
That would be perfect - just some sense that helpful activity is in progress.

User avatar
bensimmo
Posts: 4620
Joined: Sun Dec 28, 2014 3:02 pm
Location: East Yorkshire

Re: F2FS image available for testing

Mon Jul 06, 2020 7:52 am

is this still being investigated, I was going to have a look for my sensors and lora setup (were they are logging and power may be lost or pulled)

What are the benefits?
What are the downsides?

What should I be looking for as ext4 has been fine for me just yanking the power on my Zero etc.

chrisy
Posts: 21
Joined: Sat May 25, 2013 7:34 pm

Re: F2FS image available for testing

Fri Jul 10, 2020 2:55 pm

busywait wrote:
Mon Jun 08, 2020 3:31 pm
ShiftPlusOne wrote:
Thu May 21, 2020 3:18 pm
That may be a bug rather than a feature. I'm yet to look into why it's doing a full scan on each boot.
This turned out to be an annoying inconvenience this morning - I had to wait for the Pi to boot from a 64GB card while my daughter was waiting to do some schoolwork - it seemed longer with someone else waiting :/ Not sure why I'd powered it down anyway.

Is it an fsck? Is there a safe config change that avoids the scan?
Try finding the line in /etc/fstab and changing the last number to a 0.

I've not yet tried this image but the 0 in fstab tells it not to scan. I suspect it is set to something else.

busywait
Posts: 63
Joined: Sat May 09, 2020 10:48 pm
Location: Southampton, UK

Re: F2FS image available for testing

Mon Jul 13, 2020 12:42 pm

chrisy wrote:
Fri Jul 10, 2020 2:55 pm
busywait wrote:
Mon Jun 08, 2020 3:31 pm
ShiftPlusOne wrote:
Thu May 21, 2020 3:18 pm
That may be a bug rather than a feature. I'm yet to look into why it's doing a full scan on each boot.
Is it an fsck? Is there a safe config change that avoids the scan?
Try finding the line in /etc/fstab and changing the last number to a 0.
Thanks - that works. It disables *any* automated fscking on that partition I think?

Since my original post I've re-imaged the SD card and used PINN to install Raspbian 32 bit along-side Manjaro. I converted the Manjaro partition to F2FS, and that behaves better - it fscks the partition every time it detects a kernel change, but not every boot.

ShiftPlusOne
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6229
Joined: Fri Jul 29, 2011 5:36 pm
Location: The unfashionable end of the western spiral arm of the Galaxy

Re: F2FS image available for testing

Fri Jul 31, 2020 10:43 pm

bensimmo wrote:
Mon Jul 06, 2020 7:52 am
is this still being investigated, I was going to have a look for my sensors and lora setup (were they are logging and power may be lost or pulled)

What are the benefits?
What are the downsides?

What should I be looking for as ext4 has been fine for me just yanking the power on my Zero etc.
Still investigated, but sitting on a shelf for the time being. The questions you're asking is what this test image is meant to help figure out.

I'd mostly like to hear from people who frequently experience data corruption on their SD cards.

I think I'll try using dm-flakey and dd to simulate data corruption next time, since I've never actually experienced it first hand it in any real world scenario.

Return to “Raspberry Pi OS”