f2fs : why not distribution with root fs as f2fs

[dgerman]

Why isn't the root file system of the distribution image an f2fs file system.
Since raspberry pi is based on an SD card and f2fs was specifically designed with them in mind,
shouldn't the distributed image have root on an f2fs?

If not maybe /var should be a separate f2fs partition?

Sorry for the poorly worded subject.

[ShiftPlusOne]

Ext4 is working well. F2fs hasn't been proven to be a worthwhile. When people have tried it for Android, they found that performance degrades over time, requiring them to reformat. Then they went back to Ext4. If you look on XDA, the general view of it isn't positive. To consider switching, we'd need a bit more than the fact that it's meant to be designed with flash storage in mind. What are the measured long term benefits?

[incognitum]

A more interesting question would be whether or not it is more resilient to file system corruption on unclean shutdowns.
Could imagine log structured file systems scoring better at that, than traditional file systems that modify data in-place.

[blackshard83]

Ext4 is working well. F2fs hasn't been proven to be a worthwhile. When people have tried it for Android, they found that performance degrades over time, requiring them to reformat. Then they went back to Ext4. If you look on XDA, the general view of it isn't positive. To consider switching, we'd need a bit more than the fact that it's meant to be designed with flash storage in mind. What are the measured long term benefits?

I control some hundred of devices and a small amount of them run their caching filesystem with f2fs on sdcard.

Performance is degrading often very fast on cheap 32 GB USB sticks formatted as ext4 devices, but is not degrading on f2fs system over cheap 8 GB sdcards. f2fs systems are also much more stressed than ext4 (at least 700 Megabytes wrote per-day, in small and large blocks and an average of 38 GB read per-day).

The only real problem so-far we encountered with f2fs is that if you don't cleanly shutdown your filesystem, it loses most of the recently written data.

[ShiftPlusOne]

Performance is degrading often very fast on cheap 32 GB USB sticks formatted as ext4 devices, but is not degrading on f2fs system over cheap 8 GB sdcards. f2fs systems are also much more stressed than ext4 (at least 700 Megabytes wrote per-day, in small and large blocks and an average of 38 GB read per-day).

Thanks, those are good data points.

The only real problem so-far we encountered with f2fs is that if you don't cleanly shutdown your filesystem, it loses most of the recently written data.

Is that regardless of whether you 'sync' beforehand. Or is data safe as long as applications fsync appropriately?

IIRC, the default behaviour of linux is to flush data that has been cached for more than 30 seconds, using too much RAM and block IO if cached data is taking up more than a certain percentage of total RAM. So there shouldn't be too much of a difference in behavior after data is synced.

You shouldn't lose more than 30 seconds worth of changes with either filesystem and very little for applications which handle IO safely (like dpkg tries).

I guess complexity comes in when the controllers on the SD cards themselves lie about what they're doing. What happens when the card claims a write has occurred when it hasn't really. How does wear leveling interact with random power resets. Do you get in states where one physical block is addressed incorrectly? How can a filesystem address issues if the behavior of different SD cards can vary widely and isn't trustworthy? These aren't just rhetorical questions, I'm actually curious, since I haven't seen much information about this, given the proprietary nature of the inner workings of SD cards.

I might have gone off on too much of a tangent there, sorry.

[hommar]

The only real problem so-far we encountered with f2fs is that if you don't cleanly shutdown your filesystem, it loses most of the recently written data.

I use elevator=noop for minimize this problem, because f2fs have own buffer, and doesn't need a second system buffer

PS You can test my release for russian users with f2fs and zswap

PPS I use f2fs-tools from buster raspbian release becouse I think new f2fs inside kernel need new outside f2fs-tools

[incognitum]

I guess complexity comes in when the controllers on the SD cards themselves lie about what they're doing. What happens when the card claims a write has occurred when it hasn't really. How does wear leveling interact with random power resets. Do you get in states where one physical block is addressed incorrectly?

And that problem is made worse by the fragility of ext4.
It does not have redundant copies of file system meta data.
When using that even a single block going corrupt/missing can give you an awful lot of trouble, if it happened to contain the directory listing of say a system folder or your home.

If only recently written data goes missing with F2FS, but filesystem itself remains intact, that would be an improvement.

[dgerman]

Ext4 is working well. F2fs hasn't been proven to be a worthwhile. When people have tried it for Android, they found that performance degrades over time, requiring them to reformat. Then they went back to Ext4. If you look on XDA, the general view of it isn't positive. To consider switching, we'd need a bit more than the fact that it's meant to be designed with flash storage in mind. What are the measured long term benefits?

dgerman responded:
I wonder what version "they" were using and if recent changes have addressed those issues.

XDA:"F2FS Put to the Test Against EXT4" 3/31/2014. ...database operations were consistently around an order of magnitude faster on F2FS than on EXT4.... F2FS is actually about 20% slower in sequential reads ...

This actually suggests, as did the second part of my posting, should the distributed image contain a separate partition for /var (whatever the file system )

XDA: Interview with Francisco Franco... September 3, 2017. quote a Google engineer which says that based on their test, F2FS doesn’t perform any faster than the ext4 . (Kinda of a poor reference without any details)



Have newer SD cards demonstrated better speeds.

Maybe it's time to take another look?

[Joel_Mckay]

XDA: Interview with Francisco Franco... September 3, 2017. quote a Google engineer which says that based on their test, F2FS doesn’t perform any faster than the ext4 . (Kinda of a poor reference without any details)

Have newer SD cards demonstrated better speeds.

Maybe it's time to take another look?

The last time I took a deep dive into SDCARD performance, and specifically the more recent Sandisk cards. Pointed to several reasons most people anecdotally recall F2FS doesn't really perform much better than ext4:

1. Primarily F2FS was intended for raw memory chips without the built-in wear-leveling controller of modern sdcard or SSD. Thus, wear leveling in the OS only wastes CPU time, and can't by definition know what the sdcard or SSD controller has already done internally.

2. Without low level chip operation access (specifically truncation), F2FS can't really improve performance much. sdcards are more than a simple flash chip, but are easily conflated at the storage layer... Note since many versions of "ATA Trim" on dodgy SSD hardware was unstable, the likelihood of a full disk-crash became legendary among the users even when using ext4/btfs/ntfs/zfs.

3. Many U1 and Class 10 cards are i/o limited to only 80MHz overclock on some Pi... faster flash sdcards exist... but so far I can't seem to get them to run any faster on a Pi3B+... and the documentation seems to support this constraint.... Bottle meet Neck... F2FS might be faster for raw chips, but still needs to queue writes like every other format.

4. Some people do not setup the kernel cache eviction policy with flash queues in mind, and many cards are limited to a 4MB buffer.

The fastest setup on consumer hardware I have seen (excluding SSD RAID), was bench-marked using ext4 with an external journal_data_ordered setup.
https://raid6.com.au/posts/fs_ext4_external_journal/

Cheers,
J

[hommar]

1. Primarily F2FS was intended for raw memory chips

I think you do not right.
F2FS is not for raw memory chips

The motive for F2FS was to build a file system that, from the start, takes into account the characteristics of NAND flash memory-based storage devices (such as solid-state disks, eMMC, and SD cards), which are widely used in computer systems ranging from mobile devices to servers.

Jaegeuk Kim, the principal F2FS author, work in Samsung, one of main developers of chips for SDs and SSDs (not only flash-memory, but controllers too)

PS https://lkml.org/lkml/2012/10/5/205

PPS https://www.kernel.org/doc/Documentatio ... s/f2fs.txt

[blackshard83]

Is that regardless of whether you 'sync' beforehand. Or is data safe as long as applications fsync appropriately?

IIRC, the default behaviour of linux is to flush data that has been cached for more than 30 seconds, using too much RAM and block IO if cached data is taking up more than a certain percentage of total RAM. So there shouldn't be too much of a difference in behavior after data is synced.

You shouldn't lose more than 30 seconds worth of changes with either filesystem and very little for applications which handle IO safely (like dpkg tries).

the application running on the devices writes in block of 64 kbytes and each block is synced using os.fdatasync() (it is a python application). fdatasync() is supposed to force immediate write of data but does not sync filesystem metadata.
When the file is completed, an os.fsync() call is issued (force write of data and metadata).
This proved to be effective with ext4: no data is lost in case of a power outage. This is not true when we use f2fs, data can be lost even if it has been "wrote" many minutes before, so it seems that f2fs is not flushing with conventional userland sync calls nor on a time basis.

Invoking sync command from bash seems instead to be really effective even with f2fs, but it is too coarse for our use case.

[blackshard83]

The last time I took a deep dive into SDCARD performance, and specifically the more recent Sandisk cards. Pointed to several reasons most people anecdotally recall F2FS doesn't really perform much better than ext4:

1. Primarily F2FS was intended for raw memory chips without the built-in wear-leveling controller of modern sdcard or SSD. Thus, wear leveling in the OS only wastes CPU time, and can't by definition know what the sdcard or SSD controller has already done internally.

I think this is wrong, f2fs is supposed to be used on flash memory with a Flash Translation Layer (so not raw).

3. Many U1 and Class 10 cards are i/o limited to only 80MHz overclock on some Pi... faster flash sdcards exist... but so far I can't seem to get them to run any faster on a Pi3B+... and the documentation seems to support this constraint.... Bottle meet Neck... F2FS might be faster for raw chips, but still needs to queue writes like every other format.

I don't understand the sense of "f2fs might be faster..." sentence in relation with the clock of the device

[jdb]

One point worth considering is that SD card manufacturers may be specifically optimising their FTL behaviours for workloads like ext4 and FAT - small in-place writes for metadata, arbitrary linear writes of file data.

F2FS will have a completely different access pattern, namely avoiding repeated writes to the same sector by treating the whole disk as a set of circular buffers. It may well have been created (and work well) for simple FTLs that have static wear leveling, but more complex FTLs with usage profiling may "guess" suboptimally when faced wtih a log-based workload.

[Joel_Mckay]

I don't understand the sense of "f2fs might be faster..." sentence in relation with the clock of the device

I suspect the communication overhead with F2FS may appear more efficient with a linear write in some scenarios (like avoiding partial erasure that can cause a read, modify, and write routine). For example, because of the internal buffer on modern under-spec'ed sdcard hardware, sparsely timed larger write-bursts could appear to perform faster as the internal queue will always finish draining before the next write operation is queued from the OS due the inherent transfer latency. This phenomena has been documented on some high-end enterprise SSD setups, that are able to handle the full throughput of a pci bus.

One point worth considering is that SD card manufacturers may be specifically optimising their FTL behaviours for workloads like ext4 and FAT - small in-place writes for metadata, arbitrary linear writes of file data.

Good point, this certainly has been documented with the sector alignment issues, and is the primary reason to always keep a copy of the original OEM partition information that comes on a drive itself. The most common mistake is setting a SSD 4kiB sector layout to not be aligned, as some models do seem to follow a layout and buffer optimized with FAT or NTFS writes in mind. Many usually specify the vendor specific disk utilities. Yet in general one can't assume anything, and must still copy the existing format layout on the drive media given lurking nonstandard platform specific optimizations (I use Sandisk sdcards because all of the drive memory actually exists.. not all manufactures include areas they expect to be unused).
This is a serious consideration, as it not only can significantly slow things down, but also tends to prematurely wear out the storage with excessive erasures and writes.

[hommar]

One point worth considering is that SD card manufacturers may be specifically optimising their FTL behaviours for workloads like ext4 and FAT - small in-place writes for metadata, arbitrary linear writes of file data.

Different Applications Require Different Cards (and more on this site)

I use SunDisk A2 now, for example it support trim (BLKDISCARD), and I can erase full card with one command blkdiscard /dev/mmcblk0 now

PS more about ssd