Aydan
Posts: 587
Joined: Fri Apr 13, 2012 11:48 am
Location: Germany, near Lake Constance

ext3/4 and SD-card wear leveling

Tue Oct 09, 2012 8:03 am

While thinking about how to implement a backup suite for the Raspberry I ran across something that might be a killer for SD cards:
It seems linux is not aware it is running on a flash medium and does not trim freed blocks, which basically kills wear leveling after a while.
When I used fstrim on my 24/7 pi, which has been running since july, it trimmed the whole free space, meaning there was almost no space free for wear leveling.
Solution:
run

Code: Select all

sudo ionice -c 3 fstrim -v /
often. I made it a daily cronjob on my pi.
The first time might take a while, depending on the random access times of your SD card, but afterwards it's quite quick.

Regards
Aydan

User avatar
jojopi
Posts: 3041
Joined: Tue Oct 11, 2011 8:38 pm

Re: ext3/4 and SD-card wear leveling

Tue Oct 09, 2012 10:11 am

Aydan wrote:It seems linux is not aware it is running on a flash medium and does not trim freed blocks, which basically kills wear leveling after a while.
When I used fstrim on my 24/7 pi, which has been running since july, it trimmed the whole free space, meaning there was almost no space free for wear leveling.
The first time might take a while, depending on the random access times of your SD card, but afterwards it's quite quick.
The kernel is well aware that SD is flash, as is evidenced by the fact that it flags its swap space as "SS" (solid state). But whether it attempts to trim is controlled by the "discard" mount option, not the underlying block device medium. And whether the trim succeeds, and does anything useful, is dependent on the implementation of all the driver levels, and hardware.

fstrim will always (attempt to) discard all the free space of the filesystem. Not only the first time you run it, but at least the first time you run it in any given boot. That is because the ext4 on-disk bitmaps only record whether blocks are free or used; there is simply no space to record whether the free blocks have been trimmed or not. So, you would probably be better to do just one "fstrim" and then switch to mounting with the "discard" option to get the supposed effect in real time.

I am not convinced there actually is an effect; that the discards make it through the SDHCI and BCM2708_Arasan layers in a format that does anything to the hardware. Do you have any evidence of this?

Aydan
Posts: 587
Joined: Fri Apr 13, 2012 11:48 am
Location: Germany, near Lake Constance

Re: ext3/4 and SD-card wear leveling

Tue Oct 09, 2012 10:22 am

I have no evidence yet, but I was planning on going through the partition with dd and check the number of erased blocks.
I sure hope the trim command will at least write all zeros to a block (or just erase it) to tell the SD-card it's not used.
If this is not implemented this will kill SD-cards quicker then necessary for uses with lots of writes.

Regards
Aydan

stecklars
Posts: 27
Joined: Sat Mar 03, 2012 5:42 pm

Re: ext3/4 and SD-card wear leveling

Tue Oct 09, 2012 11:36 am

By the way:
Sandisk cards have got lifetime warranty, so you're safe when using one of those :)

User avatar
Mortimer
Posts: 894
Joined: Sun Jun 10, 2012 3:57 pm

Re: ext3/4 and SD-card wear leveling

Tue Oct 09, 2012 12:46 pm

Aydan wrote:I sure hope the trim command will at least write all zeros to a block (or just erase it) to tell the SD-card it's not used.
If this is not implemented this will kill SD-cards quicker then necessary for uses with lots of writes.
Writing zeroes to flash will use up a write cycle too wouldn't it. Surely the idea of TRIM is simply allow the operating system to inform the flash device's logic which blocks are no longer in use, and therefore available for garbage collection and reuse.
stecklars wrote:By the way:
Sandisk cards have got lifetime warranty, so you're safe when using one of those :)
If the lifetime of the Sandisk device is determined by how long it takes the blocks to wear out after being overwritten too many times, that won't help will it?

Aydan
Posts: 587
Joined: Fri Apr 13, 2012 11:48 am
Location: Germany, near Lake Constance

Re: ext3/4 and SD-card wear leveling

Tue Oct 09, 2012 1:53 pm

Mortimer wrote:Writing zeroes to flash will use up a write cycle too wouldn't it. Surely the idea of TRIM is simply allow the operating system to inform the flash device's logic which blocks are no longer in use, and therefore available for garbage collection and reuse.
The SD standard does not know TRIM itself. You can erase a block, which will go back into the pool and as far as I know writing all zeros to a block will not allocate a block from the pool.
Reading an unallocated block will return all zeros. So even if you can't explicitly erase or trim a block, because the driver doesn't support it, writing zeros to a block will cause it to be erased and not reallocated.

Regards
Aydan

User avatar
Mortimer
Posts: 894
Joined: Sun Jun 10, 2012 3:57 pm

Re: ext3/4 and SD-card wear leveling

Tue Oct 09, 2012 3:27 pm

I still can't see that there is any advantage to writing zeros to an unwanted block. Having it marked for deletion should be enough, and use up the block next write cycle count for writing wanted data to it when the time comes. Wanting to write zeros to a block that is no longer wanted will in my mind effectively double the wear rate across the whole device.

Aydan
Posts: 587
Joined: Fri Apr 13, 2012 11:48 am
Location: Germany, near Lake Constance

Re: ext3/4 and SD-card wear leveling

Tue Oct 09, 2012 4:20 pm

Mortimer wrote:I still can't see that there is any advantage to writing zeros to an unwanted block. Having it marked for deletion should be enough, and use up the block next write cycle count for writing wanted data to it when the time comes. Wanting to write zeros to a block that is no longer wanted will in my mind effectively double the wear rate across the whole device.
As long as the SD card does not know that the block is free, the SD card controller's wear leveling can't use it.
This basically means the file system takes control over which block gets rewritten and this might wear blocks earlier than necessary.
Take the following example:
You've been using your SD-card for months and have done a lot of writes to it. It might even have been almost full at one time, meanig all blocks have some sort of data on them, even if the card is only half full.
Now you have a program that updates a file quite often. This means that for each write of this file, the same flash block gets written again and again, because the wear leveling pool is empty.
I know this is a very extreme case, but I'm trying to make a point here.
Now if you could tell the SD card that you're not using half of it's blocks, it can use a new block for each write and wear many blocks very little instead of wearing one block a lot.
And this is actually a real problem. I mean why would SSDs support a dedicated command (TRIM) for this if it wouldn't matter?

lb
Posts: 232
Joined: Sat Jan 28, 2012 8:07 pm

Re: ext3/4 and SD-card wear leveling

Tue Oct 09, 2012 4:41 pm

The discards do make it to the hardware, that's for sure. The SDHCI driver even includes a hack for SD cards that are especially slow at this.

Please do not use the "discard" mount option. Trimming on demand is often very bad for performance. It's much preferable to run fstrim from time to time, just like Aydan does. However, I am not sure if the simple wear-leveling algorithms in most SD cards will actually take advantage of the trimmed space.

Aydan
Posts: 587
Joined: Fri Apr 13, 2012 11:48 am
Location: Germany, near Lake Constance

Re: ext3/4 and SD-card wear leveling

Wed Oct 10, 2012 2:14 pm

I can confirm that the unused blocks are zeroed out.
Ran a little python script counting zeroed 1k chunks from /dev/root and it came out a bit more than what df / reports as free.
I can also confirm that fstrim will trim all unused blocks after a reboot, no matter if they are zeroed already.
I have no Idea what the SD card does if you erase an already erased block again.
Might be an idea to "improve" it so that it reads a block first to check if it is empty already before actually erasing.

@lb: does the driver only erase or does it write zeroes as well?

lb
Posts: 232
Joined: Sat Jan 28, 2012 8:07 pm

Re: ext3/4 and SD-card wear leveling

Wed Oct 10, 2012 3:07 pm

The driver just emits an "ERASE" command for the blocks in question.

mole125
Posts: 228
Joined: Tue Jan 10, 2012 2:01 pm

Re: ext3/4 and SD-card wear leveling

Wed Oct 10, 2012 3:54 pm

Of course all you have proved is that if you read an erased/trimmed block the SD card controller return zeros, you haven't proved whether it actually wasted time erasing them, it may just have set a flag somewhere and have code in the format:
if(blockErased) return zeroData else return blockData.

Aydan
Posts: 587
Joined: Fri Apr 13, 2012 11:48 am
Location: Germany, near Lake Constance

Re: ext3/4 and SD-card wear leveling

Wed Oct 10, 2012 5:00 pm

mole125 wrote:Of course all you have proved is that if you read an erased/trimmed block the SD card controller return zeros, you haven't proved whether it actually wasted time erasing them, it may just have set a flag somewhere and have code in the format:
if(blockErased) return zeroData else return blockData.
What I proved is that IF the SD card controller uses wear leveling, then it can use the erased blocks for it.
It is known that an SSD/Flash drive, that uses wear leveling, is full, it will wear faster because there's less space for the wear leveling mechanism to work with.
Even if the block is erased on write, it means the SD card has a bigger pool of usable blocks.
Also this thread wasn't about actually erasing SD blocks but about telling the SD card you're not using them anymore.

mcgyver83
Posts: 358
Joined: Fri Oct 05, 2012 11:49 am

Re: ext3/4 and SD-card wear leveling

Thu Dec 06, 2012 2:56 pm

I looked on google but I cannot understand how TRIM command works: the os says to filesystem that I wand to delete e file, filesystem instead of deleting the corresponding cell send a TRIM command to the SDHC and the filesystem knows that when the os asks to save a new file it can say to the SDHC to use the previuos cell using only one write instead of two (1 delete and 1 write).
Using "discard" in the mount options will enable this, is it right?

Aydan
Posts: 587
Joined: Fri Apr 13, 2012 11:48 am
Location: Germany, near Lake Constance

Re: ext3/4 and SD-card wear leveling

Fri Dec 07, 2012 8:45 pm

mcgyver83 wrote:I looked on google but I cannot understand how TRIM command works: the os says to filesystem that I wand to delete e file, filesystem instead of deleting the corresponding cell send a TRIM command to the SDHC and the filesystem knows that when the os asks to save a new file it can say to the SDHC to use the previuos cell using only one write instead of two (1 delete and 1 write).
Using "discard" in the mount options will enable this, is it right?
Not quite.
The OS deletes a file and the blocks where the file resides are marked as available for overwriting in the FAT/MFT/whatever. The harddist or SD-card doesn't know anything at this moment. That's how all filesystems work. TRIM means that the OS also explicitly tells the harddrive/SDHC/whatever, that these blocks are no longer needed. So now it's up to the harddrive/SDHC-driver what is does with them. In case of the SDHC driver of the raspberry, it will issue an eraseblock command to the SD-card, since SD-cards don't understand TRIM. An SSD does understand TRIM and erases unused blocks in it's spare time, with minimal performance impact.
Using discard on an SD-card might impact performance quite a bit. I haven't tried it yet.

Regards
Aydan

Narf03
Posts: 243
Joined: Mon Jun 11, 2012 3:44 pm
Location: Malaysia

Re: ext3/4 and SD-card wear leveling

Fri Dec 07, 2012 10:20 pm

stecklars wrote:By the way:
Sandisk cards have got lifetime warranty, so you're safe when using one of those :)
Lifetime warranty isn't your lifetime, it's the product lifetime, means if your sandisk 8gb class 10 spoil and if they still selling that 8gb class 10, then it's under warranty. So the lifetime warranty might not be a good thing.

mcgyver83
Posts: 358
Joined: Fri Oct 05, 2012 11:49 am

Re: ext3/4 and SD-card wear leveling

Sat Dec 08, 2012 12:16 pm

Aydan wrote:
mcgyver83 wrote:I looked on google but I cannot understand how TRIM command works: the os says to filesystem that I wand to delete e file, filesystem instead of deleting the corresponding cell send a TRIM command to the SDHC and the filesystem knows that when the os asks to save a new file it can say to the SDHC to use the previuos cell using only one write instead of two (1 delete and 1 write).
Using "discard" in the mount options will enable this, is it right?
Not quite.
The OS deletes a file and the blocks where the file resides are marked as available for overwriting in the FAT/MFT/whatever. The harddist or SD-card doesn't know anything at this moment. That's how all filesystems work. TRIM means that the OS also explicitly tells the harddrive/SDHC/whatever, that these blocks are no longer needed. So now it's up to the harddrive/SDHC-driver what is does with them. In case of the SDHC driver of the raspberry, it will issue an eraseblock command to the SD-card, since SD-cards don't understand TRIM. An SSD does understand TRIM and erases unused blocks in it's spare time, with minimal performance impact.
Using discard on an SD-card might impact performance quite a bit. I haven't tried it yet.

Regards
Aydan
Thanks.
So using TRIM on raspberry is bad because SDHC will execute an erase command instead of on an SSD. So why someone is saying to set TRIM command on the rasp filesystem with "discard" mount option?

Aydan
Posts: 587
Joined: Fri Apr 13, 2012 11:48 am
Location: Germany, near Lake Constance

Re: ext3/4 and SD-card wear leveling

Sat Dec 08, 2012 12:29 pm

mcgyver83 wrote:Aydan
Thanks.
So using TRIM on raspberry is bad because SDHC will execute an erase command instead of on an SSD. So why someone is saying to set TRIM command on the rasp filesystem with "discard" mount option?[/quote]
It's not bad at all, but you might needlessly sacrifice performance. The erase command is needed for the SD-card to be able to do wear leveling properly. If not used the waer leveling pool might be depleted and the SD-card will wear faster than necessary.
I myself don't use discard, but have a daily cronjob running fstrim on the root partition.

rrolsbe
Posts: 40
Joined: Fri Aug 12, 2011 4:09 pm

Re: ext3/4 and SD-card wear leveling

Sat Dec 08, 2012 10:48 pm

If forum members have found SDHC cards that actually do something with the trim command, I would like to know which ones do. The Trim command is an ATA command that can be sent to an SSD controller telling it which NAND pages (usually 4K in size) have been deleted in the file system. If the controller can process Trim commands, it will flag the NAND page(s) so the SSD controllers garbage collection algorithm can eventually recover/erase the page. AFAIK, no SDHC card has a controller that responds to the Trim command, I wish they did. The SDHC controller has no idea which pages contain deleted information. I believe some EMMC flash, used in newer smartphones, do support a form of Trim. Bottom line SDHC cards are inexpensive devices and do not have many of the performance features a good SSD does.

Regards

mcgyver83
Posts: 358
Joined: Fri Oct 05, 2012 11:49 am

Re: ext3/4 and SD-card wear leveling

Sun Dec 09, 2012 12:47 am

Aydan wrote:
mcgyver83 wrote:Aydan
Thanks.
So using TRIM on raspberry is bad because SDHC will execute an erase command instead of on an SSD. So why someone is saying to set TRIM command on the rasp filesystem with "discard" mount option?
It's not bad at all, but you might needlessly sacrifice performance. The erase command is needed for the SD-card to be able to do wear leveling properly. If not used the waer leveling pool might be depleted and the SD-card will wear faster than necessary.
I myself don't use discard, but have a daily cronjob running fstrim on the root partition.[/quote]

Ok, I understand. Please can you paste here the command line you run with fstrim command, please?

kalehrl
Posts: 350
Joined: Tue Jul 24, 2012 10:49 am

Re: ext3/4 and SD-card wear leveling

Sun Dec 09, 2012 9:07 am

Not even all SSDs support the TRIM command.
4GB SSD in my eee pc surf doesn't support it.

User avatar
toysareforboys
Posts: 136
Joined: Thu Dec 06, 2012 11:01 pm

Re: ext3/4 and SD-card wear leveling

Mon Dec 10, 2012 6:54 am

Aydan wrote:I made it a daily cronjob on my pi.
Can you post a "how-to" for complete Linux noobs? I'm running a Pi as a mission critical server, and it does lots of read/writes (captures live weather data, makes some .JPG images with it, hosts it on it's websites). Would like to make sure wear levelling has the best chance at working possible.

-Jamie M.
Seagate GoFlex Home, 1.2GHz ARM (kirkwood), 128MB RAM, Gigabit Ethernet, SATA2. Sandisk Extreme 120GB SSD running Arch ARM Linux 3.6.11-0. nginx + php-fpm = LIVE STATUS hosted right on the SGFH!! http://tafb.yi.org

Aydan
Posts: 587
Joined: Fri Apr 13, 2012 11:48 am
Location: Germany, near Lake Constance

Re: ext3/4 and SD-card wear leveling

Mon Dec 10, 2012 7:30 am

add the following line to your crontab:

Code: Select all

# m     h       dom     mon     dow     command
0       1       *       *       *       ionice -c 3 fstrim -v /
This runs the job every day at 1am
This will only trim the root filesystem.

Regards
Aydan

dms75
Posts: 25
Joined: Mon Aug 06, 2012 1:41 pm

Re: ext3/4 and SD-card wear leveling

Mon Dec 10, 2012 7:36 am

Hi

To add some information regarding the behaviour of SDHC cards,

all the sandisk SDHC cards I have come accross almost imediately return from a block erase command, after wich the block is reread as zero. Writing Zeroes to that block actually is much slower, so I guess that the erase command only marks the block as erased and does not actually write to it at all. When writing a single block or several consecutive blocks you can save writetime by preerasing blocks, but you have to use a specifig erase write sequence for that, so that the card can erase several blocks at once and save time compared to erasing each block with the standard erase and write block command one at a time.
So even if the SDHC cards don't support the trim command the erase block command of the SDHC card might actually do quite the same, since it's too fast for a real physical erase procedure but it still results in a block being reread as zero.
Note that this might well be device and or vendor specific behaviour, its what i saw while working with sandisc SDHC cards on a dataloggger project with a microcontroller using SPI protocoll for communication.

As far as I know the standard itself does not specify this behaviour, but it seams a cheap way to prolong the lifetime of a sdhc card, especially since they don't support a trim command. So since its easy to implement and the sdhc cards need to do some kind of wear leveling anyhow I guess that most cards will use this approach, but ofc its up to the sdcard manufacturer...

Dirk

User avatar
toysareforboys
Posts: 136
Joined: Thu Dec 06, 2012 11:01 pm

Re: ext3/4 and SD-card wear leveling

Mon Dec 10, 2012 2:48 pm

Aydan wrote:add the following line to your crontab:
Ok, I'll try and figure out what/where crontab is :)

-Jamie M.
Seagate GoFlex Home, 1.2GHz ARM (kirkwood), 128MB RAM, Gigabit Ethernet, SATA2. Sandisk Extreme 120GB SSD running Arch ARM Linux 3.6.11-0. nginx + php-fpm = LIVE STATUS hosted right on the SGFH!! http://tafb.yi.org

Return to “General discussion”

Who is online

Users browsing this forum: mrharrison and 37 guests