ext3/4 and SD-card wear leveling

[rrolsbe]

http://forums.anandtech.com/showthread.php?t=2079571

The thread above has some good discussions on trim and wear leveling.

See my comment at the very end of the thread. I believe most, if not all, SDHC cards only perform dynamic wear leveling as opposed to static wear leveling.

http://en.wikipedia.org/wiki/Wear_leveling

[lb]

Yeah, but that's pretty obvious. Static wear levelling is hard, dynamic wear levelling is easy and you almost get it for free when implementing the block mapping. Still, the SD protocol (as opposed to the USB mass storage protocol) has block erase commands, and the controller might take advantage of them. They're passed to the card at least, so on the software side, everything is ready.

[rrolsbe]

Still, the SD protocol (as opposed to the USB mass storage protocol) has block erase commands, and the controller might take advantage of them.

I agree with your post. From my reading some (but not all) SDHC cards support block erase from the computer side interface. If the the SD Formatter 3.1 windows utility is used to format SDHC cards, it appears to be able to erase the NAND blocks during the format process (a good thing); however, it is not clear how the computer side block erase command could be used in conjunction with the (Flash controller side) Trim(if supported)/Garbage Collection/Block Erase/Wear Leveling functions for an SDHC card with an active file system . Block erase (from the USB/SDHC/SSD controller side) has to be supported for all NAND devices otherwise the flash controller could only write data to a NAND page one time. I hope I'm missing something obvious so ideas/clarifications from others are very welcome.

Regards, Ron

[polarlight]

This is interesting discussion, but could somebody who really knows about these things please state clearly, whether the most recommended Raspbian distribution handles SD-card wear leveling correctly off-the-shelf in the default configuration or that it doesn't?
If it does, from what version forward?
If it doesn't troubles are likely after running a couple of months in 24/7 with default logging on and using quite common software like Apache2, Motiong, having cronjobs running each minute etc.

It is just my opinion, but I think that if and when wear leveling is an issue with most typical SD-cards that are used with Raspberry Pi then taking into account the vast number of RPi users and that most of them are only starting to use Linux with RPi, this surely has been - or should have been - a design issue when crafting the default configuration of Raspbian. Otherwise these newcomers (like myself more or less) will get false negative feeling about Linux and Raspberry Pi itself if the systems start suddenly crashing.. I myself have seen two or three times different RPis dying (actually only their SD-cards, since replacing the SD with fresh image helps) after 1-3 months of use. Those have been running Motion and Apache2 for example, but just for home automation use. However Crontab, Apache2, Motion etc create easily lots of I/O to the SD-card in whatever typical home automation use.

Meanwhile I've tried to ease up the situation by creating small RAMDISKs into which at least Motion writes its webcam snapshots. For those who don't know how to make a RAMDISK, it can be
this simple to create a 32MB RAMDISK:
# create the ramdisk-folder
mkdir /tmp/ramdisk
# mount the ramdisk at /tmp/ramdisk
mount -t tmpfs -o size=32M tmpfs /tmp/ramdisk/

[minkowski]

From what I understood, the actual Raspbian version uses some settings for mounting the SD-card in fstab that reduce the number of writes (e.g. noatime). That's something to keep in mind: reduce the number of writes of your applications and put temporary files in a ramdisk (maybe even your logs with ramlog - link). Apart from that, no real wear leveling is done by the system as this is something the SD-card should take care of (corrections welcome).

But I'm still not sure if there is anything you can do to tell the SD-card which blocks are back to the pool at some point. Actually that keeps bothering me as I had a similar scenario to that described by Aydan: I used the SD-card as storage until it was nearly full and then switched to an external USB drive.

So I guess, one interesting question would be if the SD-card uses blocks with zeros written to them for wear leveling. The idea would be that if it does not support fstrim, you could use something like zerofree (link) to put zeros in the empty blocks such that the SD-card would use these blocks again. Of course that would take a write cycle of these blocks.

Oh and btw: As rrolsbe pointed out, it would be interesting to know which SD-cards support the fstrim command. So if someone uses it succesfully, share the info

[Aydan]

From what I understood, the actual Raspbian version uses some settings for mounting the SD-card in fstab that reduce the number of writes (e.g. noatime). That's something to keep in mind: reduce the number of writes of your applications and put temporary files in a ramdisk (maybe even your logs with ramlog - link). Apart from that, no real wear leveling is done by the system as this is something the SD-card should take care of (corrections welcome).

But I'm still not sure if there is anything you can do to tell the SD-card which blocks are back to the pool at some point. Actually that keeps bothering me as I had a similar scenario to that described by Aydan: I used the SD-card as storage until it was nearly full and then switched to an external USB drive.

So I guess, one interesting question would be if the SD-card uses blocks with zeros written to them for wear leveling. The idea would be that if it does not support fstrim, you could use something like zerofree (link) to put zeros in the empty blocks such that the SD-card would use these blocks again. Of course that would take a write cycle of these blocks.

Oh and btw: As rrolsbe pointed out, it would be interesting to know which SD-cards support the fstrim command. So if someone uses it succesfully, share the info

You're only half right
The OS never does wear leveling, that's for the flash controller to do, the one on the SD-card or SSD.
SD cards don't support trim, the SD-spec doesn't define it. It does however define the block erase command. As I understood it the SD-card driver translates trim commands into erase commands.
Also if I haven't misunderstood the SD spec, if you write zeroes to a block, it will be erased but not rewritten, since an erased block is already zeroed.
Erased blocks get deassigned and put into the wearleveling pool.
The SD-card can only do wear leveling if there's blocks in the pool, the more blocks in the pool the more effective the wear leveling.
Running fstrim does exacctly this, it erases unused blocks and they go back into the wearleveling-pool

Regards
Aydan

[minkowski]

Also if I haven't misunderstood the SD spec, if you write zeroes to a block, it will be erased but not rewritten, since an erased block is already zeroed.
Erased blocks get deassigned and put into the wearleveling pool.

In that case, using zerofree would have the same effect as fstrim and not even use a write cycle which is good news.

It does however define the block erase command. As I understood it the SD-card driver translates trim commands into erase commands.

Why does fstrim, as written below, then return the error? Shouldn't it just go ahead and emit the erase command?

Code: Select all

$ sudo fstrim -v /
fstrim: /: FITRIM ioctl failed: Operation not supported

Thanks for your clarifications

[Aydan]

Why does fstrim, as written below, then return the error? Shouldn't it just go ahead and emit the erase command?

Code: Select all

$ sudo fstrim -v /
fstrim: /: FITRIM ioctl failed: Operation not supported

Thanks for your clarifications

Is your root still on the SD-card?
It won't work for a HDD.
I'm using rasbian and have the root on the SD-card and for me fstrim works.

Regards
Aydan

[milhouse]

fstrim is working fine in Raspbian with a 32GB SanDisk Ultra card - I have a cron which runs at midnight:

Code: Select all

/var/log/syslog:Apr 22 00:00:01 raspberrypi fstrim: /: 312655872 bytes were trimmed
/var/log/syslog:Apr 23 00:00:02 raspberrypi fstrim: /: 2610241536 bytes were trimmed
/var/log/syslog:Apr 24 00:00:01 raspberrypi fstrim: /: 16539648 bytes were trimmed
/var/log/syslog:Apr 25 00:00:04 raspberrypi fstrim: /: 27977240576 bytes were trimmed
/var/log/syslog:Apr 26 00:00:01 raspberrypi fstrim: /: 2063556608 bytes were trimmed
/var/log/syslog:Apr 27 00:00:02 raspberrypi fstrim: /: 1151201280 bytes were trimmed
/var/log/syslog:Apr 28 00:00:02 raspberrypi fstrim: /: 306884608 bytes were trimmed
/var/log/syslog:Apr 29 00:00:01 raspberrypi fstrim: /: 1941381120 bytes were trimmed
/var/log/syslog:Apr 30 00:00:02 raspberrypi fstrim: /: 2900729856 bytes were trimmed

The /etc/crontab entry is:

Code: Select all

00 00   * * *   root    ionice -c 3 fstrim -v / | logger -t fstrim

[minkowski]

yes, the root is still on the sd-card. probably they are too cheap and don't support it (i have a panasonic class 10 with 16 gb and a intenso class 4 with 8 gb).
but if the driver "translates" the commmand it is a bit surprising that it works for some cards and doesn't for others (assuming that it is the same driver for all cards).

[polarlight]

This is in big part theory, but AFAIK SD-cards are essentially just flash memory devices typically with SPI and SD-bus connections to their host systems. As such they don't (necessarily) know about how they are formatted (FAT32,ext3/4,NTFS), but just provide means to read and write blocks with (hardware) addresses. Wear leveling in that case is something that the whatever host operating system's hardware-aware drivers should take care of. On the other hand SD-card specification allows SD-card to contain embedded microcontroller and functions, which allows it to internally handle wear leveling for example by offering virtual block addresses on the host side meanwhile using changing hardware block addresses internally. In this case the SD card would use part of its storage and performance capacity for bookkeeping and this storage area would also have to be subject for wear leveling.
I have not read what exact details the host can read form the SD-card, but the host can read what type of SD-card it is dealing with, its capacity and capabilities, possibly among those is wear leveling support. If so, the host's hardware driver should adapt to those capabilities reported by the connected SD-card.
If we think from the ordinary user's perspective the wear leveling SHOULD of course "just happen" when it is beneficial for the life time of SD-card. If it is not happening in that case, then there is something to do for software developers of that particular host system. If I got something wrong, feel free to correct.

[Aydan]

This is in big part theory, but AFAIK SD-cards are essentially just flash memory devices typically with SPI and SD-bus connections to their host systems. As such they don't (necessarily) know about how they are formatted (FAT32,ext3/4,NTFS), but just provide means to read and write blocks with (hardware) addresses.

True, The sd card is essentially a block organized RAM

Wear leveling in that case is something that the whatever host operating system's hardware-aware drivers should take care of.

Wrong, wear levelling if present at all is always done by the flash controller, in this case the controller on the SD-card, because there's no interface defined to get information on block wear.
e.g. here http://media.digikey.com/pdf/Data%20She ... Manual.pdf on page 1-4

On the other hand SD-card specification allows SD-card to contain embedded microcontroller and functions, which allows it to internally handle wear leveling for example by offering virtual block addresses on the host side meanwhile using changing hardware block addresses internally. In this case the SD card would use part of its storage and performance capacity for bookkeeping and this storage area would also have to be subject for wear leveling.

The bookkeeping will likely be done on eeprom memory which has a factor of 10 more write cycles.

I have not read what exact details the host can read form the SD-card, but the host can read what type of SD-card it is dealing with, its capacity and capabilities, possibly among those is wear leveling support.

No, you only get basic capabilities like voltage range, speed, capacity. No wear leveling is defined in the interface specification. Have a look at the pdfs here if you want al the gory details.

If so, the host's hardware driver should adapt to those capabilities reported by the connected SD-card.
If we think from the ordinary user's perspective the wear leveling SHOULD of course "just happen" when it is beneficial for the life time of SD-card. If it is not happening in that case, then there is something to do for software developers of that particular host system. If I got something wrong, feel free to correct.

Basically the only thing you can do to improve wearleveling is to make sure the pool doesn't deplete by either trimming or not partitioning the whole SD-card.

Regards
Aydan

[polarlight]

That makes sense, Aydan. Thanks for clarifications!

[Heater]

wear leveling or not. TRIM commands or not. Has anyone here found an SD card that is in any way reliable?

I have on my desk a bunch of ARM boards that use SD cards, including Raspi. I have a pile of SD cards from various manufactures and of various sizes from 1GB to 8GB.

I'm down to about two cards that don't give me some error or failure...so far.

What's the trick? These things seem to have a life span of 10 minutes. It's not as if I was constantly rewriting anything on them

[minkowski]

i am using the panasonic card since january now, without any problems. maybe i'm lucky but i don't think there is any trick. just continue trying until you find a good one (maybe sandisk?)

[minkowski]

today, i was using the sd formatter toolsd formatter tool from the sd association on one of my sd-cards. it allows to set the format option to "full (erase)". doing so gave me the response: "Flash erase" operation has been canceled because this device didn't support it.
so there apparently is a difference between the cards and some just don't support the erase command (which i couldn't imagine) . that's why fstrim doesn't work neither.

[Aydan]

Could it be that you're using an MMC card? As far as i know the SD spec requires the erase command for all SD cards.

[minkowski]

it says: "Intenso SDHC Memory Card Class 4" on it and it has the SDHC logo in the corner. so i guess it is proper sd card.

[Aydan]

That's really strange.
I just had a look into the SD spec and it says that CMD38 (which is the ERASE command) is mandatory.

[minkowski]

could it be that the sd-card reader / controller doesn' support the command? i just have the one built into my notebook an an external one to try it and they are both not working to flash erase the cards. but still it could be their fault as well.

[Aydan]

could it be that the sd-card reader / controller doesn' support the command? i just have the one built into my notebook an an external one to try it and they are both not working to flash erase the cards. but still it could be their fault as well.

Try it in the Raspi. Makes really no sense to do it outside.

[minkowski]

all the previous fstrim tests were of course done in the raspi.

[speculatrix2]

an important point to note is that erasing flash memory writes ONES not ZEROES to it.

if you can't trim a flash drive, writing ones to it like this will allow most flash controllers to mark blocks as unused:

tr '\000' '\377' < /dev/zero | dd bs=1024 count=$BLOCKS

[Aydan]

an important point to note is that erasing flash memory writes ONES not ZEROES to it.

if you can't trim a flash drive, writing ones to it like this will allow most flash controllers to mark blocks as unused:

tr '\000' '\377' < /dev/zero | dd bs=1024 count=$BLOCKS

I'll have to check with the SD spec but i'm pretty sure it states that reading an erased block returns zeros. It doesn't matter what is actually written in the physical memory, but if reading an erased block returns zeros, then writing ones will most likely not deallocate the block, because it does not contain default (erased) information any longer. So the block would not be available for wear leveling.

Regards
Aydan

[speculatrix2]

I'll have to check with the SD spec but i'm pretty sure it states that reading an erased block returns zeros. It doesn't matter what is actually written in the physical memory, but if reading an erased block returns zeros, then writing ones will most likely not deallocate the block, because it does not contain default (erased) information any longer. So the block would not be available for wear leveling.

yes, it does depend on the controller recognising that you've told it the block is no longer used because it's all ones. I got the idea for doing this here:
http://www.ocztechnologyforum.com/forum ... -OCZ-SSD-s

as I have an SSD in a netbook which doesn't support TRIM.

anyway, "erasing" a flash drive by writing to the entire device is a bad idea unless you're sure that's what you really want.

if you have Windows, then Panasonic will let you download a specific SD card formatter:
http://panasonic.jp/support/global/cs/sd/download/
I remember coming across this in the context of erasing a card and letting the flash controller in it know you were erasing it.