Still confused, what is better for my SD card?

[Musketeer]

If you have on some few GB, is it better to let AI shuffle data after each write and let it move and re-write data randomly or after each cca 1 month write whole card again, so written places "move around" more?

Pls advice! Thx!

[RetroBoyAdvance]

Purposely increasing writes would only degrade the card faster, no? If you only have a few GB in use there is plenty of over-provisioning available. If you only have few GB of space left, then you should be using a larger SD card for your purpose.

[Musketeer]

Some products are limited to 32 GB SDHC - ebook readers, tablets, consoles (NDS), ...

If you have 2-3 GB free, you are still writing on the same place, not?

[Andyroo]

Cards have some hidden space for wear levelling - I’m running 16Gb on Pi with HDD for machines needing more than around 12Gb

If you are that bothered get a HDD or SSD on USB and boot from that or via hybrid (SD and HDD) boot.

I’ve used SD cards for years in cameras and never had them fail from over use. I’ve only had one fail in a Pi and we think that was fake given what it reported

It also depends on how much data is being written and no one can advise that as the internal controller and cache routines are proprietary to the card manufacturers...

[rpdom]

If you have 2-3 GB free, you are still writing on the same place, not?

That's where the wear-leveling on the SD card comes into things. The card's controller keeps track of how many times a particular page has been written to. Every now and then it swaps data from a block that has been used a lot to a block that hasn't been used much. So even if you have a file that never gets written to, it could move around the card. But because the blocks on the card get remapped internally it doesn't appear to have moved for the OS.

[Musketeer]

But i am not sure still - if you fill 32 GB to 30 GB and let it sit, it will really move and rewrite itself all data around? Is this directly in SD specs somewhere? Logic says that if you have low GB capacity on SD, you should move it on PC sometimes and write again so it is always on another new fresh sector of card?

I got few died after leaving too little space... (few %)

I am not talking about Pi only , generally.

[HawaiianPi]

It could depend on the card. Wear levelling is not required in the SD Association's specifications, but I suspect most cards from well known makers support it (for example, SanDisk white papers say that all their SD cards support wear levelling).

In normal operation blocks will be moved around by the wear levelling process. This all happens in the background and is not visible to the user, and it's why performance degrades on cards that are nearly full (additional background data shuffling). This does introduce some write amplification, but less so that re-writing the whole card yourself (which will introduce more wear levelling and write amplification and decrease the life of the card).

SD cards that are written to frequently will wear out over time, and if those cards are nearly full, they will have more write amplification and will wear out faster. Moving all the files to a PC, formatting the card and then re-writing all of the data is only going to make things worse. Assume the companies that make these things know more about it than you do.

On a Raspberry Pi there are ways to minimize writes to the card with ramfs or tmpfs or certain mount options, but the best defense against wearing out your card is to use a card larger than you need, and write large amounts of date elsewhere (HDD, SSD, cloud storage).

There are also High Endurance SD cards which are designed for devices such as security cameras and dash cams that write a lot. They tend to have slower performance, but should last longer.

[Musketeer]

Ahha! So good card = good leveling!

Definitely will take the one with the biggest RAM after mein Atom ist kaput...

[HawaiianPi]

I've had good luck with the SanDisk Ultra A1 cards. They are affordable and have better performance than standard Class-10 cards (due to higher IOPS). I've had a Minecraft multiplayer server running 24/7 from one of those cards since 2017, and it's still going strong. I make weekly backups, just in case, and update the server software and OS monthly.

There are others on the forums who've had systems running on SD cards much longer than that. In general they are pretty reliable, however, with millions of Pi computers running from countless who-knows-what-brand cards, there are, of course, lots of people who have reported problems. I think that SD card problems and their unreliability is exaggerated, but there are certainly more appropriate devices for long term collection and storage of data.

In fact, most SD card makers will void the warranty for cards that are used as a computer OS drive (that's not their intended use), but it's hard to beat the compact size and convenience of a micro SD card.

It's possible to run any model of Raspberry Pi computer from a USB drive, such as an SSD or HDD (which are designed to be used as computer OS drives). The way its done varies with certain models, and some will need an SD card in the system, but the card is treated mostly as a read-only device and will be subject to litte wear.

Any storage device will fail eventually, so a good backup strategy is the most important thing. Most Raspberry Pi backup tutorials suggest making images with dd or win32diskimager, and those are both terrible solutions. Look up tutorials for making Linux backups using rsync, tar or other tools for a more robust and reliable solution (Raspbian is based on Debian, but most Linux backup tutorials should work on any distro).

[Musketeer]

I heard ARM drivers in SD cards are created to 1 000 cycles at least...

I used Pi 2 year with Verbatim 32 GB and card is still kicking too!

I am data minimalist! Few MB up to few hunderds of documents are more then enough for me, so I use my old cards (SLC only!) from cameras: CF, MS Pro, MMC, SD, ...

[Lomax]

Re. wear levelling, if I only need a fraction of the space on a given card, is it better to leave some space unpartitioned, or should my partitions take up 100% of what's available? Maybe it makes no difference?

[Heater]

Lomax,

..if I only need a fraction of the space on a given card, is it better to leave some space unpartitioned, or should my partitions take up 100% of what's available? Maybe it makes no difference?

SD cards do not know anything about your partitions or your file systems or your files.

All they know about is the block numbers you (your OS driver) has asked to write to and read from. Of course, what with all the wear leveling going on, those block numbers are not references to actual blocks on the device, the SD controller will put them wherever it likes. And swap things around subsequently.

So, just because you have defined partitions for yourself, which are just ranges of block numbers, does not mean the SD card won't use whatever actual physical space it likes.

We should not try an guess what is going on in those damn cards, nobody but the card makers knows what their controllers are actually doing. And they are not about to tell us.

[Lomax]

just because you have defined partitions for yourself, which are just ranges of block numbers, does not mean the SD card won't use whatever actual physical space it likes.

I've read this sentence several times but still not sure if that's a "yes, leaving unpartitioned space helps with wear levelling" or "no, it makes no difference if your partitions use up all the space". Maybe I need more coffee...

[RaTTuS]

just because you have defined partitions for yourself, which are just ranges of block numbers, does not mean the SD card won't use whatever actual physical space it likes.

I've read this sentence several times but still not sure if that's a "yes, leaving unpartitioned space helps with wear levelling" or "no, it makes no difference if your partitions use up all the space". Maybe I need more coffee...

well it depends on the controller on the card
it may or may not work...
the only way you can help is not by writing it so it's nearly full all the time

[pica200]

You don't want to do more writes than necessary! Just leave the controller alone doing wear leveling for you. There is a rule of thumb to leave at least 10% of the capacity free to avoid slowdowns and early failure. If you need more space it's time for an upgrade.

Another thing you can do is add the "discard" mount option for the root partition. discard makes the filesystem driver in the kernel send block erase commands to the SD whenever you "delete" files. This is also known as TRIM for SSDs (the idea is the same but different names). This will give the controller infos about what space is used or not and it can increase speed aswell because it can pre-erase unused flash pages skipping a step when data is next written to it.

[Lomax]

well it depends on the controller on the card
it may or may not work...
the only way you can help is not by writing it so it's nearly full all the time

I usually use these cards when I need high reliability:

Sandisk Industrial MicroSD.jpg (70.76 KiB) Viewed 621 times

Not terribly expensive, IIRC ~£10 each when bought in quantity.

As for not writing, I already set everything I can to use tmpfs (e.g. /var/log), with no automatic updates, and my apps generally do not write anything anywhere. My reason for asking is that I typically only use ~2Gb of storage for rootfs on my cards, so a 4Gb rootfs on an 8Gb card would be ~50% empty, while still leaving ~50% of the card unpartitioned.

[Lomax]

You don't want to do more writes than necessary! Just leave the controller alone doing wear leveling for you. There is a rule of thumb to leave at least 10% of the capacity free to avoid slowdowns and early failure. If you need more space it's time for an upgrade.

Thanks, I'll go with 6Gb partitioned / 2Gb unpartitioned. That leaves me with ~4Gb free space on rootfs, which is ample.

Another thing you can do is add the "discard" mount option for the root partition. discard makes the filesystem driver in the kernel send block erase commands to the SD whenever you "delete" files. This is also known as TRIM for SSDs (the idea is the same but different names). This will give the controller infos about what space is used or not and it can increase speed aswell because it can pre-erase unused flash pages skipping a step when data is next written to it.

Nice tip, I don't remember seeing this one before. So that would be (in fstab)

Code: Select all

PARTUUID=892be571-02  /               ext4    defaults,noatime,discard  0       1

Correct?

[pica200]

Yes. I don't know if the drivers of the older Pi models can do erase commands but the Pi 4 definitely can as using the "fstrim" tool succeeds. On the SanDisk A1 rated card i have this made it a little faster.

Another nice side effect is if you have a PCIe based card reader on your PC you can wipe the entire card within seconds using the "blkdiscard" tool. Be warned. This is unrecoverable!

[Fraoch]

I've read some articles that state that the "discard" option is probably not a good idea. Here's a rather lengthy but quite recent one:

https://lwn.net/Articles/787272/

fstrim is recommended but must be run manually or by cron. It's intended for SSDs though - it may or may not work on an SD card - the card reader has to pass the command and the SD card controller has to understand and implement it. Given that these are both undocumented or secretive black boxes, it's a crapshoot.

One thing you can do in fstab to minimize writes and to speed up the filesystem in general is to mount drives and directories with the "noatime" option. "noatime" will not write the latest access time to the file attributes. You won't know when a file was last accessed, but the OS won't care. This eliminates a write every time a file is accessed.

Leaving unallocated space is a good idea though, even if the controller uses whatever block it likes. It ensures the card will never be 100% full which has disastrous implications for drive performance and could even destroy the data. If the filesystem gets 100% full, performance will be poor but there are things you can do to recover. If the SD card ever gets 100% full, there may be nothing you can do to save your data.

[pica200]

SD cards do report the erase block size in the CSD (card specific data) so that's not a problem. Whenever erase is slow or fast depends on the card, yes. At least i had no SD card yet where blkdiscard on the whole card took longer than 15 seconds (with up to 128 GB capacity). That shows also that the card is not actually doing it immediately but takes it as a hint and probably does it in the background. For eMMC erase commands or TRIM like some call it has been standard for many years which probably also contributes to its long lifetime.

That post you linked sounds like a rather heavy commercial usecase where the SSD is often very busy. That will rarely be the case on most RPi's

I guess we could do benchmarks and see how discard affects performance.