SD cards - the Pi's greatest weakness

[Heater]

In my experience of these failures, dd from a Debian Linux PC could write 0x00 or 0xff to the entire SD. With no sign of failure. But then when reading the whole card back with dd I find that all or some of the blocks did not take the data.

Writing a Raspbian image to such a card could be done without error but the resulting card could not be booted due to the unwritten blocks.

I have not seen this since the early Pi days. Fingers crossed, touch wood.

[achrn]

One of my Pis is currently out of action due to card failure (though I have today got round to buying a new card, so it won't be much longer).

In this case, the card has been treated with kid gloves - it was the Pi that manages backup for my network, and is on a UPS. The power has never been just pulled, it has always had an orderly shutdown (generally it only shuts down when I do a reboot for updated software).

I think this is a Pi foundation branded card - most of my cards are Sandisk, but this one is plain black, and I did once buy two or three branded RPi ones, so I suspect this is one of them. I don't buy dodgy-cheap cards (as a matter of principle - I don't want my camera photos on a flaky card either - a dodgy card is so obviously a false economy it doesn't attract me).

The symptoms I have is that the card seems fine, the system runs fine, but file changes are not written to the card. Thus, when it does re-boot, files on the filesystem that were perfectly fine suddenly revert to a previous version. I think the updated file has remained exclusively in some buffer somewhere and never made it to the non-volatile part of the card.

I only found the problem because I was moving the system into a new case. Before I did that I made some changes to the script that logs the UPS state. I changed the script, tested it, everything fine. Shut down the pi, moved into new enclosure, booted, and the script was back to the old one. I cursed myself for somehow not saving or overwriting the new one or doing something equally dozy. I did it again, it failed again. I did it very carefully - I can change the file, save it, open it and it has the changes, reboot, and it's back to the unchanged state. 100% repeatable.

This is the second card that has done it, across about 20 unit-years of use. Both cases have been in pis that run near 24/7 (most of my Pis do) and I am always careful with power - my IT background is in Unix servers with lots of users, I'm just not going to pull power from a computer.

But I still think that SD card for the non-volatile memory is an inspired design choice.

[jamesh]

Note, 17M sold so far, and the failure rate of SD cards, as seen on the forums, is not huge. So I suggest it is actually quite rare.

Compare these 17M sold Pi-s with the number of users of this forum. It is extremely clear that many SD failures are not reported here.

That doesn't follow. This is the largest Pi forum. If huge numbers of card's were failing, this forum would be inundated.

[HawaiianPi]

That doesn't follow. This is the largest Pi forum. If huge numbers of card's were failing, this forum would be inundated.

QFT.

People look for support forums when they are having problems, which is why we seem to get more complaints than praises here, but even considering the negative bias present in a support forum, this is not a big issue here. As jamesh said, if it really was a widespread, common problem this forum would be flooded with posts about it.

[Gavinmc42]

Wow, a card that is on all the time for a long time, with probably Raspbian which does write to the card a lot.
Probably in a low ventilated enclosure running at high temperatures?
Extreme use case scenario, bound to fail sooner or later.
You found a problem, you identified the issue, now fix the issue, get new card.

Replace the card with a new one, this one has reached it's use by date.
I use PiCore OS which does NOT write back to the SD card whenever it feels like it.

Flash wears out, alpha particles can make them loose bits etc.
Go read up on flash memory, I am amazed they work for so long.
Some old equipment might still be running from UV EEPROMs,
They are supposed to have about 20year life, most would be due to fail any time now

A small SLC card (Single Layer Cell) should be more rugged than bigger capacity MLC (Multi Layer Cell).
And now they have 3D layered ones.

[HawaiianPi]

Wow, a card that is on all the time for a long time, with probably Raspbian which does write to the card a lot.
Probably in a low ventilated enclosure running at high temperatures?
Extreme use case scenario, bound to fail sooner or later.
You found a problem, you identified the issue, now fix the issue, get new card.

Way to ignore the contents of a thread and post an irrelevant answer. Next time try actually reading what you are replying to, so that you might actually contribute to the discussion.

[Gavinmc42]

Way to ignore the contents of a thread and post an irrelevant answer.

I may not be good at software but I have been in hardware for decades, including working for Toshiba as a field applications engineer specifically in memory and micros.

http://www.snia.org/sites/default/educa ... 1-0-nc.pdf
http://www.zdnet.com/article/how-does-f ... rage-fail/
Google will find much more info.

Wear leveling is only as good as the controller inside the card.
I may have been too sarcastic? Sorry
But this sounded exactly like a typical failure mode issue.

Lucky the solution is easy, replace the card.
This is NOT a weakness of the Pi's, this is the strength of going this way.
Anyone designing CM's into products and are using eMMC should or would be aware of this issue.

It could still be a software issue but swapping cards can give a clue to this.
Pi's do run very complex software and SD cards are made for low prices.
There are now lots of Pi's and SD cards working out there, I am suggesting the OP has reached his card's limit of of endurance.

I have had cards do weird things on Raspbian, when I put PiCore OS on them I get more life out of them.
I now use Ultibo so my memory requirements have gone down to SLC based card sizes.
I can pull power on PiCOre and Ultibo at anytime as the OS's run from ram, I may lose data but not the OS.
There are very good reasons for me going this way, number one is SD card reliability
But that is just based on my 6 years experience of using Pi's.

[Heater]

Gavinmc42,

Re. Your points:

During the time I was experiencing a lot of SD failures I can say the following:

"Wow, a card that is on all the time for a long time" - No. Those Pi were only on when I was tinkering with them.

"with probably Raspbian which does write to the card a lot." - Don't think it does write so much. Unless I as it to. Apart from a bit of logging and such.

"Probably in a low ventilated enclosure running at high temperatures?" - No. These were original Pi. No case. They run cool.

"Extreme use case scenario" - No. See above.

"You found a problem, you identified the issue" - No. See above.

"Replace the card with a new one, this one has reached it's use by date." - No. I don't believe a new card used as above should have reached it's use by date in only a matter of days. Are you really suggesting we get a new SD every week!

"Flash wears out ..." - Sure it does. Eventually. Not relevant here.

Having said all that. I do agree with you. For use in situations where high reliability is required don't ever write to the SD. For this reason I use Raspbian and make the root partition read only. After that all writes go to RAM. It's only a small config tweak to do that.

Luckily I have not seen this issue for a long while anyway. I don't know if it's because I have different SDs, or better PSUs, or some change in Raspbian or what.

[hippy]

this is not a big issue here. As jamesh said, if it really was a widespread, common problem this forum would be flooded with posts about it.

I don't think anyone is saying it's widespread or common. It's obviously not, but that doesn't necessarily mean it's not a problem or can simply be ignored. Without knowing the cause of a problem one cannot say whether it's something to worry over or not.

Let's pretend there's a product with a 1% chance of failure when it's turned off. With 20 million device sold one might expect 200K complaints of failure. Instead there are just 2K complaints. That doesn't mean the risk is not there, just that 99% of people are not exposing themselves to the risk, are not turning the product off or are getting lucky.

20 million sold, 2K complaints; it's not a widespread nor common problem. But consider it from a different angle -

Everyone who turns their product off is at risk of failure, statistically, everyone who turns their product off every day will experience multiple failures every year.

Everyone who turns off their product every day will see that as a huge problem, even if the overwhelming majority do not.

[jamesh]

this is not a big issue here. As jamesh said, if it really was a widespread, common problem this forum would be flooded with posts about it.

I don't think anyone is saying it's widespread or common. It's obviously not, but that doesn't necessarily mean it's not a problem or can simply be ignored. Without knowing the cause of a problem one cannot say whether it's something to worry over or not.

Let's pretend there's a product with a 1% chance of failure when it's turned off. With 20 million device sold one might expect 200K complaints of failure. Instead there are just 2K complaints. That doesn't mean the risk is not there, just that 99% of people are not exposing themselves to the risk, are not turning the product off or are getting lucky.

20 million sold, 2K complaints; it's not a widespread nor common problem. But consider it from a different angle -

Everyone who turns their product off is at risk of failure, statistically, everyone who turns their product off every day will experience multiple failures every year.

Everyone who turns off their product every day will see that as a huge problem, even if the overwhelming majority do not.

Pulling the plug instead of do sudo halt is a problem, but what can be done about it - we don't ignore the problem, we tell people to use sudo halt. But there is nothing that can be done if people simply pull the plug. We can warn them not to do it, or advise to buy certain SD cards that seem to be less susceptible (do you know which ones? No, me neither), but there are limits to what can be done.

[fanoush]

So maybe some capacitor on SD power pin would be enough to prevent damage of the card even when power plug is pulled in the middle of the write? Of course the filesystem would need check and fix but the card would not be damaged if it had a second to finish the write. Typically such capacitor should be inside the card but there is probably no space for such thing in microsd?

[hippy]

Pulling the plug instead of do sudo halt is a problem, but what can be done about it - we don't ignore the problem, we tell people to use sudo halt. But there is nothing that can be done if people simply pull the plug. We can warn them not to do it, or advise to buy certain SD cards that seem to be less susceptible (do you know which ones? No, me neither), but there are limits to what can be done.

Absolutely. People who just pull the power can and should expect problems. But that doesn't explain the cases where people ( like achrn above ) are not doing that, appear to be doing everything right, but are encountering problems.

Just pulling the power, poor power supply, excessive writing, poor quality cards, explains some of the cases but not all.

[bensimmo]

We 'pull the plug' here a lot and (touch wood) there has never been a problem.
Samsung Evo/Evo+ and some Kingston branded ones.

Even when it's turned itself off* datalogging at full pelt from SensHat, GPS and the Camera taking pics.
The worst we have had (and get all the time as a known) is three or four blank image files and one partly corrupt file.
Nothing to shout home about, they've taken some abuse.



* hitting the ground hard tends to do that

[hippy]

We 'pull the plug' here a lot and (touch wood) there has never been a problem.

I have stepped into the road without looking and have never been run over or killed. I have touched live mains and was not injured or killed.

But it's not about getting away with it when doing something wrong or inadvisable, when an adverse outcome could be expected. It's about why there is an adverse outcome when nothing wrong or inadvisable is being done.

[jamesh]

Pulling the plug instead of do sudo halt is a problem, but what can be done about it - we don't ignore the problem, we tell people to use sudo halt. But there is nothing that can be done if people simply pull the plug. We can warn them not to do it, or advise to buy certain SD cards that seem to be less susceptible (do you know which ones? No, me neither), but there are limits to what can be done.

Absolutely. People who just pull the power can and should expect problems. But that doesn't explain the cases where people ( like achrn above ) are not doing that, appear to be doing everything right, but are encountering problems.

Just pulling the power, poor power supply, excessive writing, poor quality cards, explains some of the cases but not all.

AFAWK, there are no HW issues with the SD card interface, and no real evidence to support a HW or SW issue on the Pi itself. These issues will almost certainly be down to the type of card used, power supply, cabling, high usage, or perhaps even a defective PI, or any combination of the above.

[achrn]

Pulling the plug instead of do sudo halt is a problem, but what can be done about it - we don't ignore the problem, we tell people to use sudo halt. But there is nothing that can be done if people simply pull the plug. We can warn them not to do it, or advise to buy certain SD cards that seem to be less susceptible (do you know which ones? No, me neither), but there are limits to what can be done.

Except that this problem is not confined to people that pull the plug.

I've had it happen twice, and interestingly (I discovered checking my notes yesterday) both times was the same Pi - a B+v1.0 with rev code 0010. I absolutely never pull the plug, and in fact that particular pi is on a UPS.

Also, I'm confident it's not just flash wearing out - that machine does relatively little and the latest card has lasted 13 months. Cron.daily runs a script that rsyncs data from about eight other devices onto a spinning magnetic disk on the pi. It records the files transferred into a MySQL database which is on the card, but only the name and size of them - it's not a vast database. It was an 8GB card, with raspbian lite, rsync, emacs and mysql added. It wouldn't surprise me if there has been less than 8GB of data written to the card in aggregate across its entire life - ie a 8GB WORM device would not have filled up.

As to other speculation on the thread - it's located in my (concrete wall) shed, a nice cool location. It's in an enclosure that's about 3 litres of internal volume, has trickle ventilation, and never sees extreme temperature (there's a PoE module in with it and the disk drive, so it never drops too cold even when the shed is below freezing, but it never gets very hot either) - it has a DS1820 attached so I can see the environment temperature inside the enclosure). The shed itself is dry (weathertight) but draughty (mostly under the door, which doesn't quite reach the ground, and through at eaves where the roof is a bit bodged onto the walls).

[jamesh]

Hmm, quite an old device, but that shouldn't make a difference. The problem with figuring out problems like this is the infrequency of the occurrence, and the inability to replicate it in the first place.

Of course, I suppose SD card can simply fail, for reasons other than the known flash wear.

[fanoush]

These issues will almost certainly be down to the type of card used, power supply, cabling, high usage, or perhaps even a defective PI, or any combination of the above.

Not entirely correct. My case was opening too much tabs in Chrome causing out of memory and swapping hell lasting for minutes with no response from the device and with SD light constantly on. Again, how is typical Pi user supposed to solve this correctly? The card that died by powering off in this case was Samsung.

there are no HW issues with the SD card interface, and no real evidence to support a HW or SW issue on the Pi itself.

Well technically yes, but SD cards are designed for cameras, phones etc. where pulling the power abruptly is not typical and software and design of hardware tries to prevent it as much as possible. If you e.g. read datasheet for SD cards they explicitly warn you as a designer to take care of this, see e.g. https://www.mikrocontroller.net/attachm ... 08G7B7.pdf , page 47, 9.2.3 Power control of SD memory card , 9.6.Prohibition during Write (Mandate)

So if you design product with no safe poweroff button, no battery, where user can just pull the plug anytime and where users are supposed to use general purpose OS (linux) that can go into state where card is constantly written with no way to interrupt it, then maybe what you correctly described is still not enough to prevent damage which is not strictly user or card manufacturer fault and can be expected.

[jamesh]

These issues will almost certainly be down to the type of card used, power supply, cabling, high usage, or perhaps even a defective PI, or any combination of the above.

Not entirely correct. My case was opening too much tabs in Chrome causing out of memory and swapping hell lasting for minutes with no response from the device and with SD light constantly on. Again, how is typical Pi user supposed to solve this correctly? The card that died by powering off in this case was Samsung.

Odd that this caused corruption. Did it kill the card completely? I've pulled the plug in similar circumstances multiple times, with no ill effects. TBH, the first 3 years of using Pi's I didn't use sudo halt at all...

[bensimmo]

We 'pull the plug' here a lot and (touch wood) there has never been a problem.

I have stepped into the road without looking and have never been run over or killed. I have touched live mains and was not injured or killed.

But it's not about getting away with it when doing something wrong or inadvisable, when an adverse outcome could be expected. It's about why there is an adverse outcome when nothing wrong or inadvisable is being done.

I've walked on the pavement and been hit by a car and was in traction for 3 months, so what.

Just adding my happy two pence worth.
and in reply to the pulling the plug comments.

Many people do, it's the way the 'Pi' is designed that makes the 'pulling the plug' happen and the normal thing to do.
Many (most ?) people do not see problems.

I have had SSD and many mechanical drives corrupt in a PC and they have controlled shutdowns.
Stuff happens.

[fanoush]

Odd that this caused corruption. Did it kill the card completely?

The permanent read only state. Bad luck perhaps.
The small 100MB swap turned on by default in raspbian probably helped it. The pi3 responded fine and I made a mistake of opening yet another link in new tab that pointed to some heavy page (like raspberry blog with lots of youtube videos?) and suddenly it was too late to do anything.

Again I wonder if simple capacitor on SD power would keep the card alive (or possibly with some diode to prevent backfeeding rest of the pi?) so the main CPU would power down and stop writing a bit earlier before card power goes down completely.

[jamesh]

I have had SSD and many mechanical drives corrupt in a PC and they have controlled shutdowns.
Stuff happens.

Good point, I've had more SSD's fail on me (two) than SD cards fail on me (one). And I have loads of SD cards, and have only ever bought 4 SSD's.

[Imperf3kt]

So maybe some capacitor on SD power pin would be enough to prevent damage of the card even when power plug is pulled in the middle of the write? Of course the filesystem would need check and fix but the card would not be damaged if it had a second to finish the write. Typically such capacitor should be inside the card but there is probably no space for such thing in microsd?

How would that help at all?
Lets say, theoretically, the cap will give the pi 1 second to complete its write cycle. Lets say this write takes 0.5 seconds.
What do you propose happens, if the power is pulled, and the Pi begins a write cycle 0.6 seconds later?

[Heater]

I guess the idea is that the capacitor powering the SD is separated from the Pi power supply by a diode. Then if the Pi supply dies, all action on the Pi stops, but the SD still has power from it's capacitor for some seconds after. During which time it can complete whatever outstanding command it has.

Interesting idea. No idea if it makes any sense or not.

[rpdom]

Interesting idea. No idea if it makes any sense or not.

It makes perfect sense. I believe most hard disks have a similar thing, maybe including capacitors and using the energy from the slowing disk to generate enough power to flush the immediate cache and park the heads safely.