johnspackman
Posts: 18
Joined: Tue Jul 16, 2013 9:29 am

Re: Prevent SD-Card Corruption

Tue Dec 05, 2017 12:59 pm

All the suggestions above are attempting to fix a minor problem, and all cost a hat load of money.
^^^ exactly.

Last month, we broke the 1,000 mark on shipped devices based on Raspberry Pi and SD cards - these are not general purpose devices, they are digital signage devices that live in waiting rooms and get a certain amount of mistreatment - getting unplugged by the cleaner, or just routinely turned off overnight. I can count the number of SD card corruptions on one hand, including two instances where the user decided to remove the SD card while the Pi was running :o

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23683
Joined: Sat Jul 30, 2011 7:41 pm

Re: Prevent SD-Card Corruption

Tue Dec 05, 2017 4:42 pm

johnspackman wrote:
Tue Dec 05, 2017 12:59 pm
All the suggestions above are attempting to fix a minor problem, and all cost a hat load of money.
^^^ exactly.

Last month, we broke the 1,000 mark on shipped devices based on Raspberry Pi and SD cards - these are not general purpose devices, they are digital signage devices that live in waiting rooms and get a certain amount of mistreatment - getting unplugged by the cleaner, or just routinely turned off overnight. I can count the number of SD card corruptions on one hand, including two instances where the user decided to remove the SD card while the Pi was running :o
As an aside, think I went to school (One on Hills Road, Cambridge) with a John Spackman - any relation?
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

hippy
Posts: 5962
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Prevent SD-Card Corruption

Tue Dec 05, 2017 4:56 pm

johnspackman wrote:
Tue Dec 05, 2017 12:59 pm
All the suggestions above are attempting to fix a minor problem, and all cost a hat load of money.
^^^ exactly.
Indeed. Same line of argument as to why there's no need for more gun control in America. 325 million people, only a few murdered by guns each year: 0.003%

johnspackman
Posts: 18
Joined: Tue Jul 16, 2013 9:29 am

Re: Prevent SD-Card Corruption

Tue Dec 05, 2017 5:02 pm

jamesh wrote:
Tue Dec 05, 2017 4:42 pm
As an aside, think I went to school (One on Hills Road, Cambridge) with a John Spackman - any relation?
No, sorry, I went to Kingswood in Bath. That would be an amazing small world coincidence though, there don't seem to be that many of us around!

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23683
Joined: Sat Jul 30, 2011 7:41 pm

Re: Prevent SD-Card Corruption

Wed Dec 06, 2017 9:16 am

hippy wrote:
Tue Dec 05, 2017 4:56 pm
johnspackman wrote:
Tue Dec 05, 2017 12:59 pm
All the suggestions above are attempting to fix a minor problem, and all cost a hat load of money.
^^^ exactly.
Indeed. Same line of argument as to why there's no need for more gun control in America. 325 million people, only a few murdered by guns each year: 0.003%
Wow, conflating a minor SD card problem with mass murders. Strawman argument in the extreme.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

User avatar
HawaiianPi
Posts: 4616
Joined: Mon Apr 08, 2013 4:53 am
Location: Aloha, Oregon USA

Re: Prevent SD-Card Corruption

Wed Dec 06, 2017 11:09 am

lost wrote:
Tue Dec 05, 2017 10:06 am
... But Raspi foundation have to get involved in providing a way for individuals to buy devices that are made for 24/7 use.
Image

Where did you get the idea that RPF was there to service your personal business needs?
Image

The consumer Raspberry Pi computers were never designed for you to build a business around. They are not industrial machines. They are EDUCATIONAL TOOLS created to promote learning, and made cheap enough so that they could be used by everyone (especially children). I don't know where you get off with such an entitled attitude. RFP doesn't have to do a damn thing to support your personal business ventures. If you are going to base a business around a product that was never intended for the use you have in mind, then it's up to YOU to figure out how to make it work.
My mind is like a browser. 27 tabs are open, 9 aren't responding,
lots of pop-ups...and where is that annoying music coming from?

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23683
Joined: Sat Jul 30, 2011 7:41 pm

Re: Prevent SD-Card Corruption

Wed Dec 06, 2017 11:32 am

Its worth noting that the Compute module, which IS intended for industrial use, has onboard flash memory which is much more robust than SD flash.

However, we do sell a load of standard Pi's in to industry, and of course, where there are issues, we try to fix them. However, I doubt we will be moving away from the SD card format for a while! It's cheap, works 99.99% of the time, and fits exactly with the educational needs of the original target market. TBH, it's also fits in pretty well with industry (see above post), and if you mitigate the number of SD accesses they can also be pretty robust.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

NCRbasketcase
Posts: 2
Joined: Wed Sep 20, 2017 3:22 pm

Re: Prevent SD-Card Corruption

Wed Dec 06, 2017 4:45 pm

SD cards like all high density memory devices can be damaged by electrostatic discharge. The trouble is that the damage may not be evident at first and only becomes clear after a number of read / write cycles. The card may format OK and even accept a fresh install of the OS, but fail later. Its important to obey normal ESD handling precautions like wearing a resistive wrist strap bonded to earth whilst handling the Pi and memory card. I suspect a lot of these failures are simply down to poor handling precautions

lost
Posts: 28
Joined: Tue Dec 05, 2017 9:38 am

Re: Prevent SD-Card Corruption

Thu Dec 07, 2017 8:41 am

HawaiianPi wrote:
Wed Dec 06, 2017 11:09 am
Where did you get the idea that RPF was there to service your personal business needs?
(...)
If you are going to base a business around a product that was never intended for the use you have in mind, then it's up to YOU to figure out how to make it work.
That's clearly not my point & you know it. So be honest & stop laughing.

First point was there is a storage reliability problem that is not necessarily linked with unclean shutdowns (+ I've seen industrial devices with a clean cold restart path & an unclean hot one leading to worst corruptions, a random internal device page size, I've seen: This fact just to upgrade a bit your technical background...).

Lots of people use PI as a low power device for home management with software like domoticz or jeedom. The device do the job very well and i's design is really close to many commercial solution. Those with a few software skills & eager to keep their IoT under their own control hacve contributed to PI success.

The only problem is with storage reliability. Solutions exist but cannot be buyed by individuals like me because we don't make a business (read again & understand why your answer is not fair...): The idea is for the Pi fundation or it's resellers to make the needed proxy able to buy industrial uSD & sell them to PI users that need them. When you are close to Farnell, this should not be a problem (and another source of revenues).

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23683
Joined: Sat Jul 30, 2011 7:41 pm

Re: Prevent SD-Card Corruption

Thu Dec 07, 2017 10:07 am

lost wrote:
Thu Dec 07, 2017 8:41 am
HawaiianPi wrote:
Wed Dec 06, 2017 11:09 am
Where did you get the idea that RPF was there to service your personal business needs?
(...)
If you are going to base a business around a product that was never intended for the use you have in mind, then it's up to YOU to figure out how to make it work.
That's clearly not my point & you know it. So be honest & stop laughing.

First point was there is a storage reliability problem that is not necessarily linked with unclean shutdowns (+ I've seen industrial devices with a clean cold restart path & an unclean hot one leading to worst corruptions, a random internal device page size, I've seen: This fact just to upgrade a bit your technical background...).

Lots of people use PI as a low power device for home management with software like domoticz or jeedom. The device do the job very well and i's design is really close to many commercial solution. Those with a few software skills & eager to keep their IoT under their own control hacve contributed to PI success.

The only problem is with storage reliability. Solutions exist but cannot be buyed by individuals like me because we don't make a business (read again & understand why your answer is not fair...): The idea is for the Pi fundation or it's resellers to make the needed proxy able to buy industrial uSD & sell them to PI users that need them. When you are close to Farnell, this should not be a problem (and another source of revenues).
Not cost effective to change the storage device. All the alternatives a are more expensive, often a lot more, so the Pi goes past the $35 price. Not going to happen, since this requirement would increase the cost for everyone, not just the people who need ultrareliable storage. So the majority will be paying for a feature they don't need.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

johnspackman
Posts: 18
Joined: Tue Jul 16, 2013 9:29 am

Re: Prevent SD-Card Corruption

Thu Dec 07, 2017 10:54 am

lost wrote:
Thu Dec 07, 2017 8:41 am
Solutions exist but cannot be buyed by individuals like me because we don't make a business
So, you're saying that you can't afford a mechanical hard disk? Or that only businesses have the scale to be able to attach a drive in a caddy?

The only solution we've used to improve reliability is to repartition and use ram disks to make a read only filing system, that periodically syncs the ram disk to SD disk, or write enabled the SD card for specific things (eg large downloads). I rolled our own solution for this, but there is an open source project somewhere that does this for you - so again, does not require a business to invest in technology or time.

We stick with SD cards because we want to keep the cost and complexity down, and do not want to add a hard disk. And it's obviously not necessary because we achieve 99.5% reliability with SD cards on an estate of 1,000 units

gkreidl
Posts: 6097
Joined: Thu Jan 26, 2012 1:07 pm
Location: Germany

Re: Prevent SD-Card Corruption

Thu Dec 07, 2017 11:29 am

jamesh wrote:
Thu Dec 07, 2017 10:07 am
lost wrote:
Thu Dec 07, 2017 8:41 am
HawaiianPi wrote:
Wed Dec 06, 2017 11:09 am
Where did you get the idea that RPF was there to service your personal business needs?
(...)
If you are going to base a business around a product that was never intended for the use you have in mind, then it's up to YOU to figure out how to make it work.
That's clearly not my point & you know it. So be honest & stop laughing.

First point was there is a storage reliability problem that is not necessarily linked with unclean shutdowns (+ I've seen industrial devices with a clean cold restart path & an unclean hot one leading to worst corruptions, a random internal device page size, I've seen: This fact just to upgrade a bit your technical background...).

Lots of people use PI as a low power device for home management with software like domoticz or jeedom. The device do the job very well and i's design is really close to many commercial solution. Those with a few software skills & eager to keep their IoT under their own control hacve contributed to PI success.

The only problem is with storage reliability. Solutions exist but cannot be buyed by individuals like me because we don't make a business (read again & understand why your answer is not fair...): The idea is for the Pi fundation or it's resellers to make the needed proxy able to buy industrial uSD & sell them to PI users that need them. When you are close to Farnell, this should not be a problem (and another source of revenues).
Not cost effective to change the storage device. All the alternatives a are more expensive, often a lot more, so the Pi goes past the $35 price. Not going to happen, since this requirement would increase the cost for everyone, not just the people who need ultrareliable storage. So the majority will be paying for a feature they don't need.
I don't think you got what he is asking for: He wants the Foundation to help end users (at least him) with buying industrial grade SD cards which are not available for normal end users.
Minimal Kiosk Browser (kweb)
Slim, fast webkit browser with support for audio+video+playlists+youtube+pdf+download
Optional fullscreen kiosk mode and command interface for embedded applications
Includes omxplayerGUI, an X front end for omxplayer

i486
Posts: 172
Joined: Sun Aug 28, 2016 3:41 pm
Location: BG

Re: Prevent SD-Card Corruption

Thu Dec 07, 2017 5:47 pm

gkreidl wrote:
Thu Dec 07, 2017 11:29 am
I don't think you got what he is asking for: He wants the Foundation to help end users (at least him) with buying industrial grade SD cards which are not available for normal end users.
I think industrial SD cards can be bought by every end user. Not tried but this link seems valid and offers prices for 1 piece:

https://eu.mouser.com/new/Swissbit/swis ... SD-memory/
https://eu.mouser.com/search/refine.asp ... =179605686

lost
Posts: 28
Joined: Tue Dec 05, 2017 9:38 am

Re: Prevent SD-Card Corruption

Fri Dec 08, 2017 6:13 am

johnspackman wrote:
Thu Dec 07, 2017 10:54 am
lost wrote:
Thu Dec 07, 2017 8:41 am
Solutions exist but cannot be buyed by individuals like me because we don't make a business
So, you're saying that you can't afford a mechanical hard disk? Or that only businesses have the scale to be able to attach a drive in a caddy?
Not that I can't, but won't: Want to keep everything in the small enclosure & low power (a HDD does not).

Now, being able through Farnell of reselling some industrial uSD (MLC with good WL algos, or some pSLC ones that offer endurance 10 times MLC one, with only doubling cost per bit): That's in the 25 to 50€ range for a 16GB depending on underlying NAND techno. A bit more than consumer grade devices but worth it.

That's just a question of making these industrial devices, that exists, available to individuals that take benefit of RPI low power needs for 24/7 application.

If providing a USB port on IO pinouts is so expensive to allow eUSB (not even speaking of M.2/Sata, I gave several options in my first post), one solution exist with no HW cost for those who don't need it.

Some says buying an oversized consumer grade SD for your needs will provide better endurance: It's not true, or marginal, as dynamic wear leveling create hot-spots on the fraction of the device that is often written (i.e. log files vs OS/program ones). Doing so is just wasting money vs the right sized device with adequate specs.

You can minimize writes, but not making everything read-only to keep a usable device (+some level of logs to understand problems post-mortem), that's only going from half a year before problems occurs to a full one or so. So still a major problem for these kinds of neds

lost
Posts: 28
Joined: Tue Dec 05, 2017 9:38 am

Re: Prevent SD-Card Corruption

Fri Dec 08, 2017 6:18 am

i486 wrote:
Thu Dec 07, 2017 5:47 pm
I think industrial SD cards can be bought by every end user. Not tried but this link seems valid and offers prices for 1 piece:
Even when available per 1, at the end of the buying process you usually have to give a business registry nb (like SIRET in France) for corporate tax management. At least for UE based sellers.

MaxK1
Posts: 1043
Joined: Sun Aug 26, 2012 11:34 pm

Re: Prevent SD-Card Corruption

Sat Dec 09, 2017 4:55 pm

It' isn't a big problem if you have a backup/restoration plan (and the backups!). Plan ahead! Usually,the people who moan and groan about corrupt SD cards are the ones who just lost a $#!+load of valuable "stuff" and didn't have a "good" backup strategy. By all means, do whatever you can to avoid data loss in the first place (treat the cards and the board nicely, feed them decent power, test your backup plan and revise as needed, etc.)
You are in a maze of twisty little passages, all alike.
When General Failure and Major Disaster get together, Private Parts usually suffers.

lost
Posts: 28
Joined: Tue Dec 05, 2017 9:38 am

Re: Prevent SD-Card Corruption

Wed Dec 13, 2017 7:23 am

MaxK1 wrote:
Sat Dec 09, 2017 4:55 pm
It' isn't a big problem if you have a backup/restoration plan (and the backups!). Plan ahead! Usually,the people who moan and groan about corrupt SD cards are the ones who just lost a $#!+load of valuable "stuff" and didn't have a "good" backup strategy. By all means, do whatever you can to avoid data loss in the first place (treat the cards and the board nicely, feed them decent power, test your backup plan and revise as needed, etc.)
It's your opinion, you may have kept for yourself, and it's false (I thankfully make backups) & not the topic: You make a confusion between reliability and backups. Maybe you're also among those that think RAID is about backup instead of... availability.

You make assumptions, so do I.

MaxK1
Posts: 1043
Joined: Sun Aug 26, 2012 11:34 pm

Re: Prevent SD-Card Corruption

Wed Dec 13, 2017 8:23 am

ALL forms of data storage are prone to complete loss of data. SD cards are no different. If you want (near) 100% availability, use ucarp and rsync or something similar.
You are in a maze of twisty little passages, all alike.
When General Failure and Major Disaster get together, Private Parts usually suffers.

lost
Posts: 28
Joined: Tue Dec 05, 2017 9:38 am

Re: Prevent SD-Card Corruption

Wed Dec 13, 2017 8:32 am

MaxK1 wrote:
Wed Dec 13, 2017 8:23 am
ALL forms of data storage are prone to complete loss of data. SD cards are no different. If you want (near) 100% availability, use ucarp and rsync or something similar.
Do you understand that, with specific hardware controlled by the PI, this would also double all the associated devices (for ucarp)? For just not dealing with the root cause: The SD that is in practice the PI SPOF!

And the rsync suggestion just don't make the job.

Pointless comment...

MaxK1
Posts: 1043
Joined: Sun Aug 26, 2012 11:34 pm

Re: Prevent SD-Card Corruption

Wed Dec 13, 2017 8:38 am

OK - so what are you proposing to "fix" the problem?
You are in a maze of twisty little passages, all alike.
When General Failure and Major Disaster get together, Private Parts usually suffers.

lost
Posts: 28
Joined: Tue Dec 05, 2017 9:38 am

Re: Prevent SD-Card Corruption

Wed Dec 13, 2017 8:55 am

MaxK1 wrote:
Wed Dec 13, 2017 8:38 am
OK - so what are you proposing to "fix" the problem?
RPi fundation is linked to Farnell. So, again, they could make industrial grade uSD available to end users that need it without any added cost on the PI hardware.

Consumer grade removable storage devices are not designed/tuned for long uptimes & thus do not implement good wear levelling algorithm that need time to operate in background & for some other reasons (erase counters stored in ram & used by internal FTL are lost on storage device firmware restart) need large uptimes to be efficient: A device with a use-case of being on just for storing or retrieving a few photos/videos cannot do background jobs.

lost
Posts: 28
Joined: Tue Dec 05, 2017 9:38 am

Re: Prevent SD-Card Corruption

Wed Dec 13, 2017 9:11 am

johnspackman wrote:
Tue Dec 05, 2017 12:59 pm
Last month, we broke the 1,000 mark on shipped devices based on Raspberry Pi and SD cards (...) I can count the number of SD card corruptions on one hand, including two instances where the user decided to remove the SD card while the Pi was running :o
Something like 5/1000 for a device that may be tuned not to write at all is not really what I would call good figures. As well, failure time vs deployment is unknown: IMO, it is just a question of time before your after sales being overflowed by device storage problems!

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23683
Joined: Sat Jul 30, 2011 7:41 pm

Re: Prevent SD-Card Corruption

Wed Dec 13, 2017 9:44 am

We won't be selling industrial cards. In our experience (i.e. testing) they give no benefits over a good quality off the shelf card. (i.e. Sandisk). We've even have one make actually lock up after a few uses, not corruption, just some sort of incompatibility.

If you mitigate the chances of corruption (use a RAM disk, network storage, take care if you do write to the card etc), you can make the devices very robust, but there is always the chance (as with any device with a storage medium, even a HD) that yanking the power can cause corruption.

That said, RS and Farnell I think, do actually sell industrial grade cards, so you can try it for yourself and see if in your particular use case they are an improvement.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

MaxK1
Posts: 1043
Joined: Sun Aug 26, 2012 11:34 pm

Re: Prevent SD-Card Corruption

Wed Dec 13, 2017 10:15 am

They do sell industrial grade cards (>100 quid/ea for an 8G card). I doubt they'll just toss one in at no extra cost with an order for a Pi...
You are in a maze of twisty little passages, all alike.
When General Failure and Major Disaster get together, Private Parts usually suffers.

ShiftPlusOne
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5967
Joined: Fri Jul 29, 2011 5:36 pm
Location: The unfashionable end of the western spiral arm of the Galaxy

Re: Prevent SD-Card Corruption

Wed Dec 13, 2017 10:27 am

People tend to recommend swissbit cards as good 'industrial' cards, but those are the ones which are incompatible:
https://github.com/raspberrypi/firmware/issues/838

I (ab)use the official cards quite heavily and haven't had any data loss - ever. In ~3 years, I've had two go read-only. Having said that, I'd probably avoid running a high traffic database off an SD card (use a USB HDD). If you want to go for high reliability, set up a read-only OS, with tmpfs for all the small writes and and overlayfs for data you have to keep (minimise those writes). That's an effective approach I've seen used by some digital signage vendors. They can deploy updates easily, by changing the squashfs root filesystem and all the user data is safe on the overlayfs.

Return to “Troubleshooting”