jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 19582
Joined: Sat Jul 30, 2011 7:41 pm

Re: Prevent SD-Card Corruption

Wed Dec 13, 2017 11:17 am

MaxK1 wrote:
Wed Dec 13, 2017 10:15 am
They do sell industrial grade cards (>100 quid/ea for an 8G card). I doubt they'll just toss one in at no extra cost with an order for a Pi...
Why would they? If you want industrial grade, you'll have to pay for it.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Please direct all questions to the forum, I do not do support via PM.

MaxK1
Posts: 1045
Joined: Sun Aug 26, 2012 11:34 pm

Re: Prevent SD-Card Corruption

Wed Dec 13, 2017 1:30 pm

@lost suggested that they could... LMAO
You are in a maze of twisty little passages, all alike.
When General Failure and Major Disaster get together, Private Parts usually suffers.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 19582
Joined: Sat Jul 30, 2011 7:41 pm

Re: Prevent SD-Card Corruption

Wed Dec 13, 2017 1:54 pm

MaxK1 wrote:
Wed Dec 13, 2017 1:30 pm
@lost suggested that they could... LMAO
No, I think the suggestion was no additional cost on the HW. There would clearly be additional cost on the card .

However, as above, Swissbit don't work correctly (and yet all other SD cards do work correctly), and in our testing, a good quality e.g. Sandisk is just as good as an industrial card. Moot point anyway, we won't be specifically selling an industrial rated card. They are available to just go out and buy, should you wish to.

One thing to note, since I have just gone through this thread. AFAIK, SD card corruption (rare thought it is) is almost always caused by yanking the power - as with any device. There are other SD card failure mechanisms, it might 'just fail' of its own accord. But we are not aware of any HW issue with the SD card interface, or the driver software itself.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Please direct all questions to the forum, I do not do support via PM.

lost
Posts: 19
Joined: Tue Dec 05, 2017 9:38 am

Re: Prevent SD-Card Corruption

Fri Dec 15, 2017 7:55 am

jamesh wrote:
Wed Dec 13, 2017 9:44 am
We won't be selling industrial cards. In our experience (i.e. testing) they give no benefits over a good quality off the shelf card. (i.e. Sandisk). We've even have one make actually lock up after a few uses, not corruption, just some sort of incompatibility.
Well, is this the "incompatibility" you think about:
https://github.com/raspberrypi/firmware/issues/838

So, after getting plugged on another device (but not as a boot device I presume!) this just works!

It's probably too late, but if you had dmesg'ed just after device mounting, my bet you would have got a journal replay message of the ext4 partition...

=> This may not be a device incompatibility, but just the PI boot loader simplified ext reader is not able to do this journal replay a full ext4 driver under OS will do.

If I'm right, any SD can trigger this kind of behavior: You plug the device on a Linux PC, everything looks correct... you retry on the target/PI... this just works again! When this kind of thing happen, check logs, there's good chances there is no mystery for a device that suddenly stops booting.

lost
Posts: 19
Joined: Tue Dec 05, 2017 9:38 am

Re: Prevent SD-Card Corruption

Fri Dec 15, 2017 8:19 am

jamesh wrote:
Wed Dec 13, 2017 1:54 pm
we are not aware of any HW issue with the SD card interface, or the driver software itself.
I already answered on what you presume is an incompatibility & in my experience may not. But I agree problems are usually not on SoC storage subsystems (with IPs shared in so many devices) nor drivers (on a world of workloads, problems are quickly spotted).

I only saw 2 software flaws: The ext reader of u-boot unable to do journal replay when needed on a boot partition, leading to an unbootable device (but in the industry we usually have a fallback mechanism on another boot partition) and another which was linked to RT patches. Applications were sometimes preempting kernel (only possible with theses patches) between scsi command and data phase, leading to an internal timeout with some storage devices that were thus disconnecting themselves until a reboot.

All problems I've seen were device problems. Even industrial ones, you're true, but usually after weeks/months of testing when consumer grade did not last a full day of our own test suites. So, in my experience, there is a difference even if controllers are usually shared with consumer grade ones. The difference is the way NAND chips are used to avoid caching and firmware that is tuned for fixed storage use instead of removable (a static WL have no time to trigger on a removable device, so they use dynamic & create hot-spots as already stated).

But if you're not willing to do anything about it, I'll just get some samples when we'll have to select a new source... Just take care competition does not address this problem providing a fixed storage interface (M.2) to be able to cope with this using reliable internal storage that users can buy: Nature hates vacuums!

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 19582
Joined: Sat Jul 30, 2011 7:41 pm

Re: Prevent SD-Card Corruption

Fri Dec 15, 2017 10:31 am

lost wrote:
Fri Dec 15, 2017 7:55 am
jamesh wrote:
Wed Dec 13, 2017 9:44 am
We won't be selling industrial cards. In our experience (i.e. testing) they give no benefits over a good quality off the shelf card. (i.e. Sandisk). We've even have one make actually lock up after a few uses, not corruption, just some sort of incompatibility.
Well, is this the "incompatibility" you think about:
https://github.com/raspberrypi/firmware/issues/838

So, after getting plugged on another device (but not as a boot device I presume!) this just works!

It's probably too late, but if you had dmesg'ed just after device mounting, my bet you would have got a journal replay message of the ext4 partition...

=> This may not be a device incompatibility, but just the PI boot loader simplified ext reader is not able to do this journal replay a full ext4 driver under OS will do.

If I'm right, any SD can trigger this kind of behavior: You plug the device on a Linux PC, everything looks correct... you retry on the target/PI... this just works again! When this kind of thing happen, check logs, there's good chances there is no mystery for a device that suddenly stops booting.
That is the issues, yes, with some Swissbit cards. The card appears to get itself in some weird state - it's the only card we have ever tested that has done this, Swiss bit say it's us, we disagree, because all (ALL) other cards work ok.

I wasn't present when the tests were done, but I'm pretty sure the engineers involved in the testing were thorough.

As for competition filling a hole in the market. They have had 5 years, and 17M devices to practice on. Not done it yet. However, never say never, if someone want to produce a high reliability flash storage system that plugs in to a Pi to replace the SD card, they are more than welcome. Probably is a small market there for something that isn't insanely expensive. However, I suspect a small USB attached SSD would probably work fine, albeit much slower than the SD card interface.

As I said, we won't be selling industrial cards - they are already available elsewhere, and since we do not believe there is a HW issue, we are not looking at any changes in that area. If anyone can show a reproducible HW fault, then clearly we will look in to it.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Please direct all questions to the forum, I do not do support via PM.

lost
Posts: 19
Joined: Tue Dec 05, 2017 9:38 am

Re: Prevent SD-Card Corruption

Sat Dec 16, 2017 9:01 am

jamesh wrote:
Fri Dec 15, 2017 10:31 am
That is the issues, yes, with some Swissbit cards. The card appears to get itself in some weird state - it's the only card we have ever tested that has done this, Swiss bit say it's us, we disagree, because all (ALL) other cards work ok.
Hello, Did tout really read(&understood) what I think the root cause may (in my "small" experience of 20 years of base core software) be? Because I'm not as sure as you this one was digged with good care!

User avatar
rpdom
Posts: 12503
Joined: Sun May 06, 2012 5:17 am
Location: Essex, UK

Re: Prevent SD-Card Corruption

Sat Dec 16, 2017 9:29 am

lost wrote:
Fri Dec 15, 2017 7:55 am
jamesh wrote:
Wed Dec 13, 2017 9:44 am
We won't be selling industrial cards. In our experience (i.e. testing) they give no benefits over a good quality off the shelf card. (i.e. Sandisk). We've even have one make actually lock up after a few uses, not corruption, just some sort of incompatibility.
Well, is this the "incompatibility" you think about:
https://github.com/raspberrypi/firmware/issues/838

So, after getting plugged on another device (but not as a boot device I presume!) this just works!

It's probably too late, but if you had dmesg'ed just after device mounting, my bet you would have got a journal replay message of the ext4 partition...

=> This may not be a device incompatibility, but just the PI boot loader simplified ext reader is not able to do this journal replay a full ext4 driver under OS will do.

If I'm right, any SD can trigger this kind of behavior: You plug the device on a Linux PC, everything looks correct... you retry on the target/PI... this just works again! When this kind of thing happen, check logs, there's good chances there is no mystery for a device that suddenly stops booting.
The Pi bootloader doesn't read ext at all. It reads vfat, then loads the Linux kernel which has the full ext4 driver included.

Somewhere else I believe you mentioned the uboot bootloader, which would be an issue with uboot, not the Raspberry Pi.

lost
Posts: 19
Joined: Tue Dec 05, 2017 9:38 am

Re: Prevent SD-Card Corruption

Mon Dec 18, 2017 10:40 am

rpdom wrote:
Sat Dec 16, 2017 9:29 am
The Pi bootloader doesn't read ext at all. It reads vfat, then loads the Linux kernel which has the full ext4 driver included.
Somewhere else I believe you mentioned the uboot bootloader, which would be an issue with uboot, not the Raspberry Pi.
Issue may have been shared with another boot loader as they usually implement simplified FS drivers (readers). And this can cause problems with journalized ones but mostly after an upgrade of one of the few files needed for boot (those read by the boot loader): Always more reliable than an unjournalized FS like FAT, even with this common boot loaders flaw, but this can be quite surprising.

Other issues on boot loader side can also be linked to how LBA accesses are tuned: To avoid latencies, under OSes, they are usually split in short consecutive LBAs sequences even for files with low fragmentation (this have long been 140 LBAs for Linux usb-storage, always 128 for Windows, for instance ; 1024 for the usual SCSI layer). With block nb very far from what is allowed by a read/write_10 command (accepts up to a u16, so 65535 blocks).

This is where issues can occur with devices like SD/USB whose use-case is to be removable (i.e. only read/written from already started OSes, not some boot loaders): You may thus see problems that managed to pass manufacturers testing if your boot loader can read much more blocks at once because it's FS reader is tuned for boot time performance.

And you'll only see problems when you have files fragments big enough to reach internal device limits (that can be linked to device buffering limits or even firmware mistakes! I've seen a device unable to boot when linux kernel+init rootfs file had fragments over 16MB. With 512b block size, this was a half u16 size... The bug was a s16 bad cast on read_10 parameter in storage controller firmware, thus seing block nb values over 32767 as negative & hanging FW immediately).

The problem is that a storage manufacturer will only clean his own mess if you're able to provide a test able to reproduce the problem, preferably on a standard platform like a Linux PC.

billyhk
Posts: 1
Joined: Tue May 01, 2018 4:20 pm

Re: Prevent SD-Card Corruption

Tue May 01, 2018 6:11 pm

I started to use pi 3 for my design as controller of DIY audio amplifier last year, I got same weird problem too. But I think that is not problem of SD card, because many products are using SD card or CF card in market without this problem such as smartphone, camera, PLC and HMI etc.

I tested the raspberry pi 3 in my design for few months, I found a problem causing SD card corruption under condition as follows,
Windows 10 IOT OS installed
pi 3 connected to Linksys router through Ethernet port
Power supply of pi3 connected to 220VAC 4 way socket

Then I repeated to plug and unplug the power supply of other device connected to the same 4 way socket for few times, the SD card of pi3 was easily corrupted. Finally, I removed the Ethernet cable from pi3, and repeat the test, the SD card of pi3 was not corrupted again. Besides, according to the schematic of pi3, I found that the filter ground of pi3's Ethernet port is connected to pi3's digital ground, I think that the noise from router influences stability of SD card and other components. So I think that is one of root cause of SD card corruption.

Now I do not use the Ethernet port for my design, but there is one time of SD card corruption occurred in the past two months, there may be other noise influence.

Return to “Troubleshooting”

Who is online

Users browsing this forum: meisam.arash and 44 guests