Gumby
Posts: 5
Joined: Sun Apr 21, 2013 11:38 pm

SD Card Corruption

Mon Apr 22, 2013 12:25 am

Hi all. I'm using some raspberry pis as a media player running mpd to play approx 8-10 mp3 files. We have thus far seen about 20 SD cards that have become totally corrupt, we can't read from them, write to them, or even format the card. The only thing running other than mpd is a few wget http queries run via cron. SInce the firt 10 or so cards were corrupt I changed from running arch linux to running raspian. I forget which exact version but it is running kernel 3.6.11+ I have mounted /tmp and /var/log as tmpfs. The RPI is rebooted each morning at 5AM. I am starting to see corruption in the SD cards with the new setup as well (raspbian and the tmpfs). Is there any more I can do to prevent the corruption?

I also should mention that I am using a range of cards all purchased from major resellers so the likelihood of them being fakes is relatively slim. Brands include Kingston, Adata, Sandisk, Duracell, and Lexar in 4GB and 8GB sizes.

User avatar
jbeale
Posts: 3500
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: SD Card Corruption

Mon Apr 22, 2013 12:54 am

You are not the only one, see this thread for example: http://www.raspberrypi.org/phpBB3/viewt ... 28&t=36533

Some recommend class 4 cards, better power supplies, etc. but its not clear what the exact problem is. It doesn't happen to everyone. I have 3 R-Pis and do not see this problem. One of my Pis has an uptime of 156 days, and the logs show no problems.

Gumby
Posts: 5
Joined: Sun Apr 21, 2013 11:38 pm

Re: SD Card Corruption

Mon Apr 22, 2013 1:05 am

Thanks for the reply. I have read that thread and unfortunately there doesn't seem to be any answers there for me. The cards I've been using are all class 4 cards and I have tested the PSUs (which I bought from directly from Newark with the PIs) and they seem to also be within spec. Back to the drawing board I guess unless someone manages to figure out why/how these cards are becomming corrupt. I've been seeing corruption after about two weeks of use with almost zero data being written to the cards :(

obcd
Posts: 917
Joined: Sun Jul 29, 2012 9:06 pm

Re: SD Card Corruption

Mon Apr 22, 2013 5:22 am

Maybe you already did, but one other thing you could try is disabling the swapfile as well.
This should bring all writing close to none.
(Some files like resolv.conf are still updated once during boot.)
I assume you are aware that overclocking also can cause corruption.

Gumby
Posts: 5
Joined: Sun Apr 21, 2013 11:38 pm

Re: SD Card Corruption

Mon Apr 22, 2013 5:51 am

Thanks for the suggestion obcd. Unforunately I have already disabled swap and I've never overclocked any of the RPIs

obcd
Posts: 917
Joined: Sun Jul 29, 2012 9:06 pm

Re: SD Card Corruption

Mon Apr 22, 2013 6:18 am

I assume you tried a setup with the partitions mounted read only in fstab as well?
It's a pretty scary problem. Specially since no one seems to know why it affects some of us and not others. All possible causes are brought up like supply voltage, chinese usb devices, not properly shutting down, non genuine sd cards, sun flux activity....which makes it very hard to figure out what really is going wrong. Next, people will point to the fact that the Pi is mentionned for educational purposes so that stability issues like sd card corruption are not considered high priority, followed by others that will say they don't have problems at all, so it must be something you are doing wrong.
Back to the facts now.
If a card becomes corrupted, does that mean it's no longer recognised by any device like a pc or digital camera?

User avatar
Jim JKla
Posts: 2218
Joined: Sun Jan 29, 2012 11:15 pm
Location: Newcastle upon Tyne UK

Re: SD Card Corruption

Mon Apr 22, 2013 7:27 am

That does rather depend on how it's corrupted.

After a car crash a car
may be drivable with paint scratches.
it may be driveable but unsafe as it has broken indicator lights
it could start but be undriveable due to a missing wheel
It could have a loose battery connection and be unable to start

In the same way an RPi SD card can be in a veriety of states post corruption the only way to re-establish it's integrity is to re-image the card.

One of the difficulties with SD card corruption is it's inevitable invisibility your card may be corrupted and yet run fine till you activate some function that acesses the corrupted area. ;)
Noob is not derogatory the noob is just the lower end of the noob--geek spectrum being a noob is just your first step towards being an uber-geek ;)

If you find a solution please post it in the wiki the forum dies too quick

Gumby
Posts: 5
Joined: Sun Apr 21, 2013 11:38 pm

Re: SD Card Corruption

Mon Apr 22, 2013 7:37 am

Jim JKla wrote:That does rather depend on how it's corrupted.

After a car crash a car
may be drivable with paint scratches.
it may be driveable but unsafe as it has broken indicator lights
it could start but be undriveable due to a missing wheel
It could have a loose battery connection and be unable to start
All of the SD cards I have used have been bought new, formatted, then OS installed and put into a Raspberry Pi immediately afterwards. If we go with the car crash analogy, the SD cards are are a convertible coup getting into a head on collision with a semi trailer with a full load. The only questions are why is the semi trailer crossing into oncoming traffic and who is driving (the OS, the hardware, etc)

RaspberryGeek
Posts: 20
Joined: Mon May 21, 2012 10:58 pm
Location: Evesham, Worcs
Contact: Website

Re: SD Card Corruption

Tue Apr 23, 2013 7:41 pm

Quite a few of us are running continuos data logging on weather stations, and we 've found that SD Cards have a limited number of write / rewite cycles, since we were writing and logging data every few minutes, they seemed to corupt the file systems at regular intervals (as opposed to irregular intervals).

I experienced the same using a Pi running XMBC after a fair bit of watching iPlayer and locally recorded content - I presumed that XMBC was using the SD Card as a cache so would also be doing a large number of write / rewrites.

If you are playing mp3 files, possibly it's the same issue?

obcd
Posts: 917
Joined: Sun Jul 29, 2012 9:06 pm

Re: SD Card Corruption

Wed Apr 24, 2013 8:27 am

The limited number of write / rewrite cycles is not the cause of the corruption. If an sd card starts to fail due to wear out of the flash technology, it's own controller will mark the bad blocks so that they are no longer used, and the available size of the sd card will become smaller.
Gumby has tried a lot of tricks to limit the writings to the card to a minimum. (Not sure if he tried a read only rootfs) He still get's corrupted sd cards that seems to be unrecoverable.
Maybe our description of what is happening is wrong, and the sd cards aren't corrupted but just broken.
After all, unless you set the read only bit, how would you be able to corrupt a sd card so that it can't be recovered? Writing every possible flash block to it's maximum perhaps? From a technical point of view, it doesn't make much sense, unless the internal sd card controller firmware is buggy as hell.

User avatar
jojopi
Posts: 3085
Joined: Tue Oct 11, 2011 8:38 pm

Re: SD Card Corruption

Wed Apr 24, 2013 10:16 am

obcd wrote:If an sd card starts to fail due to wear out of the flash technology, it's own controller will mark the bad blocks so that they are no longer used, and the available size of the sd card will become smaller.
I assume you do not mean that the card will report a smaller size to the operating system. The output of "cat /sys/block/mmcblk0/size" must never decrease, or the existing partitions and filesystems may no longer fit, and the card would need to be reformatted.

The number of addressable blocks reported is usually around 7% less than the total size of the flash. So there is a limit to the number of bad blocks that can be remapped before the card has to report media errors to the host and/or return bad data.

I am not sure if flash wear is the explanation, however. I have yet to wear out any cards after months of use, and I have taken no specific action to reduce writes. You can obtain the number of blocks written to the card since boot with the command:

Code: Select all

awk '{print $7}' /sys/block/mmcblk0/stat
It may be worth comparing the number of writes per day when running your application with the background level when the Pi is just idling and running the normal system cron jobs.

obcd
Posts: 917
Joined: Sun Jul 29, 2012 9:06 pm

Re: SD Card Corruption

Wed Apr 24, 2013 11:19 am

I agree that the size of the sd card reported to the OS should stay consistent.
But, shouldn't a low level format of the sd card make it usable again with a decrease in size?
I read an article of a firm that developed a fail safe rootfs partition type for sd cards. They tested it out and also reported several months before the card started to fail due to the flash chips being written 2 much.
rootfs corruption due to improper shutting down while the card is written makes sense, but again, shouldn't fubar the sd card. After a low level reformat, or even just re imaging the card, it should be usable again.

User avatar
jbeale
Posts: 3500
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: SD Card Corruption

Wed Apr 24, 2013 2:34 pm

In case it is relevant, here are the numbers from two of my Pis, one has been up a lot longer than the other one. Have not noticed any corruption to date. Both have 4 GB CF cards.

Code: Select all

[email protected] ~ $ awk '{print $7}' /sys/block/mmcblk0/stat
6079632
[email protected] ~ $ uptime
 07:29:39 up 159 days,  7:08,  1 user,  load average: 0.09, 0.04, 0.05

Code: Select all

[email protected] ~ $ awk '{print $7}' /sys/block/mmcblk0/stat
137936
[email protected] ~ $ uptime
 07:30:48 up 2 days, 12:25,  3 users,  load average: 0.24, 0.13, 0.10

Gumby
Posts: 5
Joined: Sun Apr 21, 2013 11:38 pm

Re: SD Card Corruption

Wed Apr 24, 2013 8:43 pm

@obcd

I missed your question a few posts back about mounting the card read-only. I'll give that a shot.

gabedot
Posts: 5
Joined: Tue Jan 01, 2013 4:51 pm

Re: SD Card Corruption

Wed Apr 24, 2013 8:51 pm

Hi!
I noticed this problem at my pi today. I have installed minidlna an when I let the service restart, nothing happens and syslog looks like that:

Code: Select all

Apr 24 08:05:01 raspberrypi /USR/SBIN/CRON[10477]: (root) CMD (service minidlna force-reload)
Apr 24 08:05:04 raspberrypi kernel: [835482.104589] mmcblk0: error -110 transferring data, sector 3359712, nr 8, cmd response 0x900, card status 0x200b00
Apr 24 08:05:04 raspberrypi kernel: [835482.104617] mmcblk0: retrying using single block read
Apr 24 08:05:06 raspberrypi kernel: [835484.791298] mmcblk0: error -110 transferring data, sector 3359712, nr 8, cmd response 0x900, card status 0x200b00
Apr 24 08:05:06 raspberrypi kernel: [835484.791331] end_request: I/O error, dev mmcblk0, sector 3359712
Apr 24 08:05:09 raspberrypi kernel: [835487.477987] mmcblk0: error -110 transferring data, sector 3359713, nr 7, cmd response 0x900, card status 0x200b00
Apr 24 08:05:09 raspberrypi kernel: [835487.478041] end_request: I/O error, dev mmcblk0, sector 3359713
Apr 24 08:05:12 raspberrypi kernel: [835490.164716] mmcblk0: error -110 transferring data, sector 3359714, nr 6, cmd response 0x900, card status 0x200b00
Apr 24 08:05:12 raspberrypi kernel: [835490.164750] end_request: I/O error, dev mmcblk0, sector 3359714
Apr 24 08:05:14 raspberrypi kernel: [835492.851418] mmcblk0: error -110 transferring data, sector 3359715, nr 5, cmd response 0x900, card status 0x200b00
Apr 24 08:05:14 raspberrypi kernel: [835492.851453] end_request: I/O error, dev mmcblk0, sector 3359715
Apr 24 08:05:17 raspberrypi kernel: [835495.538112] mmcblk0: error -110 transferring data, sector 3359716, nr 4, cmd response 0x900, card status 0x200b00
Apr 24 08:05:17 raspberrypi kernel: [835495.538144] end_request: I/O error, dev mmcblk0, sector 3359716
Apr 24 08:05:20 raspberrypi kernel: [835498.224797] mmcblk0: error -110 transferring data, sector 3359717, nr 3, cmd response 0x900, card status 0x200b00
Apr 24 08:05:20 raspberrypi kernel: [835498.224833] end_request: I/O error, dev mmcblk0, sector 3359717
Apr 24 08:05:22 raspberrypi kernel: [835500.912504] mmcblk0: error -110 transferring data, sector 3359718, nr 2, cmd response 0x900, card status 0x200b00
Apr 24 08:05:22 raspberrypi kernel: [835500.912538] end_request: I/O error, dev mmcblk0, sector 3359718
Apr 24 08:05:25 raspberrypi kernel: [835503.599255] mmcblk0: error -110 transferring data, sector 3359719, nr 1, cmd response 0x900, card status 0x200b00
Apr 24 08:05:25 raspberrypi kernel: [835503.599288] end_request: I/O error, dev mmcblk0, sector 3359719
After rebooting, everything seems to work perfectly again! :?

User avatar
jojopi
Posts: 3085
Joined: Tue Oct 11, 2011 8:38 pm

Re: SD Card Corruption

Wed Apr 24, 2013 9:13 pm

gabedot wrote:Apr 24 08:05:04 raspberrypi kernel: [835482.104589] mmcblk0: error -110 transferring data, sector 3359712, nr 8, cmd response 0x900, card status 0x200b00
After rebooting, everything seems to work perfectly again! :?
The fact that the reboot fixes it means that it must be somehow intermittent.

I had one card that used to do this. It got worse and worse until a reboot would no longer clear it because the problem would occur during the boot sequence. By that time I was starting to believe that the card was faulty, but I actually fixed it by thoroughly cleaning the contacts with 99% isopropyl alcohol.

gabedot
Posts: 5
Joined: Tue Jan 01, 2013 4:51 pm

Re: SD Card Corruption

Wed Apr 24, 2013 9:33 pm

okay... I have to make a little correction.
This time my raspberry stays dead... sounds like I have to rewrite the backup image to sd card again...

NickLanding
Posts: 3
Joined: Sat Jan 19, 2013 5:32 pm

Re: SD Card Corruption

Thu Apr 25, 2013 1:21 am

I managed to "Kill" an SD card while playing with turbo mode. i was able to reformat it using the tool on SD's website. (and that was the only way i could reformat it.)
I'm not sure if this will apply to you, but it may just save you from having to replace so many SD cards.

gabedot
Posts: 5
Joined: Tue Jan 01, 2013 4:51 pm

Re: SD Card Corruption

Thu Apr 25, 2013 5:59 am

Hello Nick Landing!

I am reformating and rewriting the image on the sd card the third time now. And I am not a lot overclocked - just the 2nd option from the top. Not the super 1GHz option... I don't get the point, why it is always destroying the sd card file system... The voltage ist still the same... :?:

User avatar
Jim JKla
Posts: 2218
Joined: Sun Jan 29, 2012 11:15 pm
Location: Newcastle upon Tyne UK

Re: SD Card Corruption

Thu Apr 25, 2013 6:16 am

Thats what we are trying to figure out gabedot.

Its one of the factors with seeking a solution on a forum remotely sometimes people offer solutions allready tried or without fully appreciating the problem.

Usually because they have not completly read the thread, or they have solved a similar problem thats not identical, or because the problem explanation was over vague,

Some problems are easy and require little or no detail but this is raely the case bare with usand theres a good chance we will find an answer.

Often the best answer to a wrong suggestion for any reason is "Nope tried that" ;)

Even wrong suggestions go towards keeping a thread active and live so increase your chances of a solution.
Noob is not derogatory the noob is just the lower end of the noob--geek spectrum being a noob is just your first step towards being an uber-geek ;)

If you find a solution please post it in the wiki the forum dies too quick

vilitux
Posts: 3
Joined: Fri Mar 29, 2013 9:22 am

Re: SD Card Corruption

Thu Apr 25, 2013 1:22 pm

Unfortunately the same problem here. 4 out of 5 pi's are crashing on a regular basis. I saw quite a few block errors, and a new image seems to resolve the issue temporarily. All are running Arch with the 3.6.11 kernel.

Currently I'm recompiling the kernel, plus compiling the 3.2.27 kernel and the newest 3.8.8 alpha. See if it matters something.

gkreidl
Posts: 6120
Joined: Thu Jan 26, 2012 1:07 pm
Location: Germany

Re: SD Card Corruption

Sat Apr 27, 2013 11:57 am

I have been running my Pi for about 6 months at medium overclocking (900 MHz) without any SD card problems. But yesterday I suddenly ran into problems. My Pi crashed and did not boot any more (I/O errors, could not access files on the ext4 partition).

Today I checked partitions on my Ubuntu system with fsck and they seemed to be ok. The Pi booted again, but the problems ocurred again after a few minutes (syslogd could not write to the ext4 partition). Finally I reset it to no overclocking at all and it ran fine for hours.

But I wanted to know, why it did not seem to be able to run with 900 MHz any more after doing so for months. So I considered, what I had changed yesterday: I had added a driver for technisat dvbs HD usb to /etc/modules. The problems occured, while the usb device was not connected, just the driver was loaded at boot time. I commented the driver out in /etc/modules, set overcklocking to 900 MHz again, rebooted and now it's running stable again for hours.

There seems to be some kind of problem with the sd card driver, when the dvb-usb-technisat-usb2 driver is loaded and the system is overclocked. It does not happen, when the system runs at 700 MHz. And it does also not happen running at 900 MHz, when the driver is not loaded.

Perhaps some of the sd card corruption problems people report are caused by similar (software) issues.
Last edited by gkreidl on Sat Apr 27, 2013 12:44 pm, edited 1 time in total.
Minimal Kiosk Browser (kweb)
Slim, fast webkit browser with support for audio+video+playlists+youtube+pdf+download
Optional fullscreen kiosk mode and command interface for embedded applications
Includes omxplayerGUI, an X front end for omxplayer

User avatar
Paul Webster
Posts: 807
Joined: Sat Jul 30, 2011 4:49 am
Location: London, UK
Contact: Twitter

Re: SD Card Corruption

Sat Apr 27, 2013 12:26 pm

I noticed some SD-card related firmware updates over recent days.
Might be worth a try with latest after making a backup of your card.

polarlight
Posts: 4
Joined: Sun Apr 28, 2013 9:42 am
Location: Finland

Re: SD Card Corruption

Tue Jun 04, 2013 8:46 pm

I've been managing 6 RPis running Raspbian since september 2012. Almost all of them have suffered from SD-card corruption in 1 to 3 months. The maximum uptime I've seen was about 6 months. The SD card corruptions have been verified by gparted and disk utility in Ubuntu. The disk utility says "NOT clean" and the details refer to corrupted data.
I just recently changed /var/log into tmpfs, but despite of that, I have already seen two RPi SD-card deaths after that change. These have been running home-automation tasks, some of those running Motion for video surveillance and some were moderately overclocked, some running with default clocking, some ethernet connected some with WiFi.

What is common to all is that I have had USB-devices connected, either WiFi, USB GSM modem, serial port or a webcam and they have been running 24/7. Most of these have also been backed up by UPS supplying the electricity to the good brand power adapters with enough juice, so not even power failures seem to be the explanation, even though some of my cases indicate that power failures can trigger the event. Just yesterday one RPi SD-card was corrupted after a power failure occured - that RPi was not behind UPS then obviously.

As a reference, in the same environment, I have two Dreamplugs running with the same apps as these RPis - and using SD-cards no differ from RPis and they have never ever ruined their SD-cards even when /var/log has been on the SD-card, like everything else. Their uptimes are now 186 and 275 days and downtimes have been caused by normal maintenance operations, not because of system failures.

So to me it looks like if there would be a fundamental stability vulnerability in the default RPi HW+Raspbian setup. If it is the writes to SD-card that sometimes fail, then making the file system read only or using jffs (or alike) could help. However, that would be covering up the real problem.
I hope my report would help in finding the problem or at least in avoiding looking in all the wrong places.

simplesi
Posts: 2327
Joined: Fri Feb 24, 2012 6:19 pm
Location: Euxton, Lancashire, UK
Contact: Website

Re: SD Card Corruption

Tue Jun 04, 2013 9:00 pm

Having had pretty much had all my SD cards trashed - I stopped over clocking and never had another failure.

In the summer hols I'm going to go back to trying to overclock again (as an extra 40% speed is not to be sniffed at) and see if I get any better results

Simon
Seeking help with Scratch and I/O stuff for Primary age children
http://cymplecy.wordpress.com/ @cymplecy on twitter

Return to “Troubleshooting”