SD card corruption after a few reboots.

Howdy gents! Greetings from sunny california. I come to these forums becasue my experience with my precious Pi has been somewhat disappointing thus far.

My setup:
Bone stock RasPi. No overclock, no mods.

PSU used:
- Motorola phone charger (5v@0.85A)
- HP touchpad charger (5.2V@2.0A)
(Both of these models are listed as compatible here.)

SD card:
- Microcenter 4GB Class 4 SDHC Made in Japan. (not listed in wiki)
- SanDisk 8GB Class 4 microSDHC w/ Samsung adapter. Made in China (not listed in wiki and likely a fake)
- Kingston 8GB Class 4 microSDHC w/ Kingston adapter. Made in China. (listed as compatible in wiki)
- Patriot 16GB Class 10 SDHC LX Series. Made in Taiwan (listed as compatible in wiki)
- Lexar 8GB Class 6 SDHC Platinum II. Made in Korea. (listed as compatible in wiki.)

Inputs:
- Generic USB keyboard + mouse.

Outputs:
- HDMI and Composite

Images:
- Lastest Raspbmc from here.
- Debian image from main download page.
- Debian wheezy image from here.

I experience massive root filesystems corruption no matter what SD card or image I use after only a couple of reboots. These symptoms are very similar to those described in this post.
The best SD cards thus far have been the lexar and the patriot (the ones who subjectively seem to survive the most reboots). The San Disk won't boot no matter what.

After a fresh flash the Pi will usually happily boot with no errors and quite quickly at that (sub 30 secs to login prompt on the wheezy beta). Alas... after only a few reboots of the system (and by reboot I certainly mean something gentle such as sudo /sbin/init 6 or shutdown -r NOW) things will quickly derail into utter uselessness. At first the boot will show either errors, timeouts or missing files. Eventually the system will no longer boot usually with a "Linux Kernel panic: VFS: Unable to mount root fs". At that point I will pop the SD card out of the pi and mount it onto another computer and fsck the root partition. fsck will show a couple hundred errors from missing inodes to invalid checksums and other miscellaneous problems. The SD card may or may not boot on the pi after that or the system will be unusable because of some missing files. (once the sudoers file disappeared... )

I believe I have done my due diligence by using the pi with "official" images and "approved" or "known to work" hardware. I am fairly linux literate in the sense that I can clearly see that the PI is badly damaging my root fs after only a couple of reboots on good SD cards. So badly in fact that even using a resilient journaling filesystem like ext4 is pointless.

The fact that I was able to find on the forum at least one other user with identical issues at least comforts me slightly (misery loves company right?). It also worries me that this may be a more widespread issue with the pi hardware. Or maybe I simply have a bad egg? I have received my Pi on July 11th.

Please tell me I'm doing something stupid or missing something simple. I'd rather it being that than having to ask for a refund or worse wait for an RMA.

Thanks! And best regards.

[mahjongg]

IMHO this is not a well known issue. The only issues I have seen are either problems arise when

1) the PI receives some power from a powered hub, via the USB polyfuses when the main PSU is powered off, but not the hub, and the hub connects 5V to its upstream connector, which may lead to unwanted behaviour, or
2) forgetting to do a clean shutdown after first boot, when the software writes back many parameters to the card, which stay stuck in the write cache if you do not do a clean shutdown.
Generic card corruptions like you describe are not common AFAIK.

The case you quote as "similar" is a completely unrelated power issue.

Sorry, my link was wrong re the other user experiencing similar issues.

I meant this one:
viewtopic.php?f=28&t=10581

also I always reboot cleanly my pi. Despite that fact, corruption arrises.

[mahjongg]

Sorry, my link was wrong re the other user experiencing similar issues.

I meant this one:
viewtopic.php?f=28&t=10581

also I always reboot cleanly my pi. Despite that fact, corruption arrises.

And that one seems, to be caused by an exemplary error, that is bad RAM in just one part.
Not a "widespread issue".
I think that if this was a widespread issue, with some 400.000 R-PI's in use we would have known by now.

Perhaps you should just return your R-Pi, and test another sample.

I have initiated an RMA with newark/element 14 with which I had placed my order.

I understand that this may very well be an isolated incident or at least a rare case. However, the very friendly newark rep on the phone told me that they have in fact a *dedicated* team to handle raspberry pi returns because they are having "a lot". While entirely subjective and certainly not related to my particular issue, this clearly tells me that there are indeed major reliability/quality assurance issues with these boards beyond simple user error. You now have two documented issues with closely matched symptoms relating to major SDcard unreliability on these forums alone. I understand that the Pi is a cheap computer and that early boards will have teething issues but I would not dismiss this issue so quickly.

I am posting this thread mainly for documentation purposes to avoid other user of the Pi with similar symptoms wasting valuable time on a busted board.

I'll keep you guys posted once I receive my replacement and I certainly hope this is an isolated incident.

I understand that this may very well be an isolated incident or at least a rare case.

I had similar issues with SD card corruption. Two things I noticed: -

1. My SD card needed to be firmly and evenly placed in its slot (possibly a red herring, but there is a contact that's made when the card is fully inserted, which I assume serves a purpose).

2. After doing firmware updates via Hexxeh's Raspbian image ( http://www.raspbian.org/HexxehImages running rpi-update to get firmware), my Pi has been absolutely rock solid, and now even works with an old generic 2GB card that previously didn't even get to the bootloader.

My guess is that the firmware updates fixed the pull-up resistor issue in the older firmware. As I understand it, pull-up configuration is persistent between power cycles, so the firmware was then able to get to the bootloaders on old cards, and the pullups made the SD interface reliable with all cards.

The last item in the Raspbmc FAQ states that they are using a recent firmware and that generally I should not be mucking around with it.
http://www.raspbmc.com/wiki/user/freque ... questions/

I am also experiencing the same issue with the beta wheezy image which AFAIK ships with one of the latest firmware.

Now maybe I am misunderstanding what "firmware" actually refers too in RasPi land. My understanding is that it is the bootcode.bin file present on the boot partition of the image.

Please correct me if I am wrong.

Now maybe I am misunderstanding what "firmware" actually refers too in RasPi land. My understanding is that it is the bootcode.bin file present on the boot partition of the image.

Please correct me if I am wrong.

I consider the firmware to be the bootrom (built into the GPU), along with the files loader.bin, bootcode.bin, and start.elf that reside in the boot partition (/boot in Linux).

Firmware files in /boot are local to your SD card - any card you use will provide whatever firmware files are in its /boot partition, so nothing you do with, for example, a Raspbian SD card will affect your Raspbmc installation.

Well, after all those red herrings and wrong conclusions, it turned I did have a hardware fault - a wonky pin in my SD card slot. It was bent to the side, so there was a poor connection to pin 8 of my cards.

I suppose the particular card on which I had Hexxeh's Raspbian image had a slightly more fortunate fit.

Gently bending the pin back in line with a small screwdriver has fixed it, and no amount of wiggling the card is causing errors.

[mahjongg]

Well, all is well that ends well I guess
Happy computing with your Raspberry PI,
have fun now.

Why was this marked as solved? My problem remains.
I am the author of this thread. Not ksangeelee.

Why was this marked as solved? My problem remains.

Are you able to force the failure of a card in a running system? My (very unscientific) approach was to boot to a command prompt, run 'find /' or 'ls -lR /' in a terminal (or on the console).

While the directory listing was running, I squeezed the card slot connectors and pressed the card from side to side. It became obvious that movement was causing errors (the file listing would pause for a couple of seconds), so I just used a magnifying glass to look a bit closer (at the card slot's connectors and solder joints).

Running 'dmesg' periodically will show any system errors that have occurred.

[mahjongg]

kraz wrote:
Why was this marked as solved? My problem remains.

My bad, removed the offending "(solved)" until all problems in this thread are solved.

The last item in the Raspbmc FAQ states that they are using a recent firmware and that generally I should not be mucking around with it.
http://www.raspbmc.com/wiki/user/freque ... questions/

IMHO They're doing early adopters a disservice because in many cases updating the firmware actually solves many problems people are experiencing. If you are having problems then dont be afraid to update the firmware!

I am also experiencing the same issue with the beta wheezy image which AFAIK ships with one of the latest firmware.

You realize the 'latest firmware' is not necessarily the latest. In fact I ran rpi-update last night and it updated some stuff. Images can get out of date particularly in regards to firmware quite quickly.

Now maybe I am misunderstanding what "firmware" actually refers too in RasPi land. My understanding is that it is the bootcode.bin file present on the boot partition of the image.

Please correct me if I am wrong.

I believe you are right... It resides in /boot or something. This means that everytime you re-image your SD card you need to redo the firmware updates. Unless you back up your image somehow.

Before I return my current pi board I am going through other possibilities just to make sure.

I have installed the rpi-update script. I read the source and this script essentially update *.elf and *.bin and the kernel .img file in the /boot partition. So by "firmware" what is actually meant is whatever is contained in /boot.

I have setup the beta wheezy image with the latest firmware updated through rpi-update and flashed he lot on my "best" full size sdcard. The card fits snugly in the slot as opposed to my other microSD card adapters which are a little loose. I would actually advise peoples against using these adapters in their pi. It's yet another mechanical component that can fail.

I have also found that my motorola charger might be a little weak at only 850mA and with the usb polyfuse dropping voltage a bit too much. My HP touchpad adapter with 5.2@V@2A seems to provide better levels. I also make sure to power off the pi by turning off the switch on a power strip instead of yanking the usb header. I'm not sure if that would make a difference but it seems gentler.

So far the /root filesystem has survived a dozen reboots which is pretty good and I am not seeing anything peculiar in dmesg yet.

Hopefully I've cracked it. I'll keep you posted.

[mahjongg]

If the PI ever boots, never just pull the plug afterwards, that will almost guarantee damage to the card contents!, as its last writes are probably still in the cache memory and will be lost when you pull the plug (mains or otherwise, doesn't matter) always do a software controlled shutdown, so in the shell do a:

sudo shutdown -h now

or shutdown from the GUI (start the GUI with startx)

Only pull the plug when the software says its ok to do so.

So far the /root filesystem has survived a dozen reboots which is pretty good and I am not seeing anything peculiar in dmesg yet.

Hopefully I've cracked it. I'll keep you posted.

Sounds promising.

Note that Dom committed an significant modification to start.elf an hour ago. If you ran rpi-update over an hour ago, I'd recommend you run it again to get this latest file (fixed occasional mmc0 errors, though they weren't destructive, just slowed things down).

If your Pi continues to survive the reboots, even after its temperature stops rising (e.g. ruling out heat affecting a solder joint), then I'd suspect something physical (e.g. my contact pin problem, or perhaps a dry joint). In any case, you should be able to apply light pressure movement to any part of the Pi without it crashing.

If the PI ever boots, never just pull the plug afterwards, that will almost guarantee damage to the card contents!, as its last writes are probably still in the cache memory and will be lost when you pull the plug (mains or otherwise, doesn't matter) always do a software controlled shutdown, so in the shell do a:

sudo shutdown -h now

or shutdown from the GUI (start the GUI with startx)

Only pull the plug when the software says its ok to do so.

From my OP:

Alas... after only a few reboots of the system (and by reboot I certainly mean something gentle such as sudo /sbin/init 6 or shutdown -r NOW)

My extra precaution with the power strip is just that. Extra precaution.

Besides, ext4 being journaling filesystem, it is designed to prevent filesystem corruption in such cases when the machine crashes or loses power in the middle of a write operation. In all fairness waiting for the machine to halt before powering it off is not entirely required with any journaling filesystem such as JFS, ext3 or ext4.

Still hacking at it and I'm going o try that newest firmware with dom's latest commit.

Thanks guys.

...waiting for the machine to halt before powering it off is not entirely required with any journaling filesystem such as JFS, ext3 or ext4.

I read in a recent thread (which unfortunately I can't find now) that SD cards can be destroyed if they lose power at an inopportune moment during a write operation. Perhaps relatively unlikely, and I guess it might depend on the card's low-level firmware.

I recently had a new card die on me, and nothing I tried recovered it - it had been pulled out during a 'dd' write operation that I thought had failed.

Still hacking at it and I'm going o try that newest firmware with dom's latest commit.

Is the corruption still there, or has your Pi been stable?

Hello. I just want to chime in that I'm having a similar problem. It seems that after a certain amount of write activity, my filesystem slowly begins to get corrupted, to the point that packages will fail to run, modules will fail to load, and the Raspberry Pi will fail to boot. It even happens during use, before a shutdown or reboot. Running fsck on another machine reveals lots of missing inode errors, missing flags, bad filesizes, and all sorts of other things.

I'm using a 5v 1A micro usb charger that's originally for my smartphone. I'm also using a 32GB SD card, so maybe that's the issue, I don't know. I'm going to try using different cards and see what happens!

[jamesh]

I believe there is an SD card fix very imminent - should be available with rpi_update I think.

I'm going to chime in on this one...

I too am having many SD card corruption issues regardless of what kind of SD card I use. It seems to be fine for a while, but it will slowly start to error out more and more when loading until it no longer boots at all.

I'm currently sitting at a "#!/bin/sh: can't access tty; job control turned off" message upon boot. Keyboard is useless.

Popped the SD card into another computer, fscked it, and it found many errors, which it attempted at fixing, but it still will not boot. Stuck at the same message, minus some of the errors before hand.

Although it's probably easier to just rewrite the image to the SD card, this is a rather annoying problem. Shouldn't a journaled file system be less prone to corruption?

Besides, ext4 being journaling filesystem, it is designed to prevent filesystem corruption in such cases when the machine crashes or loses power in the middle of a write operation. In all fairness waiting for the machine to halt before powering it off is not entirely required with any journaling filesystem such as JFS, ext3 or ext4.

Even a non-journaled filesystem *SHOULD* be safe if it has had no writes for a few minutes because sync is (or should be) scheduled to run quite frequently.

Having said that I had my first SD card corruption after accidentally turning off. But that was after half an hour of it doing nothing so I wouldn't really expect either active writes, or pending writes.

If we have to run shutdown even on apps not actively writing its certainly going to stifle creativity.

For example I have a Pi strapped to the back of the main TV. Its job is media centre. OK the code is only part written so at present it is a music recordings player. Its power is from the amplifier (switch on the amp for the main speakers and the Pi comes on). If you want to listen to radio you select the source on the tv to get to the GUI, and select the input on the amp, and away you go.

My App can only write to disk when you sit at the keyboard and tell it to do things. There are no writes when it is idle, or when a file is playing (using omxplayer).

When you are finished you turn the amp off and the Pi dies. No problems on that one yet.

No my crash is on the second Pi (taking over the development role), which has an SD card from RS (pre-loaded but that quickly got overwritten). One power down when idle and its dead.

If it turns out that shutdown is really required then that mandates a short term power reservoir (eg supercap), a power off detector generating an interrupt into GPIO and some software to launch shutdown before the supercap dies. Messy

I already asked this question several times without real answer.
What is corrupting the sd card when we cut the power?
1. Some say it's the internal housekeeping controller of the sd card itself.
It might be in the middle of reorganising sectors to get equal write cycles for every flash block.
When that process get's interrupted, some sectors might become corrupt.
2. Others think it's linux itself and believe that for instance a read only root filesystem could fix it.
3. Maybe it's the soc losing control during the power drop and corrupting the sd card.
So I asked the foundation if the broadcom soc had a brownout detection holding the Soc reset when the supply becomes 2 low, but I never got an answer on that question which is why I leave option 3 open.
If the cause is option 3, it could be fixed with the new revision 2 boards and some extra hardware.
The reset pin of the soc chip was brought out on a connector for that board revision.
I ran some tests where I made the watchdog reset the Pi. It never corrupted the sd card.
Maybe I was lucky or option 2 is not the problem. Puppy Linux for instance loads the linux OS in ram and let's it run there. It's used very often in embedded x86 setups. A read only root fs is only usefull if you have a separate medium for the OS that shouldn't be written or easily writable. Some embedded linux devices have a flash chip on board for that purpose. I am thinking about routers and for instance the roku 2 media player.
Case 1 is difficult to prove. Maybe people with different brand of sd cards and multiple Pi's could prove that indeed 1 SD card is frequently corrupting and the other hardly ever. It would prove option 1 is happening.
You seem to have the perfect setup for that Howard.
I wonder how other embedded linux boards handle this situation, besides a separate read only storage device for the OS?

Looking around the web there are a few references to brownout problems on SD cards (eg http://forum.embeddedarm.com/archive/index.php/t-3.html)

In normal use SD cards are probably turned off abruptly either by removal or by a computer cutting power to the slot.

The power supplies we use (or the many other groups that have the same problems) do not cut off abruptly and there is certainly scope for an SD card that does not *ITSELF* have brownout detection to damage itself.

Perhaps instead of holding the power up and interrupting the Pi, a simpler (though less ideal) solution might be to detect brownout and cut power immediately. Resetting the ARM would not work as the CPU (or I/F controller) in the SD card can keep working to destroy itself.