SD corruption after freeze


24 posts
by garyj » Sun Jan 13, 2013 5:08 am
My pi occasionally freezes. After power-cycling, SD card often has very bad corruption (either won't boot w/ kernel panic or boots but super bad fsck failures that eventually require re-imaging).

I've tried with two different SD cards, imaged from two different machines.

I've tried with Xbian, Rasbmc, and Rasbian.

I am not overclocking. I've changed no default settings. I usually can't even get through a couple simple Xbian XBMC setting customizations before things freeze and corrupt.

Is it normal for SD cards to get corrupted in this way? Is there something I should be doing to prevent it?
Posts: 9
Joined: Sun Jan 13, 2013 5:03 am
by malakai » Sun Jan 13, 2013 7:58 am
This is normal if your not shutting down the Pi by typing:

Code: Select all
sudo shutdown -h now


There are variations on that command -h means halt now is the time to shutdown without issuing those commands you can't guarantee that something needs to be written to the card to prevent corruption.

Once done the green light will blink about 5 times consecutively that's when you know it's safe to disconnect power.
http://www.raspians.com - always looking for content feel free to ask to have it posted. Or sign up and message me to become a contributor to the site. Raspians is not affiliated with the Raspberry Pi Foundation. (RPi's + You = Raspians)
User avatar
Posts: 1383
Joined: Sat Sep 15, 2012 10:35 am
by garyj » Tue Jan 15, 2013 4:33 am
:(

So if I'm running any app that might freeze, or if the power goes out, I should expect SD corruption?

How do most other set top embedded devices in the arm world (appletv, roku, bluray players, tvs, etc) handle this problem?
Posts: 9
Joined: Sun Jan 13, 2013 5:03 am
by malakai » Tue Jan 15, 2013 4:42 am
Not sure on the other devices. Probably they have their system hard coded or don't allow the main boot to change.

The Pi will only corrupt if you lose power while it it writing to the SD Card. This is not something it does constantly or a lot actually. You do have to get used to apps that become non responsive usually they do respond they just take some time. I think most users get used to this and become familiar with how different software behaves.
http://www.raspians.com - always looking for content feel free to ask to have it posted. Or sign up and message me to become a contributor to the site. Raspians is not affiliated with the Raspberry Pi Foundation. (RPi's + You = Raspians)
User avatar
Posts: 1383
Joined: Sat Sep 15, 2012 10:35 am
by garyj » Tue Jan 15, 2013 5:00 am
So, actually, it looks like I just get random corruption no matter what.

I just did a re-image, booted successfully, then before it had a chance to freeze, did a system halt.

On reboot, I "No filesystem could mount root, tried: ext4" and "Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(179,2)"

It seems I get this kind of thing no matter what I do.

Maybe my raspberry pi is defective?
Posts: 9
Joined: Sun Jan 13, 2013 5:03 am
by malakai » Tue Jan 15, 2013 5:06 am
What SD Card are you using and what power supply either one of those could be suspect. I would trouble shoot those two first to see if either has a known issue. http://elinux.org/RPi_VerifiedPeripherals

I wouldn't think the Pi is bad unless you have a good power supply and can verify it isn't dropping voltage.

http://elinux.org/R-Pi_Troubleshooting#Troubleshooting_power_problems
http://www.raspians.com - always looking for content feel free to ask to have it posted. Or sign up and message me to become a contributor to the site. Raspians is not affiliated with the Raspberry Pi Foundation. (RPi's + You = Raspians)
User avatar
Posts: 1383
Joined: Sat Sep 15, 2012 10:35 am
by garyj » Tue Jan 15, 2013 5:42 am
Excellent advice. I measure 4.8-4.9 volts fairly consistently, but dipping down as low as 4.74 briefly during boot and then fairly commonly while running xbmc (though it's usually around 4.8). That's barely out of range, but at least gives me something to try replacing.

Thanks for the help!
Posts: 9
Joined: Sun Jan 13, 2013 5:03 am
by garyj » Tue Jan 15, 2013 6:27 am
for reference, I was using a Freevo FW7713. It's supposed to be a really good power supply. 5v, 1000mA.

Trying a Samsung 5V 1A now....
Posts: 9
Joined: Sun Jan 13, 2013 5:03 am
by gsh » Tue Jan 15, 2013 6:49 am
I don't think it is likely to be the power, but then again never say never!

More likely to be an issue with the SDCard...

To be a bit clearer about the corruption issue, any system that writes to a hard drive or SDCard and gets powered off whilst doing it can corrupt the storage system, in the old days (BBC micro for example) we specifically had to remember to remove the storage media from the drive before powering down (although this was because the drop in power would mean the hard drive heads would clear the bit of information just under the heads at the time! For hard drives this stopped becoming a problem for most people around 1990s when the hard drives had a big enough capacitor to allow them to detect the brown out and move the heads into the parked position!)

Anyway I digress! After that issue we still have the File Allocation Table to worry about (in the DOS formatted first partition anyway) If you are writing to the disk and someone powers it off you can leave the disk in a state where there was valid and current information in the memory (which obviously is never written to the disk!) This can be the allocation table (its a linked list on the disk which tells the operating system where the blocks of the data are stored since they don't have to be contiguous on disk) which can mean the whole disk can no longer make sense!

With the advent of journalling filesystems this problem mostly went away, (ext2, ext4,. jffs2 etc.) I don't actually know how they work, but am under the impression they keep a journal of the work they are about to do before doing it so the file allocation table can always be re-created correctly no matter when the power was pulled...

So yes corruption will always happen unless you have a battery backup to make sure you sync the file cache before powering off, but the corruption shouldn't be final (i.e. Linux should be able to recover) on the ext4 partition (type mount on the command line to find out the type of the partitions)

So we are still unsure what the type of corruption is and why the journalling isn't fixing the problem!

Gordon
--
Gordon Hollingworth PhD
Raspberry Pi - Director of Software Engineering
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 788
Joined: Sat Sep 10, 2011 11:43 am
by gsh » Tue Jan 15, 2013 6:57 am
Also at what step of the system boot does it lock up? I'm wondering whether an image of your flash when it is broken is useful... Are you definitely not writing to the /boot partition (this uses a standard windows FAT and will break badly when you power it off)

Can you post the fsck errors ? (Or just post a picture!)

Gordon
--
Gordon Hollingworth PhD
Raspberry Pi - Director of Software Engineering
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 788
Joined: Sat Sep 10, 2011 11:43 am
by garyj » Tue Jan 15, 2013 7:13 am
After booting with a new power supply and new usb power cable, *and* leaving an optical mouse disconnected, I get sustained power levels of 4.88-4.99V. But I still get freezes randomly, and reboots that result in kernel panics.

I waited a good 15m for it to unfreeze before power-cycling. Upon reboot, I get one of two things: a severely corrupted fs, with fsck spewing out lots of double or orphaned inodes, eventually dropping me to a prompt to do an unautomated fsck (which is usually fruitless), or more, commonly, it logs finding the mmc0 SDHC card, some lines about usb devices and hub, then

EXT4-fs error (device mmcblk0p2): ext4_map_blocks:577: inode #8: block 10658: commm swapper: 1block 334 mapped to illegal pblock (length 1)
jbd2_journal_bmap: journal blcok not found at offset 334 on mmcblk0p2-8
JBD2: bad block at offset 334
JBD2: IO errot -5 recovering block 334 in log

repeated for blocks 335, 336, 337, and 338

then some more lines about usb and ehternet, keyboard, etc., then

JBD2: recovery failed
EXT4-fs (mmcblk0p2): error loading journal
List of all partitions:
[some lines here about partitions]
No filesystem could mount root, tried: ext4
Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(179-2)


First time I boot these images, they work fine. It's only after either freezing or powering down properly that they go corrupt.

One card is a SanDisk 4GB class 4 microSDHC in a SD adapter.

The other is a PNY 8GB class 10 "Professional" rated at 20 MB/s.

Both exhibit the same behavior.

Should I try buying a third?
Posts: 9
Joined: Sun Jan 13, 2013 5:03 am
by matthijsfh » Tue Jan 15, 2013 8:49 am
I had the same issue with one "no-name" SD card. My current "branded" card works fine.

But it remains an issue that is ofter impossible to properly shutdown the PI when it is frozen.

grtz mathtijs
Posts: 9
Joined: Mon Jan 14, 2013 7:51 am
by gsh » Tue Jan 15, 2013 9:32 am
How often does it happen? Is it everytime without fail?

What are you doing to make it lock up? (It should never lock up, my latest RaspBMC box has stayed up for three days!)

Thanks

Gordon
--
Gordon Hollingworth PhD
Raspberry Pi - Director of Software Engineering
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 788
Joined: Sat Sep 10, 2011 11:43 am
by obcd » Tue Jan 15, 2013 11:58 am
Maybe it's a chicken egg issue? Isn't it possible that the system freezes in the first place because the rootfs on the sd card became corrupt?
If possible, I would test it out with a different brand and type of sd card and see if the same problems still exists with it.
Posts: 890
Joined: Sun Jul 29, 2012 9:06 pm
by garyj » Tue Jan 15, 2013 3:17 pm
How often? I think I've only managed to keep it running for about 15 or 20m.

I bought the second card because I suspected this might be the issue, and I spent more on it on purpose. The matrix of sd cards that work / don't work isn't super helpful for the reasons outlined there -- super hard to find any of those exact models on the shelf, and I think both of mine are listed as working anyway...

So, I'm almost at the point of spending more on sd cards than I did on the raspberry pi already. Any tips on finding one that will work for sure?

Or is it more likely this is something else, and it won't matter which sd card I put in?
Posts: 9
Joined: Sun Jan 13, 2013 5:03 am
by garyj » Tue Jan 15, 2013 3:19 pm
oh, and what am I doing to make it lock up? Seems pretty random. I've had it lock when doing nothing at all but just leave it sitting there.

Most recently, I'm trying to run xbmc via xbian, but I had this with raspbian and raspbmc images too.
Posts: 9
Joined: Sun Jan 13, 2013 5:03 am
by Dilligaf » Tue Jan 15, 2013 4:51 pm
Try having a fan blow air over the PI while it's running, see if it runs for a longer time, the PI probably has a bad solder joint that is opening when it heats up. If this makes it run longer then return it for a warranty exchange. You could also try placing it on an ice pack to cool it.

See this thread viewtopic.php?f=28&t=20657&p=260021&hilit=powered+for+long#p260021
Posts: 283
Joined: Wed May 23, 2012 6:48 pm
by garyj » Wed Jan 16, 2013 5:55 am
I put it on top of an ice pack (with a cloth between), but after doing a shutdown and then a reboot after 10-15m, fs was corrupt again.
Posts: 9
Joined: Sun Jan 13, 2013 5:03 am
by Dilligaf » Wed Jan 16, 2013 7:18 am
Closely inspect the sd card contacts in the sd slot, use a magnifying glass/microscope if you have one, make sure there is no solder debris in the "pockets"
Posts: 283
Joined: Wed May 23, 2012 6:48 pm
by obcd » Wed Jan 16, 2013 10:02 am
I would request an RMA and return the board under warranty if I was you.
You have tried 2 different supplies and 2 different SD cards.
TP1 -TP2 voltage seems to be within allowable range.
Some people seem to have fixed such issues by rebaking their Pi in an Oven, but this procedure has a high risk to go wrong and will void the warranty.
Posts: 890
Joined: Sun Jul 29, 2012 9:06 pm
by csabka » Sun Mar 03, 2013 5:15 pm
I had a same problem... I bring the SD to my Ubuntu laptop and I can see that the /dev/mmcblk0p2 (Linux Ext4) partition was corrupted.
Code: Select all
fsck.ext4 /dev/mmcblk0p2

solved my problem.

My Raspbmc has been frozen during pressing arrow buttons for feeding virtual keyboard with Sony Bravia TV's remote controller (libcec).
First it does not react too fast, then I pressed more times... then screen goes black. I tried to Ctrl+Alt+Delete on each console Ctrl+Alt+F[1-7]...

I think I should lock my SD card to be read only... but will Raspbmc tolerate it.... will it work?

Regards:
Csabka
Posts: 1
Joined: Sun Mar 03, 2013 5:03 pm
by Gomoto » Mon Mar 04, 2013 12:41 am
I removed swap (I never like swap on any of my systems, I prefer out of memory :-) )

Never had sd corruption since removing swap. Trashed 2 sd file systems with swap. (Not saying swap was the cause)
Raspberry Pi Model B 512 MB @ 900/250/450 Mhz, 64 MB GPU, Raspian 09.02.13, HDMI 1980x1050
Posts: 96
Joined: Tue Feb 12, 2013 1:21 am
by Gomoto » Mon Mar 04, 2013 12:49 am
Two tips on freezing:
Use Ctrl+Alt+Backspace to shut down the x server (If it is a desktop app).
Use ssh to issue shutdown command from another computer if you are testing an app that is constantly freezing or use shutdown -h 10 before you start the probably freezing app to power down in 10 minutes automatically.
Last edited by Gomoto on Mon Mar 04, 2013 6:17 pm, edited 2 times in total.
Raspberry Pi Model B 512 MB @ 900/250/450 Mhz, 64 MB GPU, Raspian 09.02.13, HDMI 1980x1050
Posts: 96
Joined: Tue Feb 12, 2013 1:21 am
by obcd » Mon Mar 04, 2013 8:07 am
Making your sd card read only (setting the slider in the wp position) won't work as that feature isn't implemented on the Pi.
You can make the linux rootfs mounting read only. Some stuff like dhcp and smb don't like this as they need to update some configuration files. You can create a tmpfs in ram and mount the needed folders and files on that location. Also move the log file folders and log files over there, as they also write to the sd card. It has a nasty drawback that the loggings are gone after a reboot. You can also mount an usb stick for the logfiles if that gives you issues.
Posts: 890
Joined: Sun Jul 29, 2012 9:06 pm