zoi
Posts: 7
Joined: Sun Sep 02, 2012 2:37 pm

Debian: Power down without proper shutdown

Sun Sep 02, 2012 2:44 pm

I've been playing with Raspberry and trying to get the debian based releases work after un-proper shutdown (instant power off). However after maybe 1 to 10 this kind of shutdowns the filesystem on SD corrupts.

I don't think this is un-usual scenario. For example if you want to use Raspberry Pi as part of some mobile device, there is usually no way to make proper shutdown since you don't have a good terminal to it. I was thinking using it for a syntherizator project and maybe something that flies.

I read that one could make the filesystem "read-only" (which would be OK, I don't need to change the SD card data after installation) it would maybe work? However it was so geekish stuff that I don't know how to do it.

I think this is an issue with Raspberry uses. If anyone is doing a release of the image that would survive power outs it would help many users and projects.

obcd
Posts: 917
Joined: Sun Jul 29, 2012 9:06 pm

Re: Debian: Power down without proper shutdown

Sun Sep 02, 2012 4:51 pm

How exactly did you test this?
I wrote a script that triggered the watchdog every 3 minutes and made the pi reboot like that. So, due to the fact it was the watchdog rebooting, it could happen any moment. It ran for a couple of days like that, and didn't corrupt the sd card. Only issue I had was that sometimes the wifi stick didn't enumerate properly. Power cycling the system solved this issue. I also read from a member that he had issues with sd card corruption. Using another sd card solved those.

Wendo
Posts: 142
Joined: Sun Jun 10, 2012 8:27 pm

Re: Debian: Power down without proper shutdown

Sun Sep 02, 2012 7:40 pm

The answer is basically, "You're doing it wrong"

The Raspberrypi is a running a full OS. This comes with all the restrictions of that, including requiring clean shutdowns. The same thing would happen on Windows or MacOS or anything else if you just pulled the power. Even cell phones have shut down sequences now (although they also generally run on a read only OS unless you switch to diagnostic mode

You may be able to mitigate this by using a read only OS, although you'd still need to make some areas available to writing /tmp definitely, /var likely since that's where all the log files go, however if you don't need logging you may be able make it read only.

Your best bet will probably be to have a small program running the monitors a GPIO pin for a button press and shuts down the OS when that's detected

User avatar
Dweeber
Posts: 606
Joined: Fri Aug 17, 2012 3:35 am
Location: Mesa, AZ
Contact: Website

Re: Debian: Power down without proper shutdown

Sun Sep 02, 2012 8:13 pm

Wendo wrote:Your best bet will probably be to have a small program running the monitors a GPIO pin for a button press and shuts down the OS when that's detected
That is actually a cool idea. Been thinking of ways to notify the RPi that I wanted it to shutdown when I have lost access or have no access to the device.

Would be a nice simple project I think to document. Getting a momentary switch to close the connection and adding it to a case would be simple enough. Writing the software to watch for the event should be trivial as well. Unless someone has already done it.
Dweeber A.K.A. Kevin...
My RPI Info Pages including Current Setup - http://rpi.tnet.com

obcd
Posts: 917
Joined: Sun Jul 29, 2012 9:06 pm

Re: Debian: Power down without proper shutdown

Sun Sep 02, 2012 8:25 pm

The question makes sense. If you intend to use the Pi as an embedded system, it should be prepared for an instant power failure. While the idea of a dedicated power button is great, it still requires an auxilary power source to power the system till it's shutdown is completed. (Considering you monitor the main power) I know routers that run linux, and I don't think they have a power down sequence when you pull the power plug. In embedded setups, properly shutting down isn't really an option.

zoi
Posts: 7
Joined: Sun Sep 02, 2012 2:37 pm

Re: Debian: Power down without proper shutdown

Mon Sep 03, 2012 4:58 am

Thanks for all the replies!

I think all of you have a point here. However the best solutions would be making the base filesystem read-only so that it can boot every time. If some "temp" file system crashes, it could be just re-created.

Using a button is not really an option in my case, but a nice idea.

Anyone have a good solution to make the root system read-only?

TarjeiB
Posts: 157
Joined: Thu Jul 12, 2012 3:33 pm

Re: Debian: Power down without proper shutdown

Mon Sep 03, 2012 5:16 am

I've already ordered a light sensor switch which will turn off a 3 v 3 pin 5 minutes before power cut for this exact purpose.
Someone on the forum has already posted a script to shutdown the pi on gpio low, you should find it early with search :)

As for read-only, maybe if you put the logging and similar folders in /var to tmpfs and just "ro" on the root filesystem in fstab? May be missing something though.

Wendo
Posts: 142
Joined: Sun Jun 10, 2012 8:27 pm

Re: Debian: Power down without proper shutdown

Mon Sep 03, 2012 7:46 am

Yup, if it's really an embedded system with no user interaction a read only filesystem is the only option (that's what other embedded linux setups do), however it's a non-trivial job to get a working linux system with a read only file system.

You also want the smallest system possible so remove as many programs and running daemons as possible, thi salso lowers the number of things that may want to write to the filesystem

User avatar
Burngate
Posts: 5971
Joined: Thu Sep 29, 2011 4:34 pm
Location: Berkshire UK Tralfamadore
Contact: Website

Re: Debian: Power down without proper shutdown

Mon Sep 03, 2012 10:58 am

If you're willing to move out of your comfort zone, RISC OS is quite capable of surviving an unexpected power-down

TarjeiB
Posts: 157
Joined: Thu Jul 12, 2012 3:33 pm

Re: Debian: Power down without proper shutdown

Mon Sep 03, 2012 2:08 pm

Linux OS is also very capable of surviving a power cut, but the Raspberry (arm? SDcard?) can't very well - I can crash my regular PC with linux as often as I wish without a recovery problem.
If I do the same on my ARM OpenWRT USB stick router, it will destroy the whole partition table 50% of the time. If I do the same on my Raspberry, it will corrupt files of the filesystem 25-50% of the time... So there must be something to fix there too :)

zoi
Posts: 7
Joined: Sun Sep 02, 2012 2:37 pm

Re: Debian: Power down without proper shutdown

Mon Sep 03, 2012 6:14 pm

TarjeiB wrote:Linux OS is also very capable of surviving a power cut, but the Raspberry (arm? SDcard?) can't very well - I can crash my regular PC with linux as often as I wish without a recovery problem.
If I do the same on my ARM OpenWRT USB stick router, it will destroy the whole partition table 50% of the time. If I do the same on my Raspberry, it will corrupt files of the filesystem 25-50% of the time... So there must be something to fix there too :)
Indeed. I think my "extreme usage" problem is not so extreme at all. I can guarantee that everyone faces sometimes a power-out or must just turn off power when the OS has freezed. It would be really nasty to loose all data & installed programs & settings. I as well have taken power off on hard disk OS machines without worries and never faced this kind of unreliability.

I also know that SD saves data in "chunks" (512 bytes?) so that definely can cause problems when the power goes out while unwritten chunk or so.

I also don't believe that good solution would be "try some other SD card". If that is so, I think there should be a suggested card that everyone should buy to avoid these problems. If this problem is avoidable in this file system & SD card combination.

I've decided to put RPi aside and wait for better days, I'm sure that this is a problem that should be fixed somehow. I cannot do it, but I truly hope someone from the community can fix it. Or tell how to avoid the problem so that I can rely that my synth's OS goes up just before gig..

User avatar
Dweeber
Posts: 606
Joined: Fri Aug 17, 2012 3:35 am
Location: Mesa, AZ
Contact: Website

Re: Debian: Power down without proper shutdown

Mon Sep 03, 2012 6:19 pm

zoi wrote:I've decided to put RPi aside and wait for better days, I'm sure that this is a problem that should be fixed somehow. I cannot do it, but I truly hope someone from the community can fix it. Or tell how to avoid the problem so that I can rely that my synth's OS goes up just before gig..
I've never actually had this happen yet. I rarely am in the position where my RPI has frozen or I don't have some sort of access to it.

However, doing a backup of your SDcard would seem to be a wise thing to do if you are worried about losing data you don't have saved. Shouldn't stop your fun with the device then...
Dweeber A.K.A. Kevin...
My RPI Info Pages including Current Setup - http://rpi.tnet.com

Serac
Posts: 124
Joined: Wed Jul 18, 2012 2:49 pm

Re: Debian: Power down without proper shutdown

Tue Sep 04, 2012 11:52 am

zoi wrote:I read that one could make the filesystem "read-only" (which would be OK, I don't need to change the SD card data after installation) it would maybe work? However it was so geekish stuff that I don't know how to do it.
Making the file system read-only is indeed a practical and viable solution - This is the only option when running a Live CD... On a flash media drive (SD/CF/MMC), one of the simplest descriptions I've come across is http://kristof.vanhertum.be/?p=3 - For anything not requiring X11, it works a treat. The basic concept can be extended to use a tmpfs for the home directory, although that would require an init script to populate it with a few key files. Another way that is used with Live CDs is to use an overlay (or stacking) file system such as unionFS.

Looks like there is a 3.x series kernel patch for unionfs - See http://unionfs.filesystems.org/ for further details.[ Note:] Not suitable for beginners with no experience of mild kernel hacking.

bredman
Posts: 1415
Joined: Tue Jan 17, 2012 2:38 pm

Re: Debian: Power down without proper shutdown

Tue Sep 04, 2012 12:05 pm

zoi wrote:Or tell how to avoid the problem so that I can rely that my synth's OS goes up just before gig..
If you really need your RPi to start up, make sure you have multiple SD cards with the same image. If one SD card does not work, just pop in another SD card.

If there's something really important on your SD card, you should be making a backup of the card regularly anyway.

zoi
Posts: 7
Joined: Sun Sep 02, 2012 2:37 pm

Re: Debian: Power down without proper shutdown

Wed Nov 14, 2012 6:18 pm

Picked my RPi from under the dust and tried this again with another SD card and newest debian distro + rpi_update and the problem persist still. I guess I got filesystem stuck after 10-20 power cut-downs (no proper shutdown for filesystem).

Has anyone any quick + easy ideas for this yet? Or is the only change to make some part of the filesystem read-only?

I'm still amazed that this is not known as "widespread" issue. I'm sure everyone using RPi faces this problem sometime. I guess unplanned power-cut-out is just reality.

User avatar
pluggy
Posts: 3635
Joined: Thu May 31, 2012 3:52 pm
Location: Barnoldswick, Lancashire,UK
Contact: Website

Re: Debian: Power down without proper shutdown

Wed Nov 14, 2012 6:40 pm

My "totally not a Raspberry Pi, whats an SD card ?" hacked Debian NAS box occasionally minces its file system with an unscheduled power cut. About 1 in 10 times if its any consolation....
Don't judge Linux by the Pi.......
I must not tread on too many sacred cows......

zoi
Posts: 7
Joined: Sun Sep 02, 2012 2:37 pm

Re: Debian: Power down without proper shutdown

Wed Nov 14, 2012 7:56 pm

pluggy wrote:My "totally not a Raspberry Pi, whats an SD card ?" hacked Debian NAS box occasionally minces its file system with an unscheduled power cut. About 1 in 10 times if its any consolation....
Uhm? I have never seen a file system crash on regular HDD because of power failure. And I have used HDD's since 90's. So there must be something else wrong with your NAS box I think.

obcd
Posts: 917
Joined: Sun Jul 29, 2012 9:06 pm

Re: Debian: Power down without proper shutdown

Wed Nov 14, 2012 10:21 pm

Some say it's due to the nature of sd cards. They have some housekeeping running in hardware to ensure that unused sectors are combined into a page that can be erased later on. They also have to try to give each sector an equal number of writes. Every brand of sd cards probably handles this stuff differently, and it happens without interference of the host the sd is connected to.
It's an easy explanation, as it can't be proved. It's there, but is it responsable for the file system corruption?
A normal embedded linux device has a separate memory device for it's kernel and root fs that is read only. You have to follow a procedure to make that memory device write enable, which is only done for firmware upgrades. It's logical this decreases the possibillities of corruption.
Before you can mount the root fs read only, you have to be sure the os doesn't write to it.
Under normal os operating conditions, besides the log files, not much should be written to the root fs.
So, basically, creating a ramdisk and mounting /var to that should bring you a whole end into making the root fs read only. If the os doesn't write to it, I don't think mounting it "read only" makes much difference.
There is an overclocking way of the pi that is almost a guarantee for sd card corruption, even when the Pi is properly shutdown. Stay away from that.
You can place the root fs on an usb stick and even on a nfs network drive. I have no idea if this helps against corruption.
Usb sticks have similar housekeeping issues as sd cards. nfs isn't having such issues.

zoi
Posts: 7
Joined: Sun Sep 02, 2012 2:37 pm

Re: Debian: Power down without proper shutdown

Thu Nov 15, 2012 10:12 am

Nice insight obcd!

Few googling on this and I found pretty good articles. The articles are for SSD disks, but I think it doesn't matter..

http://tombuntu.com/index.php/2008/09/0 ... te-drives/

http://superuser.com/questions/248078/c ... an-sd-card

http://danweinreb.org/blog/using-solid- ... s-on-linux

Maybe just the tempfs will do it. Maybe I just acknowledge that SD cards are not 100% trustworthy in any case. And stop messing with my AC-clock-timer :)

greenleaf108
Posts: 9
Joined: Sun Jul 22, 2012 3:53 am

Re: Debian: Power down without proper shutdown

Mon Nov 19, 2012 1:10 am

There's definitely something that sometimes goes wrong with the RasPi is powered down abruptly without a proper shutdown. It's something worth thinking about, as many people will want to use these in portable or headless devices that won't have normal console output for sending a shutdown command.

I'm considering using one of the GPIO pins for this purpose, but it would be better if we could build a distro that had the important bits on a read-only partition, then we could use something a bit more robust for storing data...

Return to “Troubleshooting”