KeithSloan
Posts: 321
Joined: Tue Dec 27, 2011 9:09 pm

ARRGGH Power cuts.

Mon Apr 04, 2016 4:06 pm

Yet another power cut today ARRRGH!!!!!

Two Pi SD-card trashed.

Giving up on Raspberry Pi's cannot be bothered with the hassle of re imaging everything when we get a power cut.

Will sleep on it but probably a couple of 2B's headed the way of ebay.

stderr
Posts: 2178
Joined: Sat Dec 01, 2012 11:29 pm

Re: ARRGGH Power cuts.

Mon Apr 04, 2016 4:23 pm

KeithSloan wrote:Yet another power cut today ARRRGH!!!!!
Two Pi SD-card trashed.
Giving up on Raspberry Pi's cannot be bothered with the hassle of re imaging everything when we get a power cut.
Will sleep on it but probably a couple of 2B's headed the way of ebay.
You'll have this problem with any board that is running off sdcards or flash if you are writing to it when you kill the power. So selling the pi and buying the Beaglebone Blacks isn't probably going to change anything.

I don't know why you are having so many power outages, I'd be more worried about that than the pi problems. If you are in Baghdad or something like that, I'd be looking into some sort of a battery back up system so electronics in general weren't in such obvious danger.

But one thing you could do is use a boot to memory distribution. One example is Tiny Core. Since you can boot to that and remove the sdcard, you won't have any problems with the card being trashed unless you mount it and, generally speaking here, try writing to it when the power problem occurs.

No one knows, at least no one who is talking knows, how the wear levelling in the cards you own works but it is likely that it is at the drive level. So keep the operating system and the data on different physical devices. Have back up sdcards and flash drives of your os and your data. Or you could just sell the things on ebay but then you wouldn't have any pi.

W. H. Heydt
Posts: 10585
Joined: Fri Mar 09, 2012 7:36 pm
Location: Vallejo, CA (US)

Re: ARRGGH Power cuts.

Mon Apr 04, 2016 4:29 pm

UPS. Even a very small one. I've got a couple of 250VA units that are only there to protect fully set up Pis. Pretty much every piece of electronics in the house...Pis, PCs, modem, router, switches, etc. is behind a UPS. As a result, I don't lose anything due to power outages. What's more, a UPS alerted me to the opposite problem one night. The incoming voltage was 140v. Needless to say, I notified the utility and they sent someone out to fix it.

jahboater
Posts: 4595
Joined: Wed Feb 04, 2015 6:38 pm

Re: ARRGGH Power cuts.

Mon Apr 04, 2016 4:36 pm

We get about one every two years or so, but I have a remote controlled power strip and my cat trod on it during the night - powering off all three Pi's.
No SD card problems with any of the Pi's!!
Last edited by jahboater on Mon Apr 04, 2016 7:16 pm, edited 1 time in total.

mutley
Posts: 61
Joined: Sat Jan 02, 2016 8:06 pm

Re: ARRGGH Power cuts.

Mon Apr 04, 2016 4:47 pm

Use RO root on the CF card, that will solve all your corruption problems. You will probably also need a USB flash drive mounded in RW mode for your data depending on your system. But this way you should not see any OS corruption on a power problem, and if you do see any corruption it should be isolated to a single file(s) that have nothing to do with the OS.

stderr
Posts: 2178
Joined: Sat Dec 01, 2012 11:29 pm

Re: ARRGGH Power cuts.

Mon Apr 04, 2016 5:28 pm

jahboater wrote:We get about one every two years, but I have a remote controlled power strip and my cat trod on it during the night - powering off all three Pi's.
No SD card problems with any of the Pi's!!
The problem generally happens when the system is writing to the sdcard. So if you have, say, a swap partition and you are using it, there will be writes to it regularly, perhaps often enough that most any power loss messes up the card. Wear levelling is likely at the whole physical disk level so writing to any partition might destroy data all over the card. Ideas include:

1. UPS
2. separate swap device and put on flash drive or usb hard drive
3. don't have swap partition or if you do, only use it to avoid crashing processes and not as a way to generally expand available ram. Don't sit in the swap file and thrash, very bad.
4. boot an os that is entirely in ram and doesn't need to write to the sdcard at all.
5. sell pi on ebay.

drushtx
Posts: 36
Joined: Wed Mar 21, 2012 6:16 pm
Location: Houston, Texas
Contact: Website Yahoo Messenger Facebook YouTube

Re: ARRGGH Power cuts.

Mon Apr 04, 2016 5:36 pm

If you go for option 5 (eBay), we have a post up in "Wanted" for 6 - 9 2B units for a classroom. Skip ebay - set your price and contact me. Actually, I hope you stick with it and UPS it or similar but if you go with 5...

Thanks!

User avatar
DougieLawson
Posts: 35537
Joined: Sun Jun 16, 2013 11:19 pm
Location: Basingstoke, UK
Contact: Website Twitter

Re: ARRGGH Power cuts.

Mon Apr 04, 2016 5:50 pm

I've pulled power on many occasions and haven't corrupted an SDCard since the firmware was fixed a couple of years ago. If you're running 4.1.19 with matching firmware things should be stable.

What are your RPis doing? How much data do you collect, how often? Have you thought about writing data to a USB device (so the SDCard becomes read-only)? My "mission critical" system runs with /boot on an 8GB SDCard and the rootfs on a 160GB hard drive in a USB carrier.
Note: Having anything remotely humorous in your signature is completely banned on this forum. Wear a tinfoil hat and you'll get a ban.

Any DMs sent on Twitter will be answered next month.

This is a doctor free zone.

KeithSloan
Posts: 321
Joined: Tue Dec 27, 2011 9:09 pm

Re: ARRGGH Power cuts.

Mon Apr 04, 2016 7:07 pm

DougieLawson wrote:I've pulled power on many occasions and haven't corrupted an SDCard since the firmware was fixed a couple of years ago. If you're running 4.1.19 with matching firmware things should be stable.

What are your RPis doing? How much data do you collect, how often? Have you thought about writing data to a USB device (so the SDCard becomes read-only)? My "mission critical" system runs with /boot on an 8GB SDCard and the rootfs on a 160GB hard drive in a USB carrier.
Have two 2B's both run [email protected] One reads pressure and puts the data to a ram filing system it also is a webserver with very low trafic. The other acts as a print server but I was not printing when the power cut

User avatar
DougieLawson
Posts: 35537
Joined: Sun Jun 16, 2013 11:19 pm
Location: Basingstoke, UK
Contact: Website Twitter

Re: ARRGGH Power cuts.

Mon Apr 04, 2016 7:32 pm

It might be worth investing in more expensive branded SDCards that are likely to have better wear levelling.

Or stick all of your data on an ext4 formatted hard drive connected to one RPi and share that out to the other RPi using NFS. My data collection system writes its data on a 2TB NAS box that's accessed by all my RPis using an NFS share.
Note: Having anything remotely humorous in your signature is completely banned on this forum. Wear a tinfoil hat and you'll get a ban.

Any DMs sent on Twitter will be answered next month.

This is a doctor free zone.

Dutch_Master
Posts: 360
Joined: Sat Jul 27, 2013 11:36 am

Re: ARRGGH Power cuts.

Mon Apr 04, 2016 7:50 pm

There's another alternative: use PXE. In other words, the OS is not on the SD card, but stored as image on a 3rd device. The RPi only needs some propitiatory files in RO mode on the SD card to boot. The PXE environment then loads via the network and runs in RAM. Said 3rd device can be a simple Pentium range PC, serving as NAS and TFTP server (for PXE/net-boot)

A UPS is probably easier, but more expensive. Given the power consumption of a pair of RPi's, you'll get a fair few hours of backup time. And if power isn't restored in time, a smart UPS can send special messages to the attached hardware to cleanly shut down ahead of power loss.

KeithSloan
Posts: 321
Joined: Tue Dec 27, 2011 9:09 pm

Re: ARRGGH Power cuts.

Fri Apr 15, 2016 10:48 am

Okay at the Winchester Pi meeting Dougie managed to fixed the corrupted superblock on the SD card. Any chance he could documented the process as I know he tried various approaches before being successful.

Okay first part was to insert the external SD card reader and run dmesg to find out which device it was i.e. /dev/sd(x)

unmount with command ?

run fsck with command ?

List backup superblocks with command ?

A number of other commands to fix things up.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23071
Joined: Sat Jul 30, 2011 7:41 pm

Re: ARRGGH Power cuts.

Fri Apr 15, 2016 1:08 pm

All a bit odd. I've spent years yacking the power on Pi's and have no problems whatsoever with SD card corruption. I tend to be more careful now, but not that careful! If you are getting a corruption with every power failure, I wonder if it is more to do with power brownouts that a complete power failure - brownouts can do nasty things - I've lost consumer equipment that lived for years with occasional cuts to a single brown out.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

MarkDH102
Posts: 329
Joined: Fri Feb 13, 2015 3:18 pm

Re: ARRGGH Power cuts.

Fri Apr 15, 2016 2:58 pm

I've just posted about a Tecknet device which I'm trying to use as a UPS.
My Pi's run as data loggers and only write the data onto the SD card at midnight (risky, I know, but it's not critical data).
I have loads of power cuts and brownouts (living down a country lane in the middle of nowhere) and as yet, fingers crossed, for over a year I've not had a single SD card corrupted.
I use a mixture of Kingston and Sandisk 32GB cards that cost around the £6 mark.

ejolson
Posts: 3231
Joined: Tue Mar 18, 2014 11:47 am

Re: ARRGGH Power cuts.

Fri Apr 15, 2016 4:18 pm

KeithSloan wrote:Okay at the Winchester Pi meeting Dougie managed to fixed the corrupted superblock on the SD card. Any chance he could documented the process as I know he tried various approaches before being successful.

Okay first part was to insert the external SD card reader and run dmesg to find out which device it was i.e. /dev/sd(x)

unmount with command ?

run fsck with command ?

List backup superblocks with command ?

A number of other commands to fix things up.
I had this happen once as well and used a similar technique to fix it. Strangely enough it's never happened again even though there have been other power losses and I'm still using the same sdcard.

Heater
Posts: 12962
Joined: Tue Jul 17, 2012 3:02 pm

Re: ARRGGH Power cuts.

Fri Apr 15, 2016 4:40 pm

Oddly I had many problems with SD cards and fs corruption when the Pi was launched.

Now a days I yank power willy nilly and nothing fails.

Either way, if I want a reliable system I would arrange for the root fs to be read only.

User avatar
TimG
Posts: 285
Joined: Tue Apr 03, 2012 12:15 am
Location: Switzerland

Re: ARRGGH Power cuts.

Fri Apr 15, 2016 4:42 pm

KeithSloan wrote:Yet another power cut today ARRRGH!!!!!

Two Pi SD-card trashed.
Adafruit make a very dinky load-sharing battery charger, ideally suited to make an inexpensive UPS for a Pi. There's even a low battery signal, perfect for triggering your Pi to shut-down nicely if the power outage lasts too long.

https://www.adafruit.com/products/2465
Image

KeithSloan
Posts: 321
Joined: Tue Dec 27, 2011 9:09 pm

Re: ARRGGH Power cuts.

Fri Apr 15, 2016 6:12 pm

Heater wrote: Either way, if I want a reliable system I would arrange for the root fs to be read only.
Lots of people recommend this. So why is the default Raspbian image not setup in this manner.

Or do you mean I would arrange for the only file system on the SD card to be a read only root fs

edo1
Posts: 136
Joined: Sun Jun 15, 2014 3:33 pm
Location: Russia

Re: ARRGGH Power cuts.

Fri Apr 15, 2016 6:47 pm

KeithSloan wrote:Or do you mean I would arrange for the only file system on the SD card to be a read only root fs
you need totally avoid writing to sd card (and store all changeable data somewhere else)

Heater
Posts: 12962
Joined: Tue Jul 17, 2012 3:02 pm

Re: ARRGGH Power cuts.

Fri Apr 15, 2016 8:46 pm

KeithSloan,
Lots of people recommend this. So why is the default Raspbian image not setup in this manner.
Because the Pi is a tiny PC. Because Raspbian is Debian. Because Debian has been running on our PC's for decades. Because we like to read and write files on our systems and update/upgrade them. You know, like the Windows guys do.

Now, if you want to build something else, like an embedded system that has to run unattended for years in a remote location, then you need to think about things.

stderr
Posts: 2178
Joined: Sat Dec 01, 2012 11:29 pm

Re: ARRGGH Power cuts.

Fri Apr 15, 2016 9:19 pm

KeithSloan wrote:
Heater wrote: Either way, if I want a reliable system I would arrange for the root fs to be read only.
Lots of people recommend this. So why is the default Raspbian image not setup in this manner.
If you'd like to experience this on the pi, you can run the tiny core version of their distribution, called picore. The system just boots to memory so you can actually remove the sdcard. If you know power is going to be there, you could one one sdcard and then pi! Of course when you try to make changes, you'll do it normally and they are lost at the reboot. So you'll learn to do the special updating you do when shutting down if you want things saved, but then that might not work right for some reason. Once things are on the sdcard, that should be fine and once you have some way to make sure you've really updated things, fine too. But it is different to just opening a file in vi and :wq it when you are done changing it.

MarkDH102
Posts: 329
Joined: Fri Feb 13, 2015 3:18 pm

Re: ARRGGH Power cuts.

Sat Jul 16, 2016 2:57 pm

On Amazon Prime day last week I pulled the trigger on a RavPower 16750mAh power pack for £14.99.
At last - what seems to be a useable UPS type solution.
My Model B V1.1 with an Edimax WiFi adapter draws approx. 300mA.
I plug the pack into the mains via a standard 2A USB phone adapter with a standard USB cable.
Then the 2.1A output of the pack into the Pi using the RavPower supplied USB cable
I can remove mains and the Pi stays up. I've left it running for 3 days now. The pack does not even get warm.
I've removed mains from the pack 10 times now for varying periods all with no down time on the Pi.
Impressed.
This should cope with even the longest power cut (up to 6 hours for us in the past).
I will monitor the 5V input to the pack on a digin to figure out when the mains has dropped out so I can at least record it.

stderr
Posts: 2178
Joined: Sat Dec 01, 2012 11:29 pm

Re: ARRGGH Power cuts.

Sat Jul 16, 2016 10:14 pm

MarkDH102 wrote:On Amazon Prime day last week I pulled the trigger on a RavPower 16750mAh power pack for £14.99.
At last - what seems to be a useable UPS type solution.
Definitely keep people informed about your experiences with this for that use.

rln
Posts: 173
Joined: Wed Apr 09, 2014 1:43 pm
Location: Sweden
Contact: Website

Re: ARRGGH Power cuts.

Sat Jul 16, 2016 10:56 pm

Heater wrote:if you want to build something else, like an embedded system that has to run unattended for years in a remote location, then you need to think about things.
For those cases you might want to check out my Nard SDK. It's designed precisely for such scenarios.
http://www.arbetsmyra.dyndns.org/nard/
Author of the robust Nard distro http://www.nard.se

MarkDH102
Posts: 329
Joined: Fri Feb 13, 2015 3:18 pm

Re: ARRGGH Power cuts.

Tue Aug 23, 2016 2:10 pm

I've been using the RavPower mentioned above in situ for 4 weeks now.
I've had a few short power cuts in that time and have pulled the mains adapter out occasionally with no problems.
I'm drawing about 230mA fairly constantly.
When we get a power cut, Western Power toggles the power a few times in a minute (I guess it comes on then cuts due to trip at the substation) before we get a stable power back and I'm surviving that.
One thing I don't know is how much energy the RavPower is able to store after a month in this configuration but when the power cuts, all 4 led's stay lit indicating a fully charged power bank.

Return to “General discussion”