TJSmith1985
Posts: 1
Joined: Thu Jul 26, 2018 10:38 am

HELP! power outages wiping SD cards

Thu Jul 26, 2018 11:07 am

Hi,

We are currently running a project where we are rolling out a the pi3's in high volume.

When testing the units for power cuts and outages, the power outage is removing the image from the SD card meaning we have to re-image. I have spoken to the reseller and emailed raspberry and both answers were to post on here and see if someone else had this issue!

Look forward to your help

pcmanbob
Posts: 6710
Joined: Fri May 31, 2013 9:28 pm
Location: Mansfield UK

Re: HELP! power outages wiping SD cards

Thu Jul 26, 2018 4:51 pm

Hi

It's hard to believe that a power outage could wipe an SD card, I suspect what you are actually seeing is SD card corruption which will prevent the pi from booting on power restore.

It's a problem some people suffer more than others, normally programs that write to the SD card more frequently have a greater chance of actually being in the process of writing to the SD card when the power fails.

It's only the same as if your desktop pc were writing to the hard drive and the power fails which would result in hard drive corruption.
We want information… information… information........................no information no help
The use of crystal balls & mind reading are not supported

User avatar
The Traveler
Posts: 360
Joined: Sat Oct 21, 2017 3:48 pm

Re: HELP! power outages wiping SD cards

Thu Jul 26, 2018 4:58 pm

When testing the units for power cuts and outages, the power outage is removing the image from the SD card meaning we have to re-image.
Um, no. It doesn't remove the image. As mentioned, it may corrupt it. If you want "fail proof" corruption protection then you'll need a ROM with self-boot code added to it along with the OS and whatever programming you need. You could then use as SD card for reading and writing transient data if needed. The RPi isn't the device for it as far as I know.
Last edited by The Traveler on Thu Jul 26, 2018 5:01 pm, edited 1 time in total.
Retired IT professional, C programmer and "beardie weirdie".
RPi interests: Developing an Infinite Improbability Drive
“Thinking outside of the box allows you to get rewards outside of your reach.” Matshona Dhliwayo

W. H. Heydt
Posts: 10781
Joined: Fri Mar 09, 2012 7:36 pm
Location: Vallejo, CA (US)

Re: HELP! power outages wiping SD cards

Thu Jul 26, 2018 5:00 pm

There are several solutions to this. One is: protect the power with a UPS. Either a commercial stand-alone unit, or a Pi mounted "embedded" one. The former will, if the out is long enough, require a manual shut down. The later could be set up to shut down automatically before the batteries get too low. A probably cheaper solution is to research one of the methods of having a read-only SD card. How practical this is will depend on what you need to do with accumulated data. To get a more detailed description of what you need, you'll have to provide more detail of what your project is actually doing.

User avatar
The Traveler
Posts: 360
Joined: Sat Oct 21, 2017 3:48 pm

Re: HELP! power outages wiping SD cards

Thu Jul 26, 2018 5:11 pm

A probably cheaper solution is to research one of the methods of having a read-only SD card.
The problem with this is that you have no where to write the logs, etc. I think, although I haven't tried it, one can create a Read only SD card but have to direct any write operations to a USB device. My understanding it that it's not possible to mount the USB early enough in the boot process to accept the log data.

I'm not knocking the idea, perhaps a path for further discussion in another thread? It seems one critical weakness in the RPi is the SD card OS corruption problem and could use some thought.

Cheers.
Retired IT professional, C programmer and "beardie weirdie".
RPi interests: Developing an Infinite Improbability Drive
“Thinking outside of the box allows you to get rewards outside of your reach.” Matshona Dhliwayo

Heater
Posts: 13134
Joined: Tue Jul 17, 2012 3:02 pm

Re: HELP! power outages wiping SD cards

Thu Jul 26, 2018 6:16 pm

TJSmith1985,
...the power outage is removing the image from the SD card meaning we have to re-image....
Do not worry. This is normal.

If your boot media is writeable you can expected it to get f'ed.

As others have mentioned it is likely that the image is not removed but rather corrupted enough that it will not boot. Makes no odds really, the machine no longer works.

That corruption could be just a corruption of the file system(s) on the SD card, which is fixable, or it could be the SD itself doing odd things like making the file system read only. Makes no odds really, the machine no longer works.

The solution is to make the root file system on the SD card read only from the get go. Then the likelihood of anything bad happening to it on a power outage is minimal.

I have done this to a bunch of Pi in remote, expensive to get at, locations using the instructions in this great post: viewtopic.php?f=63&t=161416 After one year and many power outages they are still working fine.

Of course the question then is: Where do I save any data I need to save as the Pi runs?

Good question. Save it to an attached USB stick or better still don't save it on the Pi, send it out over the net to some server's database.

Really, you should have thought about this before coming to roll out a large scale project.

W. H. Heydt
Posts: 10781
Joined: Fri Mar 09, 2012 7:36 pm
Location: Vallejo, CA (US)

Re: HELP! power outages wiping SD cards

Thu Jul 26, 2018 8:17 pm

The Traveler wrote:
Thu Jul 26, 2018 5:11 pm
A probably cheaper solution is to research one of the methods of having a read-only SD card.
The problem with this is that you have no where to write the logs, etc. I think, although I haven't tried it, one can create a Read only SD card but have to direct any write operations to a USB device. My understanding it that it's not possible to mount the USB early enough in the boot process to accept the log data.

I'm not knocking the idea, perhaps a path for further discussion in another thread? It seems one critical weakness in the RPi is the SD card OS corruption problem and could use some thought.

Cheers.
The question becomes: Which is the bigger problem? Not getting the early log information or not being able to boot after a power outage? I susptect that, for most situations, the later is a bigger problem than the former. The next question is...how much money do you want to throw and not having a problem?

User avatar
The Traveler
Posts: 360
Joined: Sat Oct 21, 2017 3:48 pm

Re: HELP! power outages wiping SD cards

Thu Jul 26, 2018 10:32 pm

The next question is...how much money do you want to throw and not having a problem?
If it were me, was wanting to keep up market share, continue to support brand recognition and I had the development cash available, I'd throw enough money at it to solve the problem.

That being said, I still use my RPi in non-critical applications where recovering from a corrupted SD card doesn't hamstring me. But I've got 30+ years of practical experience to teach me this. Seems like a lot of new folks want more from an RPi than it was designed for.

Cheers.
Retired IT professional, C programmer and "beardie weirdie".
RPi interests: Developing an Infinite Improbability Drive
“Thinking outside of the box allows you to get rewards outside of your reach.” Matshona Dhliwayo

User avatar
mahjongg
Forum Moderator
Forum Moderator
Posts: 12131
Joined: Sun Mar 11, 2012 12:19 am
Location: South Holland, The Netherlands

Re: HELP! power outages wiping SD cards

Thu Jul 26, 2018 11:03 pm

The problem of sd-cards that can corrupt themselves when turning power to them off suddenly isn't solvable by doing something with the PI.

The problem is that an sd-card contains a micro controller that independently performs tasks that cannot suddenly be aborted by failing to provide power to the card without risking storage corruption.

The micro controller independently from the main system shuffles data around, and interrupting such shuffling (by pulling the power) leads to the micro controller losing track of where some data is actually stored.

In fact because any filing system that uses the sd-card will write to sections containing such things as directories and file allocation table, the most often, the micro controller most likely shuffles just that kind of data around, so losing that data is most likely to be fatal for the filing system.

It is all because sd-cards use flash memory that can only endure limited amounts of write cycles, so the micro controller is constantly busy copying flash sectors that have been written to too many times to sectors that are "fresh".

W. H. Heydt
Posts: 10781
Joined: Fri Mar 09, 2012 7:36 pm
Location: Vallejo, CA (US)

Re: HELP! power outages wiping SD cards

Thu Jul 26, 2018 11:38 pm

What mahjongg said if you look at it from the RPF spending money.

What I had meant (and, obviously, I was unclear, for which, my apologies) is the question of how much the *user* of the Pi is willing to spend to avoid SD card corruption.

Changing the Pi to have an energy store sufficient to provide power to the SD card long enough for it to finish any writing or wear leveling would mean that it was no longer within shouting distance of a $35 SBC.

It can, after all, be done to a considerable extent by doing as much as possible to prevent sudden power drops. One of the regular posters here--mosespi--makes UPS units that attach to Pis. I have one of his second gen units on my desk waiting until I get a set of batteries for it. He has posted results from using supercaps to provide enough energy to do a clean shut-down, including time for the SD card to finish anything it is doing.

The ultimate solution is probably to wait until someone develops a non-volatile storage device that is small enough to use in microSD cards and can handle enough write cycles to minimize (or eliminate) the need to do wear leveling. The Intel Xpoint has the *potential* to do that, if (a) it lives up to its hype, and (b) somebody decides that it's worth putting into microSD cards.

User avatar
The Traveler
Posts: 360
Joined: Sat Oct 21, 2017 3:48 pm

Re: HELP! power outages wiping SD cards

Thu Jul 26, 2018 11:54 pm

The problem of sd-cards that can corrupt themselves when turning power to them off suddenly isn't solvable by doing something with the PI.
Actually the problem can be mitigated somewhat by using onboard power, such as a lithium battery. 1.8vdc is enough to power an SD card generally speaking. Adding a battery such as a 2032 lithium can keep the SD card active. Yes, I'm familiar with the engineering and production aspects and costs, but it's just a solution for discussion.
Changing the Pi to have an energy store sufficient to provide power to the SD card long enough for it to finish any writing or wear leveling would mean that it was no longer within shouting distance of a $35 SBC.
Posted while I was typing. It's a trade off of course. But when you start looking at the application domains the RPi is being used in, maybe a $35.00 computer isn't realistic anyways for those applications.

Cheers.
Retired IT professional, C programmer and "beardie weirdie".
RPi interests: Developing an Infinite Improbability Drive
“Thinking outside of the box allows you to get rewards outside of your reach.” Matshona Dhliwayo

W. H. Heydt
Posts: 10781
Joined: Fri Mar 09, 2012 7:36 pm
Location: Vallejo, CA (US)

Re: HELP! power outages wiping SD cards

Fri Jul 27, 2018 1:56 am

The Traveler wrote:
Thu Jul 26, 2018 11:54 pm
Posted while I was typing. It's a trade off of course. But when you start looking at the application domains the RPi is being used in, maybe a $35.00 computer isn't realistic anyways for those applications.
That could be the case....but people keep trying to do so. There seems to be something really attaractive about a $35 computer with reliable hardware and good software support. Perhaps those that run into problems (like SD card corruption in a commercial setting) haven't analyzed their actual requirements very well.

Heater
Posts: 13134
Joined: Tue Jul 17, 2012 3:02 pm

Re: HELP! power outages wiping SD cards

Fri Jul 27, 2018 5:13 am

@The Traveler,
The problem with this is that you have no where to write the logs, etc. I think, although I haven't tried it, one can create a Read only SD card but have to direct any write operations to a USB device. My understanding it that it's not possible to mount the USB early enough in the boot process to accept the log data.
This is not a problem.

If you create a read only root, as per the instructions I linked to earlier here, then the system has an overlay file system overlaying the root file system. That overly is a RAM based. The upshot of this is that the system can write its logs and whatever to the root file system as per normal.

Except such file writes are no persisted, they are in RAM and lost when power is removed.

No worries, you can program/script something to copy the logs etc from the RAM overly to any other storage when it is mounted.

The read only root instructions again: viewtopic.php?f=63&t=161416
Actually the problem can be mitigated somewhat by using onboard power, such as a lithium battery. 1.8vdc is enough to power an SD card generally speaking. Adding a battery such as a 2032 lithium can keep the SD card active.
I don't see how this can work. That battery is not going to last long.
But when you start looking at the application domains the RPi is being used in, maybe a $35.00 computer isn't realistic anyways for those applications.
In many situations the $35 dollar Pi is a perfectly good replacement for many similar boards that cost many times more. The components and quality are much the same. For example the IGEP v2 industrial ARM board we were using before the Pi arrived. Cost over 200 Euro.

Our OP, TJSmith1985, hasn't told us what his application is, so it's premature to speculate that the Pi is not suitable.

W. H. Heydt
Posts: 10781
Joined: Fri Mar 09, 2012 7:36 pm
Location: Vallejo, CA (US)

Re: HELP! power outages wiping SD cards

Fri Jul 27, 2018 3:58 pm

Heater wrote:
Fri Jul 27, 2018 5:13 am
Our OP, TJSmith1985, hasn't told us what his application is, so it's premature to speculate that the Pi is not suitable.
Actually...I think a missing critical bit is to find out *where* he is using them that he is getting that many power problems. Is it actually an electric utility issue, or is someone flipping a switch at the end of the day that is cutting power to the Pis? (The answer to the later being, (a) don't do that, and (b) arrange things so that there isn't an easy and accessible switch controlling the power to the Pi.)

User avatar
The Traveler
Posts: 360
Joined: Sat Oct 21, 2017 3:48 pm

Re: HELP! power outages wiping SD cards

Fri Jul 27, 2018 4:01 pm

... so it's premature to speculate that the Pi is not suitable.
No worries. It was a bit of a generic "one size doesn't fit all" statement.
Retired IT professional, C programmer and "beardie weirdie".
RPi interests: Developing an Infinite Improbability Drive
“Thinking outside of the box allows you to get rewards outside of your reach.” Matshona Dhliwayo

crofter
Posts: 55
Joined: Thu Feb 22, 2018 11:28 pm

Re: HELP! power outages wiping SD cards

Fri Jul 27, 2018 4:40 pm

what about using a small ssd

crofter
Posts: 55
Joined: Thu Feb 22, 2018 11:28 pm

Re: HELP! power outages wiping SD cards

Fri Jul 27, 2018 4:43 pm

what about using a pi ups hat as in the power line monitor jan or feb or march magpie

hippy
Posts: 5804
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: HELP! power outages wiping SD cards

Fri Jul 27, 2018 5:47 pm

The Traveler wrote:
Thu Jul 26, 2018 4:58 pm
If you want "fail proof" corruption protection then you'll need a ROM with self-boot code added to it along with the OS and whatever programming you need.
The SoC's do provide for SPI and NAND booting so they may offer an opportunity for a ROM-like, physically read-only, booting process. Booting via SPI or NAND apparently works, but isn't documented AFAIK, and there doesn't seem to be any interest in providing for either -

https://github.com/raspberrypi/firmware/issues/874

PXE booting over ethernet is of course another option for those Pi's which support that.

ejolson
Posts: 3438
Joined: Tue Mar 18, 2014 11:47 am

Re: HELP! power outages wiping SD cards

Fri Jul 27, 2018 9:17 pm

The Traveler wrote:
Thu Jul 26, 2018 4:58 pm
If you want "fail proof" corruption protection then you'll need a ROM with self-boot code added to it along with the OS and whatever programming you need.
You could set the PERM_WRITE_PROTECT bit on the SD card. Then the data will never change on the card and the only way of updating it is to buy a new card. Alternatively, use the TMP_WRITE_PROTECT bit and only turn it off with extreme caution when doing updates. Note that you don't need the special hardware described in the above link, because it is possible to set the necessary bits using a standard Raspberry Pi.

W. H. Heydt
Posts: 10781
Joined: Fri Mar 09, 2012 7:36 pm
Location: Vallejo, CA (US)

Re: HELP! power outages wiping SD cards

Fri Jul 27, 2018 10:52 pm

crofter wrote:
Fri Jul 27, 2018 4:40 pm
what about using a small ssd
Same problem (it's still flash memory), though it would probably do wear leveling faster and less often.

W. H. Heydt
Posts: 10781
Joined: Fri Mar 09, 2012 7:36 pm
Location: Vallejo, CA (US)

Re: HELP! power outages wiping SD cards

Fri Jul 27, 2018 10:55 pm

hippy wrote:
Fri Jul 27, 2018 5:47 pm
The Traveler wrote:
Thu Jul 26, 2018 4:58 pm
If you want "fail proof" corruption protection then you'll need a ROM with self-boot code added to it along with the OS and whatever programming you need.
The SoC's do provide for SPI and NAND booting so they may offer an opportunity for a ROM-like, physically read-only, booting process. Booting via SPI or NAND apparently works, but isn't documented AFAIK, and there doesn't seem to be any interest in providing for either -
You mean like the CM and CM3? It's still flash memory, though, and that's the fundamental problem here.

ejolson
Posts: 3438
Joined: Tue Mar 18, 2014 11:47 am

Re: HELP! power outages wiping SD cards

Fri Jul 27, 2018 11:02 pm

W. H. Heydt wrote:
Fri Jul 27, 2018 10:55 pm
hippy wrote:
Fri Jul 27, 2018 5:47 pm
The Traveler wrote:
Thu Jul 26, 2018 4:58 pm
If you want "fail proof" corruption protection then you'll need a ROM with self-boot code added to it along with the OS and whatever programming you need.
The SoC's do provide for SPI and NAND booting so they may offer an opportunity for a ROM-like, physically read-only, booting process. Booting via SPI or NAND apparently works, but isn't documented AFAIK, and there doesn't seem to be any interest in providing for either -
You mean like the CM and CM3? It's still flash memory, though, and that's the fundamental problem here.
Flash memory is quite reliable if it is only written to once and never modified.

hippy
Posts: 5804
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: HELP! power outages wiping SD cards

Sat Jul 28, 2018 10:03 am

ejolson wrote:
Fri Jul 27, 2018 11:02 pm
Flash memory is quite reliable if it is only written to once and never modified.
And whatever is used doesn't actually have to be Flash. It often is because writes are catered for but it doesn't have to be Flash nor have wear-levelling and everything which compounds the problems.

The risk with SD cards and the like is that any glitches during powering-up or down could cause it to do something unintended which leads to adverse consequences. That risk appears greater and more likely when powering down and particularly with sudden power removal.

Other products using SD Card storage don't access that so regularly, rarely write, more gracefully shut down as power fails, and power is almost never abruptly removed. There is therefore less likelihood of corruption with those than when using a Pi.

Return to “General discussion”