Raspberry Pi power cut torture test using a MoPower UPS

[mosespi]

As some of you remember the prior versions of the Raspberry Pi were a bit sensitive to power cuts, many SD cards were corrupted in the early days of the Pi. What likely made this worse was a lack of good USB power supplies (or the prevalence of really bad ones!). A lot of power supplies don't seem to meet the USB 5v power specification when loaded up. They are fine for charging phones, but not good for powering a Pi. I needed reliable power and battery backup so I proceeded to design and manufacture my own UPS for the Pi. You can see it here: http://www.allspectrum.com/mopower/

Was this a hardware problem? Something in software? Firmware? Maybe a combination? I don't know. I have not seen a good solid answer as of yet. But thinking of this one day, I figured why not give the new Pi and software a good torture test!?! The UPS I built to protect the Pi can just as easily be turned against it!

The UPS controls power to the Pi and normally turns it off when the Pi signals a properly shut down operating system. A few quick software changes and the UPS can cut power immediately without a proper shutdown! With that setup, on April 15th a bit past midnight the power cut torture test began! The UPS was setup with an interrupt timer to power up the Pi every minute, the Pi would boot in about 20 seconds or so and hit an external web server to signal that it successfully started up. Then 15 seconds into a successful startup the Pi would have it's power deliberately cut. The Pi would startup (hopefully!) and have it's power cut each and every minute until something went wrong!

So what happened? Well it has been about 3 weeks since the test began, and it's pretty amazing.. the Pi is STILL BOOTING UP SUCESSFULLY!!! Wow, I didn't not expect this result!

So how many times has this successful boot and powercut occurred? Lets grep the logfile on the external web server.. so as of right now..

# grep -A10000000 "15/Apr/2017:00:31:19" /var/log/httpd/access_log | grep cuttest01 | wc -l
25002
#

Yes, you are reading that right. 25,002 times!!! And the test is still ongoing, one attempt every minute!

So what's next? Well in this test, I did not deliberately write anything to the SD card. This has long been thought of as the cause of the corruption in SD cards, an interrupted write. So the next setup will include some sort of heavy writing to the SD card during the power cut. Maybe a dd to a file somewhere. One of my theories is that the power design of the SD card circuitry changed in the Pi.. possibly with some added capacitors.. to where the SD card had just enough power during a failure to successfully write the last bit of data that came from the SOC.

Did it ever fail to boot? Actually yes, but I have yet to find out why exactly. There were 10 failures to boot, they were spaced very evenly throughout the testing and subsequent boot-ups were successful. A few were unknown with no screen output, but most appeared to be kernel panics of some sort. All in all not bad, a 1 in 2,500 chance of a boot failure is not bad at all. Is the card or OS 100% free of corruption? I do not know, but when I pull it out I can do a checksum test on all the files and compare it to the original. What I do know is that it was good enough to successfully boot up.

Some more details on the setup. The Pi is a model 3 with what I believe is an 8GB unknown manufacture card that came with a noobs bundle on one of my older setups. Installed OS is 2017-04-10-raspbian-jessie-lite with no updates. Connected are an Ethernet cable, HDMI cable/monitor and of course the UPS.

Tell me what you think! Maybe you have some ideas for the next test? And if you are interested in the MoPower UPS, I would appreciate your support of my efforts, you can buy it here:

http://www.allspectrum.com/mopower/

Thanks everyone!

Regards,
-Moses

[klricks]

....... A few quick software changes and the UPS can cut power immediately without a proper shutdown!....

I think that most unintended power losses are not immediate or abrupt. When power is removed from the mains as would happen during a power outage, then the power supply continues to output voltage for some time as the capacitors discharge..... then 'brownout' ..... then it goes off after maybe 2 or 3 seconds. I believe the during the brownout stage is when bad things are most likely to happen......

[paulv]

+1

Depending on what you want to prove, or investigate, you should indeed add writing to the SD card. That is the critical operation, not reading.

There have been some significant improvements made in both the hardware, kernel drivers and SoC firmware over the last couple of years, with the addition of the write operation, your current test could be a testament to that success.

I then suggest you do the test also with a Model (1)B a Model 2B, and a Model 2B+. There are (some significant) differences between these models and the model 3.

Finally, a more realistic SD card failure test (if that is what you want to investigate) is of course to switch the power at the mains end, as klricks suggests, without a UPS, but that adds another variable, the wall-wart.

To somewhat simulate this with your current setup, you could add a large capacitor at the 5V end, so the slower collapse would mean more time spent in the critical voltage region below say 4.7V and you could even bring it back up for a small amount of time to simulate a kind of Brown-Out.

[mosespi]

....... A few quick software changes and the UPS can cut power immediately without a proper shutdown!....

I think that most unintended power losses are not immediate or abrupt. When power is removed from the mains as would happen during a power outage, then the power supply continues to output voltage for some time as the capacitors discharge..... then 'brownout' ..... then it goes off after maybe 2 or 3 seconds. I believe the during the brownout stage is when bad things are most likely to happen......

A bit more detail.. it shuts down the 5v switching regulator and that does have an output capacitor of 150uf, so if we nitpick it doesn't really cut power 'immediately'. I would say it is similar in that respect to a regular USB power adapter, I would take a guess and say most of those would have output capacitors within a reasonable range of a couple of hundred micro farads as well.

I have definitely never seen a regular consumer USB power adapter that would continue to power, after being unplugged, a Pi for any duration that my eye could detect. Indeed nothing close to even a second. To do that you would need a few dozen milli farads at least.. something I don't think they have any reason to do.

Regards,
-Moses

[mosespi]

+1

Depending on what you want to prove, or investigate, you should indeed add writing to the SD card. That is the critical operation, not reading.

There have been some significant improvements made in both the hardware, kernel drivers and SoC firmware over the last couple of years, with the addition of the write operation, your current test could be a testament to that success.

I then suggest you do the test also with a Model (1)B a Model 2B, and a Model 2B+. There are (some significant) differences between these models and the model 3.

Finally, a more realistic SD card failure test (if that is what you want to investigate) is of course to switch the power at the mains end, as klricks suggests, without a UPS, but that adds another variable, the wall-wart.

To somewhat simulate this with your current setup, you could add a large capacitor at the 5V end, so the slower collapse would mean more time spent in the critical voltage region below say 4.7V and you could even bring it back up for a small amount of time to simulate a kind of Brown-Out.

Indeed I am going to stop this test shortly, run a checksum on the files to check for any corruption and then continue the test with an intentional file write during the power cut. I think a simple dd command writing to a file should do it. I will indeed test older hardware when this is done. I have a feeling it will not fare as well.

I don't think the wall wart switching will make a difference, since I also have an output capacitor that needs to discharge. I could add more output capacitance.. but how much do commercial USB power adapters have anyway? Can't be too much more then what I already have (150uF). Something to think about.

The successful boot count is now at 25665!

Regards,
-Moses

[gregeric]

I will indeed test older hardware when this is done. I have a feeling it will not fare as well.

I would expect little difference between hardware revisions.

OTOH older firmwares/kernels may not fare so well - there have been bugfixes, & switches between the eMMC & sdhost peripherals used for SD access over time, whilst the hardware has remained roughly the same.

[mosespi]

I stopped the test, took a snapshot of the logs and a checksum of all files. I will be checking them against a fresh install to check for any corruption. It was still booting up properly!

The final successful boot count was 26,823!

I now have the same setup but with a deliberate write. I created a 1 gig file of random data and on boot the Pi does a copy of this file to another location. Keeps the green activity light on the Pi 3 fairly solid. It never copies this file completely of course.. because in 15 seconds power goes out!

We will see if this fails sooner. 12 times so far.. I'm not even going to attempt to guess the results.

Regards,
-Moses

[Mortimer]

After this run, depending on the results maybe, can you run it again with a nice fat capacitor connected between 0V and 5V on the header to simulate the browning out of the power?

[mosespi]

An update: Over 1,000 power cuts during a write operation. Still booting up!

Regards,
-Moses

[TimG]

Moses, this is a very interesting test. It would also be interesting to know what errors fsck finds during boot, and what repairs it makes to the filesystem. However I'm not sure how it could be monitored. AFAIK fsck has to run before the filesystems are mounted, so cannot write a log file. Any ideas?

[mosespi]

The write test failed after 1,084 boot ups! Another SD card boots up in the same hardware just fine. Tried mounting the SD card on another system (not booting) and the root partition seems to be fine. The boot partition on the other hand has trouble.. kept getting the following errors in /var/log/messages

[ 2497.631514] FAT-fs (sda1): Volume was not properly unmounted. Some data may be corrupt. Please run fsck.

Tried fsck.. fsck.fat.. and "sudo dosfsck -w -r -l -a -v -t /dev/sda1"
Nothing helped. I guess it is too corrupt to fix.
The boot partition also has a bunch of these files.. FSCK4308.REC.. recovered data from fscks I believe.

The card has obviously gone corrupt.

As for the fsck errors.. I would get a little line during boot that the partitions were not cleanly unmounted and these in the syslog..

May 6 07:05:25 raspberrypi systemd-fsck[207]: fsck.fat 3.0.27 (2014-11-12)
May 6 07:05:25 raspberrypi systemd-fsck[207]: 0x41: Dirty bit is set. Fs was not properly unmounted and some data may be May 6 07:05:25 raspberrypi systemd-fsck[207]: Automatically removing dirty bit.
May 6 07:05:25 raspberrypi systemd-fsck[207]: Performing changes.
May 6 07:05:25 raspberrypi systemd-fsck[207]: /dev/mmcblk0p1: 126 files, 41524/82644 clusters

Not much else about checking the drives.

Before going any further, I may try reloading and running the write cut test again for one more data set.

Regards,
-Moses

[mosespi]

We are on boot up # 7,580 (!) on the second "write while power cut" test! Still ongoing!

Other then 2 failed boot ups, nothing unusually has happened. The activity light is pretty much solid green when the power gets cut.

So, what next? Add more capacitance on the 5v line to simulate more of a brownout condition?

Regards,
-Moses

[mosespi]

Boot # 21,056 on the second "write while power cut" test!

Regards,
-Moses

[DavidS]

look forward to the results of this test.

Also look forward to the brownout test. While it will take a lot longer than a minute to power out, may I recommend using a 10F capacitor for the brownout testing.

[mosespi]

I finally decided to end the "write while power cut" test.. final successful boot count is 27,327!!!
The system was working normally when I stopped the test!

For the next round of testing I attached a 1.5F supercapacitor (two 3F in series actually) on the 5v lines, which gives a gradual voltage drop from 5 to 2.5v in about 10 seconds. 2.5v to 3v is about when the Pi seems to loose complete operation. The SD card has been reformatted and a normal boot test, without any deliberate writing, has been started. So far on boot # 482 without trouble.

We will see how this goes!

Regards,
-Moses

[sparkyhall]

All interesting stuff. Towards the end of last year I tried to corrupt one of my SD cards by simply killing the mains power on a daily basis, I did this for several weeks without any issues. It was enough to convince me not to worry even though I have seen corruption on an early Pi with power loss.

I wonder how spinning rust would fair?

[rpdom]

All interesting stuff. Towards the end of last year I tried to corrupt one of my SD cards by simply killing the mains power on a daily basis, I did this for several weeks without any issues. It was enough to convince me not to worry even though I have seen corruption on an early Pi with power loss.

I wonder how spinning rust would fair?

Pretty well with current drives. IIRC they can use the the spinning disk to drive the motor as a generator in the case of a power loss and write their internal cache to the disk before parking the heads and shutting down.
It's been a long time since I used a drive where a power failure could be fatal and when I had to manually issue a "park" command to move the read/write head to an unused part of the disk (landing zone) so it wouldn't cause corruption when power was removed.

[davidcoton]

[quote="mosespi" I attached a 1.5F supercapacitor (two 3F in series actually)[/quote]

Umm. Assuming an adequate voltage rating, then to add the capacitance the supercaps should be in parallel.
The effective value of caps in series follows the same formula as resistors in parallel. But the overall voltage rating is the sum of the two (equal capacitors will share the voltage equally).

[mosespi]

I attached a 1.5F supercapacitor (two 3F in series actually)

Umm. Assuming an adequate voltage rating, then to add the capacitance the supercaps should be in parallel.
The effective value of caps in series follows the same formula as resistors in parallel. But the overall voltage rating is the sum of the two (equal capacitors will share the voltage equally).

I needed to increase the voltage rating since the capacitors themselves were only rated for 2.7v, hence the two in series.

Currently at boot # 1,256 with the slow brownout test.. looks like this is not going to make a difference.

Regards,
-Moses

[davidcoton]

I attached a 1.5F supercapacitor (two 3F in series actually)

Umm. Assuming an adequate voltage rating, then to add the capacitance the supercaps should be in parallel.
The effective value of caps in series follows the same formula as resistors in parallel. But the overall voltage rating is the sum of the two (equal capacitors will share the voltage equally).

I needed to increase the voltage rating since the capacitors themselves were only rated for 2.7v, hence the two in series.

Currently at boot # 1,256 with the slow brownout test.. looks like this is not going to make a difference.

Regards,
-Moses

Sorry, not sure what I was complaining about -- your calculation is of course correct. I really must stop writing posts late at night when I'm tired.

[mosespi]

An update: I stopped the slow brownout test at 4,464 startups. It occasionally failed to boot, getting stuck on the large rainbow square (note this is not the small rainbow square error). The failed boot ups became more frequent until they got troublesome and I elected to end the test. This was not a complete failure, as I could boot up most times and use the system normally. I have a feeling something on the SD card was not working 100% or something minor got corrupted or wasn't reading right all of the time. A rebuild cleared up the issue completely. You have to remember I have been using the same SD card during this entire test and I have not been kind to it!

The SD card has been rebuilt and a new test has begun. Same brownout test but this time with a heavy SD card write, copying a 1 gig file during shutdown/brownout. 1,395 successful startups so far!

Time to find my older Pi models to continue this test. As it looks like I can't reliably corrupt a Pi 3 within a reasonable number of attempts!

Regards,
-Moses

[rpdom]

It might be interesting to try this test with different models/makes of SD card to see which can handle the repeated power failure scenario best.

Also, did you check on each reboot that the files written on the previous one were maintained? SD cards do have a failure mode where things appear to be written correctly, but aren't there after a power cycle. Perhaps add a sequence number to the web server request and have the webserver verify that it is the next one in sequence. Write that number to the SD card each time a request is made.

[mosespi]

It might be interesting to try this test with different models/makes of SD card to see which can handle the repeated power failure scenario best.

Also, did you check on each reboot that the files written on the previous one were maintained? SD cards do have a failure mode where things appear to be written correctly, but aren't there after a power cycle. Perhaps add a sequence number to the web server request and have the webserver verify that it is the next one in sequence. Write that number to the SD card each time a request is made.

Before digging into different SD cards I'm going to try an older Pi model. Something to try in the future.

For the file write test.. the file copy/write never completes. A copy starts at boot on a 1 gig file with random data, it never completes this copy as the Pi gets abruptly shut down after 20 seconds, or 30 in the case of the brownout tests. So nothing to check really. I am looking for corruption of existing files on the card, to the point of not booting properly. Although I think once I repeat this test with an older model Pi.. I may rethink the test and possibly incorporate some type of automated file integrity check into the test itself.

Regards,
-Moses

[rpdom]

Ok, but before starting the copy, could you update a file with a sequence number and include that in the request to the webserver? Then you could check the webserver logs for myrequest&seq=.... and make sure that the sequence number updates. Otherwise you could be booting on a useless read-only card and not realising it.

I'm really interested in the realistic usable lifespan of SD cards. I have a few Pis that have been running continuously for years (barring power outages) for a few years now.

[mosespi]

Ok, but before starting the copy, could you update a file with a sequence number and include that in the request to the webserver? Then you could check the webserver logs for myrequest&seq=.... and make sure that the sequence number updates. Otherwise you could be booting on a useless read-only card and not realising it.

I'm really interested in the realistic usable lifespan of SD cards. I have a few Pis that have been running continuously for years (barring power outages) for a few years now.

I will consider this for the next test. I check system logs occasionally, so I don't believe anything is going read-only.

I'm sure someone somewhere must have done lifespan tests on SD cards. I do have one Pi that has been running for 3 or 4 years writing webcam files to the same USB key. It hasn't completely failed yet.. but I am starting to see corrupt files here and there.

Regards,
-Moses