jarkob
Posts: 14
Joined: Fri Jun 17, 2016 9:17 am

How to corrupt SD-Card for sure?

Fri Jun 17, 2016 10:56 am

Hi,

I want to make sure that my solution of protecting my sd-card against corruption is working properly.
What is the worst case scenario I can let it go through?

I have it powered via GPIO pins and cut the power in random intervals between 0-120000 milliseconds with an externally controlled relay. So it boots up and gets killed within two minutes anytime.
I also connected a control-pi with an unaltered system to check if my pi "should" be dead.

Now it is running for a full 24 hours, being killed around 7000 times (I keep a logfile of the killings).

Both pi's are still running!

The control pi does complain about a dirty bit, but the application and everything still works just fine.
Error Message is:

Code: Select all

[....] Checking file systems...fsck from util-linux 2.20.1
dosfsk 3.0.13, 30 Jun 2012, FAT32, LFM
0x25: Dirty bit is set. Fs was not properly unmounted and some data
 Automatically removing dirty bit.
PErforming changes.
/dev/mmcblk0p1: 53 files, 2427/7161 clusters
fsk died with exit status 1
done.
So.. slightly disappointing. I was hoping the SD-Card from the control-pi would be destroyed by now.

How can I make my control setup more extreme to check if the solution really works?

-
FYI:
I run Raspbian Wheezy (2015-05-07) on both of them.

The "safe" pi mounts /boot & / read-only. This is its altered /etc/fstab:

Code: Select all

proc            /proc           proc    defaults                               0       0
/dev/mmcblk0p1  /boot           vfat    ro                                     0       0
/dev/mmcblk0p2  /               ext4    ro,noatime                             0       0
tmpfs           /tmp            tmpfs   nodev,nosuid,size=100M                 0       0
none            /var/run        ramfs   size=1M                                0       0
none            /var/log        ramfs   size=1M                                0       0
I hope this is the right place for my question..

fanoush
Posts: 463
Joined: Mon Feb 27, 2012 2:37 pm

Re: How to corrupt SD-Card for sure?

Fri Jun 17, 2016 12:08 pm

you should do some writes during powering off, that should not only kill the filesystem but maybe even the card itself :-)

removing noatime wil trigger writes even if files are just read as the access timestamp is updated on each read of some file

jarkob
Posts: 14
Joined: Fri Jun 17, 2016 9:17 am

Re: How to corrupt SD-Card for sure?

Fri Jun 17, 2016 12:45 pm

Thanks for your reply!

Though I think I stated my question misleadingly ;-)

You gave a good solution for altering /etc/fstab to destroy the SD-card.

For me, the ultimate goal is not destroying the SD-card :mrgreen: But having a pi that survives the worst circumstances, since I had a lot of trouble with corrupted SD-cards lately in remote locations.

So I came up with the idea of mounting partly read-only and partly in RAM.
Now I want to test the setup by putting it in the worst possible environment. The setup itself I don't want to change to the worse though! I just want to make it as stable as possible.

I thought that the scenario I created (having an Arduino controlled relay randomly cutting of the power to the Pi) would be hell and sudden death. But it seems to be not hardcore enough.
Even my control-Pi running an unaltered Raspbian Wheezy survives it (almost) unharmed.

So my question is, what is the worst-case scenario in which I can test my setup?

mfa298
Posts: 1387
Joined: Tue Apr 22, 2014 11:18 am

Re: How to corrupt SD-Card for sure?

Fri Jun 17, 2016 12:57 pm

The other thing you could add in that might introduce some issues is running the test pi from a variable power supply and slowly reduce the voltage say be 0.05v at a time whilst also running the power cycle test.

Whislt it's not filesystem testing You might also want to try running the Pi for a period of time to check that things don't have issues. There's a variety of things that normally try to write to /etc and /var during normal operation which might have issues with the read only filesystem.

rln
Posts: 174
Joined: Wed Apr 09, 2014 1:43 pm
Location: Sweden
Contact: Website

Re: How to corrupt SD-Card for sure?

Fri Jun 17, 2016 1:13 pm

I've been doing lots of work in the same field. To my knowledge, read-only helps but is not perfect. I go even one step further.
jarkob wrote:For me, the ultimate goal is not destroying the SD-card :mrgreen: But having a pi that survives the worst circumstances, since I had a lot of trouble with corrupted SD-cards lately in remote locations.
How does the errors you had look? Corrupt files? Corrupted FS? Can't boot?
Author of the robust Nard distro http://www.nard.se

klricks
Posts: 6544
Joined: Sat Jan 12, 2013 3:01 am
Location: Grants Pass, OR, USA
Contact: Website

Re: How to corrupt SD-Card for sure?

Fri Jun 17, 2016 1:18 pm

+1 to previous post by mfa298. Test in 'brownout' conditions.
Note that removing AC power from the power supply is quite different than abruptly removing 5V power to the GPIO.........
Unless specified otherwise my response is based on the latest and fully updated Raspbian Buster w/ Desktop OS.

jarkob
Posts: 14
Joined: Fri Jun 17, 2016 9:17 am

Re: How to corrupt SD-Card for sure?

Fri Jun 17, 2016 2:18 pm

mfa298 wrote:The other thing you could add in that might introduce some issues is running the test pi from a variable power supply and slowly reduce the voltage say be 0.05v at a time whilst also running the power cycle test.
great idea! I don't have immediate access to a variable power supply I can automate, but it's definitely worth checking out.
Just ordered this one here: http://tuxgraphics.org/electronics/2010 ... y-v3.shtml 8-)
mfa298 wrote:Whislt it's not filesystem testing You might also want to try running the Pi for a period of time to check that things don't have issues. There's a variety of things that normally try to write to /etc and /var during normal operation which might have issues with the read only filesystem.
You mean just letting it run for a couple of hours and see if it holds up?
rln wrote: How does the errors you had look? Corrupt files? Corrupted FS? Can't boot?
Can't boot up anymore due to kernel panic after somebody just pulled the power plug. I actually didn't see the error myself, since my friend installed them at a remote location I cannot easily access. But that's what he told me.
klricks wrote:+1 to previous post by mfa298. Test in 'brownout' conditions.
Note that removing AC power from the power supply is quite different than abruptly removing 5V power to the GPIO.........
Great input! I should power the Pi via the USB port + standard power supply, and connect the relay to the powerstrip to properly test it.
I chose GPIO, because it was easier to setup, but you're right. I'm introducing a variation of the setup with an unknown effect.
Could be interesting though if powering the Pi via GPIO gives you less danger of corrupting your card.

I'll rearrange my setup and let you know.


btw, I'm now at 10409 random kills and still no serious damage

mfa298
Posts: 1387
Joined: Tue Apr 22, 2014 11:18 am

Re: How to corrupt SD-Card for sure?

Fri Jun 17, 2016 2:30 pm

jarkob wrote:
mfa298 wrote:Whislt it's not filesystem testing You might also want to try running the Pi for a period of time to check that things don't have issues. There's a variety of things that normally try to write to /etc and /var during normal operation which might have issues with the read only filesystem.
You mean just letting it run for a couple of hours and see if it holds up?
Aim for at least as long as you expect it to be running for, but I'd expect most issues to show up within 24 hours (if it could run for longer without a reboot).

It would also be sensible to test the standard operations the system will be doing during normal use (and potentially things you might do as part of normal diagnostics - can you still ssh in/ connect a keyboard, mount a usb drive etc.). A hardened system is only useful if it still does the job it's designed to do. If the sort of operations you might want to do fail it's not so useful.

jarkob
Posts: 14
Joined: Fri Jun 17, 2016 9:17 am

Re: How to corrupt SD-Card for sure?

Fri Jun 17, 2016 4:35 pm

mfa298 wrote: Aim for at least as long as you expect it to be running for, but I'd expect most issues to show up within 24 hours (if it could run for longer without a reboot).
makes sense.
I'll let them run over the weekend.
one attached to the relay, and one just running.
mfa298 wrote: It would also be sensible to test the standard operations the system will be doing during normal use (and potentially things you might do as part of normal diagnostics - can you still ssh in/ connect a keyboard, mount a usb drive etc.). A hardened system is only useful if it still does the job it's designed to do. If the sort of operations you might want to do fail it's not so useful.
yes, i'm constantly checking if it still works ;) they control leds, and I have them attached. So I can immediately see when they give up on what they're supposed to do.
ssh, keyboard etc I also check and that works without issues.



[Edit:] Update: I have the relay now attached to the power socket, and the raspberries to USB power supplies... let's see what that does.

jarkob
Posts: 14
Joined: Fri Jun 17, 2016 9:17 am

Re: How to corrupt SD-Card for sure?

Fri Jun 17, 2016 5:07 pm

SWEET!!! :mrgreen: :mrgreen: :mrgreen: :mrgreen: :mrgreen:

constant Kernel panic after just 90 kills with the new setup (Pi powered via USB with 5V power supply connected to 220V power socket. relay switches the socket randomly on/off)

I'm going to try the same with a fresh image and see if it is consistent.

jarkob
Posts: 14
Joined: Fri Jun 17, 2016 9:17 am

Re: How to corrupt SD-Card for sure?

Fri Jun 17, 2016 5:13 pm

rln wrote:I've been doing lots of work in the same field. To my knowledge, read-only helps but is not perfect. I go even one step further.
Can you elaborate? What is your best solution so far?

I tried Alpine Linux, which completely loads into RAM and should be safer.
But I couldn't get the desired applications running properly, so I temporarily gave up.

Tzarls
Authorised Reseller
Authorised Reseller
Posts: 224
Joined: Tue Feb 26, 2013 6:59 am

Re: How to corrupt SD-Card for sure?

Fri Jun 17, 2016 5:17 pm

You must be the first one around here to be happy because of a corrupted SD card... :lol:

Keep us posted!

rln
Posts: 174
Joined: Wed Apr 09, 2014 1:43 pm
Location: Sweden
Contact: Website

Re: How to corrupt SD-Card for sure?

Fri Jun 17, 2016 5:47 pm

jarkob wrote:Can you elaborate? What is your best solution so far?
I tried Alpine Linux, which completely loads into RAM and should be safer.
My solution is (of course) Nard Linux which loads completely into RAM as well.
http://www.arbetsmyra.dyndns.org/nard/
You are most welcome to try the example image. If you do find it corrupts your SD cards as well I want to know about it so I can fix it.
Author of the robust Nard distro http://www.nard.se

jarkob
Posts: 14
Joined: Fri Jun 17, 2016 9:17 am

Re: How to corrupt SD-Card for sure?

Fri Jun 17, 2016 6:32 pm

If you don't know how to break it, you're not able to fix it ;) I will keep you posted!

The plan is to kill it 8000 times at random intervals, which should take until Saturday evening and then let it run until Monday morning without a break, kill it again and see if everything still works.

btw, I use 16GB SanDisk Ultra cards.
But I'll also make a testrun with a noname card and the new setup.

What really surprises me though is that it makes such a huge difference if you power your pi from GPIO and break the 5V supply or power it via usb and break the 220V supply... worth investigating.
The "shock"-curve might have a different shape, and following mfa298's hint, that could lead to a temporary state of undervoltage which is possibly deadly if applied at the right moment.
I'd love to get an ELI5 on that.

jarkob
Posts: 14
Joined: Fri Jun 17, 2016 9:17 am

Re: How to corrupt SD-Card for sure?

Fri Jun 17, 2016 6:38 pm

rln wrote: My solution is (of course) Nard Linux which loads completely into RAM as well.
http://www.arbetsmyra.dyndns.org/nard/
You are most welcome to try the example image. If you do find it corrupts your SD cards as well I want to know about it so I can fix it.
Ha, I should've checked your signature ;)
I'm downloading it as we speak.
Right now my Rpis are occupied for the weekend, but I'll try it next week.

rln
Posts: 174
Joined: Wed Apr 09, 2014 1:43 pm
Location: Sweden
Contact: Website

Re: How to corrupt SD-Card for sure?

Fri Jun 17, 2016 7:29 pm

jarkob wrote:Right now my Rpis are occupied for the weekend, but I'll try it next week.
Great. I'll be waiting eagerly. If a corruption occur; please make a raw image copy of the entire SD card so I can analyze it.
Author of the robust Nard distro http://www.nard.se

jarkob
Posts: 14
Joined: Fri Jun 17, 2016 9:17 am

Re: How to corrupt SD-Card for sure?

Fri Jun 17, 2016 7:43 pm

I actually found another raspberry and will compile nard as soon as i have downloaded the files.

I tried the prebuilt image already, but it doesn't boot.. I read on your site that it is for the Pi 1 B. I have a Pi 2 B, could that be the issue?

rln
Posts: 174
Joined: Wed Apr 09, 2014 1:43 pm
Location: Sweden
Contact: Website

Re: How to corrupt SD-Card for sure?

Fri Jun 17, 2016 8:06 pm

jarkob wrote:I tried the prebuilt image already, but it doesn't boot.. I read on your site that it is for the Pi 1 B. I have a Pi 2 B, could that be the issue?
Doesn't boot? :shock: It should work fine for Pi 2 B as well... I'll re-check it ASAP.

Update: The prebuilt image works. I've tested it again.
Author of the robust Nard distro http://www.nard.se

jarkob
Posts: 14
Joined: Fri Jun 17, 2016 9:17 am

Re: How to corrupt SD-Card for sure?

Fri Jun 17, 2016 9:24 pm

actually also the freshly compiled image doesn't boot :?
When I put it in, I see the red/green/blue gradient square on the upper right corner, the green ACT led is flashing normally and then after a couple of seconds everything stays black and the ACT led flashes in an interval of a second.

I have a Rpi 2 Model v1.1

I followed the instructions as:

Code: Select all

wget http://www.arbetsmyra.dyndns.org/nard/download/nard-v1.00.tar.gz
tar -xzf nard-v1.00.tar.gz
cd nard/
sudo -s
apt -y install g++ bison
  flex xutils-dev gperf make
  automake libtool-bin git texinfo
  libncurses5-dev libncursesw5-dev
  libexpat1-dev socat git dc
  libx11-dev curl gawk
  libcap-dev
exit
make skeleton
zcat sdcard.img.gz >/dev/mmcblk0
I also downloaded nard-latest.tar.gz, but since the md5sum is the same as nard-v1.00.tar.gz, I guess it'll be the same ;)

I compiled on Ubuntu 16.04

got the tarball from here: http://www.arbetsmyra.dyndns.org/nard/d ... est.tar.gz
(and here: http://www.arbetsmyra.dyndns.org/nard/d ... .00.tar.gz )
and the prebuilt image from here: http://www.arbetsmyra.dyndns.org/nard/d ... rd.img.zip

not sure what I do wrong...

Update: tried it on a second Pi2 Model B v1.1, same thing
Update2: tried it on a Pi1 2011,12, same thing

Here's a video of the booting procedure (I waited around 15 minutes after the black screen): https://we.tl/KB7IL4KgUo
Last edited by jarkob on Fri Jun 17, 2016 9:49 pm, edited 1 time in total.

rln
Posts: 174
Joined: Wed Apr 09, 2014 1:43 pm
Location: Sweden
Contact: Website

Re: How to corrupt SD-Card for sure?

Fri Jun 17, 2016 9:46 pm

jarkob wrote:I see the red/green/blue gradient square on the upper right corner, the green ACT led is flashing normally and then after a couple of seconds everything stays black and the ACT led flashes in an interval of a second.
The square is an indication of bad power. Change power supply and cables. The slowly flashing ACT LED likely mean Nard has booted after all. Nard is headless by default. There is no HDMI output (unless you create it yourself). You need either a serial console or SSH to get a login prompt. Tips: Try

Code: Select all

$ make ssh 
from your build dir for automatic network scan and login.
Author of the robust Nard distro http://www.nard.se

jarkob
Posts: 14
Joined: Fri Jun 17, 2016 9:17 am

Re: How to corrupt SD-Card for sure?

Fri Jun 17, 2016 9:51 pm

ha! ooooooooohkay
that makes sense then

I guess I was too impatient, sorry for wasting your time!

I'll give it another shot and ssh in there. give me a sec

Update:
make ssh found the pi, but failed with

Code: Select all

Makefile:53: recipe for target 'ssh' failed
make: *** [ssh] Error 4
But anyway, since I know the ip I just ssh'd with 'root' & 'pass' and I'm in.

For using it in my application I'd have to dig into it more, since I need java8 and have to adjust all kind of things I guess.

I'll attach it to my relay right now though and try to corrupt the card until monday with the same principle as described before.
If something pops up I'll make a raw image copy and let you have it ;)

stderr
Posts: 2178
Joined: Sat Dec 01, 2012 11:29 pm

Re: How to corrupt SD-Card for sure?

Sat Jun 18, 2016 4:43 am

jarkob wrote:The plan is to kill it 8000 times at random intervals, which should take until Saturday evening and
To get 8000 kills in by Saturday, are you even letting the thing boot before stopping it?

jarkob
Posts: 14
Joined: Fri Jun 17, 2016 9:17 am

Re: How to corrupt SD-Card for sure?

Sat Jun 18, 2016 4:50 pm

Well, not always. Since the timer to kill is random, they sometimes boot up to run a while and sometimes are killed during boot. This way I want to make sure that almost any case is tested.

In average they should be 8000 times killed by tonight, but I could be wrong. I use analogue noise as seed, but I'm not sure how random random() really is. ;)

Anyway I'll let it run a while longer if this test goes fine.

rln
Posts: 174
Joined: Wed Apr 09, 2014 1:43 pm
Location: Sweden
Contact: Website

Re: How to corrupt SD-Card for sure?

Tue Jun 21, 2016 8:30 am

Has your weekend test finished yet?
Author of the robust Nard distro http://www.nard.se

jarkob
Posts: 14
Joined: Fri Jun 17, 2016 9:17 am

Re: How to corrupt SD-Card for sure?

Tue Jun 21, 2016 9:00 am

Yep, it finished without issues after exactly 8000 kills, and then kept both Pis running for more than a full day.
Both seem to be fine and run without errors.
I used fsck to run a simple filesystem check on the cards, and that also had no complaints.

Do you recommend another method to check the validity of the filesystem?

I'm uploading an image of the skeleton right now. I guess this is what you're interested in, right?
https://we.tl/5KhdzC9gWI


CONCLUSIONS:

Code: Select all

Raspberry Pi 2 Model B v1.1
SanDisk Ultra 16 GB

image: unchanged raspbian wheezy
pi power @ microUSB PSU     +    power cut at 220V before PSU   =    kill 12 -> corruption constant kernel panic
pi power @ GPIO             +    power cut at 5V before GPIO    =    kill 60 -> dirty bit, still functional though

image: raspbian wheezy (mount partitions read-only or in RAM, in my case negligible warnings visible on boot)
pi power @ microUSB PSU     +    power cut at 220V before PSU   =    8000 kills -> fully functional
pi power @ GPIO             +    power cut at 5V before GPIO    =    8000 kills -> fully functional

image: unchanged nard (uploaded and executed completely from RAM)
pi power @ microUSB PSU     +    power cut at 220V before PSU   =    8000 kills -> fully functional
Last edited by jarkob on Wed Jun 22, 2016 12:20 pm, edited 4 times in total.

Return to “Troubleshooting”