ilpiero
Posts: 18
Joined: Sat Jul 12, 2014 1:18 pm

SD corruption with READ ONLY filesystem

Fri Oct 03, 2014 6:16 pm

Hi all,

I have used some Pi's in an industrial test system, their task is simply to output a pattern over the HDMI and play a tune on the audio all the time.
A system has two Pi's and they work perfectly .. until they work ;) .

Some weeks after the deploy of the system i was reported that both the Rasps died... the cause was the dreaded SD card corruption problem.

After that report i made the filesystem (the pi is running raspbian) read only, but again after a week i got a corruption.

The usage condition is something like , ON 12 hours a day, 8/10 shutdowns per day ... no safe shutdown, the power is cut.

I wanted to share some toughs with you ad see if it is something that is solvable, because i have another project starting and i have to choose whether use the Pi or not.

1) Is safe shutdown needed if i use a read only filesystem? There should be no cache to flush ... no writes at all, why the SD would benefit of a safe shutdown?

2) Are there some write operation if i set the FS read only? I'm confident that if i mount the fs as ro no writes will be performed... but are there write operations to the SD BEFORE the fs is mounted? I don't think so but maybe I'm wrong...

3) How critical is a corruption ? I'm just unlucky that i hit the MBR all of the times? Or there are a lot of corruptions that will eventually make the system not bootable?

4) Is it somehow related to the PI hardware or EVERY device has the same failure rate? I mean SD cards are being used by some other embedded system ... do they all suffer this problem?

5) Someone decided to go for a FS on a USB stick... why should it work better? it's a flash memory as well ... maybe the USB controller will filter the bad operation?

I would be glad to hear your opinions about it :)

Thanks

User avatar
patrikg
Posts: 168
Joined: Sun Mar 18, 2012 10:19 pm

Re: SD corruption with READ ONLY filesystem

Fri Oct 03, 2014 7:51 pm

I am now trying linux distributions that don't use the flash/nand based memory disc for ...log...mail...tmp...and many another disc IO, like openwrt or tinycorelinux.
Or just use a nfs mounted root from a another computer or nfs nas.

NFS Root:
http://youresuchageek.blogspot.se/2013/ ... share.html

TinyCoreLinux:
http://tinycorelinux.net/ports.html

OpenWRT:
http://wiki.openwrt.org/toh/raspberry_pi
https://downloads.openwrt.org/barrier_b ... 8/generic/

And just 4 fun:
Look at this prez:
http://media.ccc.de/browse/congress/201 ... _xobs.html

FM81
Posts: 518
Joined: Wed Apr 17, 2013 4:33 pm

Re: SD corruption with READ ONLY filesystem

Fri Oct 03, 2014 8:11 pm

ilpiero wrote:After that report i made the filesystem (the pi is running raspbian) read only, but again after a week i got a corruption.
What was your steps, to make it read-only?

Greetings, FM_81
A: What does the command 'cat /dev/urandom', can you tell me please?
B: Yeah, that's very simple: It feeds your cat with radioactive material!

ilpiero
Posts: 18
Joined: Sat Jul 12, 2014 1:18 pm

Re: SD corruption with READ ONLY filesystem

Fri Oct 03, 2014 9:59 pm

I followed this tutorial :
http://blog.pi3g.com/2014/04/make-raspb ... read-only/
after that I did some trivial verifications (like writing a file...) and is seemed read only.

@patrikg

This is a very interesting video, I already dealt with wear levelling before, when I worked with SSD drives.
So, does some sort of hi quality card exist (the best part of the silicon)? Or some way to introduce redundancy at the cost of drive space...

Unfortunately a linux host is not avaliable for a network boot, otherwise it could have been a possible solution.

User avatar
patrikg
Posts: 168
Joined: Sun Mar 18, 2012 10:19 pm

Re: SD corruption with READ ONLY filesystem

Fri Oct 03, 2014 10:44 pm

Here are a good article about flash/nand.

https://wiki.linaro.org/WorkingGroups/K ... CardSurvey?

And here one youtube video about BadUsb thumb memory.
http://www.youtube.com/watch?v=xcsxeJz3blI

adambricker
Posts: 1
Joined: Sat Oct 04, 2014 3:40 am
Location: Houston, Texas, USA

Re: SD corruption with READ ONLY filesystem

Sat Oct 04, 2014 4:11 am

I was also having problems with SD corruption when I was running the RPi version of Plan 9 (Bell Labs). I believe in most cases it is due to having a swap partition enabled. If you boot Linux read only, but it's still using swap it will write to the swap partition. Linux makes heavy use of the swap partition (I assume Plan 9 does also) so that no amount of load levelling will solve the problem. Making the other partitions read only doesn't help at all in this case because the number of times a block is deleted and rewritten on a non-swap partition is a tiny fraction of how many times it happens on a swap partition. The latest Plan 9 image for RPi doesn't include a swap partition (as far as I can tell) and I haven't had any issues yet, though it's only been running for about 3 weeks straight.
Adam Bricker

FM81
Posts: 518
Joined: Wed Apr 17, 2013 4:33 pm

Re: SD corruption with READ ONLY filesystem

Sat Oct 04, 2014 6:36 am

That's very strange?
Can you test, if number of written sectors really increases permanently during runtime? For example with:

Code: Select all

awk '{print $7}' /sys/block/mmcblk0/stat
Or is it staying at (nearly) the same value?

Greetings, FM_81
A: What does the command 'cat /dev/urandom', can you tell me please?
B: Yeah, that's very simple: It feeds your cat with radioactive material!

ilpiero
Posts: 18
Joined: Sat Jul 12, 2014 1:18 pm

Re: SD corruption with READ ONLY filesystem

Sat Oct 04, 2014 8:29 am

I will check Monday, once I'm back in my lab. The swap partition is a interesting suggestion too.

Hopefully in the near future (if I'm not too busy) i may be able to do some automated stress test with a programmable power supply.

Thanks guys.

User avatar
ilguargua
Posts: 73
Joined: Thu Jun 19, 2014 8:14 pm
Location: Livorno - Italy

Re: SD corruption with READ ONLY filesystem

Sat Oct 04, 2014 11:29 am

From the article linked above (Wear Leveling section) :

"
[...]
However, some CF cards are known to do static wear leveling in a way that leads to data loss when the supply voltage gets lost while the card is doing static wear leveling. This is even the case for read-only cards, since the static wear leveling can get triggered by read accesses on those cards.
"

Cheers, Ale.

User avatar
r3d4
Posts: 978
Joined: Sat Jul 30, 2011 8:21 am
Location: ./

Re: SD corruption with READ ONLY filesystem

Sat Oct 04, 2014 11:31 am

what patrikg said
http://www.raspberrypi.org/forums/viewt ... 00#p621900
+1

Also you might find this http://www.linuxjournal.com/content/pra ... re-service
and the
'Interesting deployments' topic on the tinycore forum
intresting/usefull.
Real life is, to most, a long second-best, a perpetual compromise between the ideal and the possible.
-
Meanwhile, the sysadmin who accidentally nuked the data reckons "its best not run anything more with sudo today"
-
what about spike milligan?

ilpiero
Posts: 18
Joined: Sat Jul 12, 2014 1:18 pm

Re: SD corruption with READ ONLY filesystem

Sat Oct 04, 2014 1:57 pm

Tinycore maybe a way to go, but again, it will be the same of a read only raspbian, from the "corruption" point of view, right?

If the SD is performing static wear leveling when it's read then I'm simply doomed :shock:

Thanks!

User avatar
patrikg
Posts: 168
Joined: Sun Mar 18, 2012 10:19 pm

Re: SD corruption with READ ONLY filesystem

Sat Oct 04, 2014 2:27 pm

I don't really agree that it's the same as read only raspbian.

Because tinycore linux runs only in ram.

http://en.wikipedia.org/wiki/List_of_Li ... n_from_RAM

Therefore it will only read on the sd at boot.
If you are planning to reboot the system a lot of times there will be many reads. ;)
So much doomed, I don't think you are. :geek:

Little curious what programs you run, if you just run omxplayer?
http://forum.tinycorelinux.net/index.php?topic=16944.0

User avatar
r3d4
Posts: 978
Joined: Sat Jul 30, 2011 8:21 am
Location: ./

Re: SD corruption with READ ONLY filesystem

Sat Oct 04, 2014 3:14 pm

patrikg wrote:So much doomed, I don't think you are. :geek:
:lol:


I have had a little trouble with raspbian freeze and refuse to boot after hard shut down , vweard things i might post about if i find the time/inclination to document ,.. anyway
so far tinycore has never failed , hard shutdown removeing the plug ect
it will allways reboot

just remember its runing in ram so dont fill up the tmp-fs
write to usb storage
all will be well
8-)
Real life is, to most, a long second-best, a perpetual compromise between the ideal and the possible.
-
Meanwhile, the sysadmin who accidentally nuked the data reckons "its best not run anything more with sudo today"
-
what about spike milligan?

FM81
Posts: 518
Joined: Wed Apr 17, 2013 4:33 pm

Re: SD corruption with READ ONLY filesystem

Sat Oct 04, 2014 4:13 pm

However, some CF cards are known to do static wear leveling in a way that leads to data loss when the supply voltage gets lost while the card is doing static wear leveling. This is even the case for read-only cards, since the static wear leveling can get triggered by read accesses on those cards.
Just for clarification:
So is it correct, that only some (bad) cards may have this problem and all the others (good) didn't?
So it can be also possible, that you're in luck (having a 'good' card, which doesn't this static wear leveling) and you should never have problems with normal raspbian read-only?
It's nearly impossible to decide if you have a 'good' or a 'bad' BEFORE you (or somebody else) tested it?
But with tinycore (and other INITRAMFS-systems) it shouldn't matter if you have a 'good' or a 'bad'?

Correct? Greetings, FM_81
A: What does the command 'cat /dev/urandom', can you tell me please?
B: Yeah, that's very simple: It feeds your cat with radioactive material!

User avatar
rpdom
Posts: 16320
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: SD corruption with READ ONLY filesystem

Sat Oct 04, 2014 4:20 pm

FM81 wrote:
However, some CF cards are known to do static wear leveling in a way that leads to data loss when the supply voltage gets lost while the card is doing static wear leveling. This is even the case for read-only cards, since the static wear leveling can get triggered by read accesses on those cards.
Just for clarification:
So is it correct, that only some (bad) cards may have this problem and all the others (good) didn't?
So it can be also possible, that you're in luck (having a 'good' card, which doesn't this static wear leveling) and you should never have problems with normal raspbian read-only?
It's nearly impossible to decide if you have a 'good' or a 'bad' BEFORE you (or somebody else) tested it?
But with tinycore (and other INITRAMFS-systems) it shouldn't matter if you have a 'good' or a 'bad'?

Correct? Greetings, FM_81
Seeing as that paragraph is talking about CF cards, and the Pi uses SD cards, and
Very few SD cards use static wear leveling
(from the same part of that article), you don't really need to worry about this.

ilpiero
Posts: 18
Joined: Sat Jul 12, 2014 1:18 pm

Re: SD corruption with READ ONLY filesystem

Mon Oct 06, 2014 10:32 am

patrikg wrote:I don't really agree that it's the same as read only raspbian.

Because tinycore linux runs only in ram.

http://en.wikipedia.org/wiki/List_of_Li ... n_from_RAM

Therefore it will only read on the sd at boot.
If you are planning to reboot the system a lot of times there will be many reads. ;)
So much doomed, I don't think you are. :geek:

Little curious what programs you run, if you just run omxplayer?
http://forum.tinycorelinux.net/index.php?topic=16944.0
Yes I run omxplayer with a simple .wav file...

ilpiero
Posts: 18
Joined: Sat Jul 12, 2014 1:18 pm

Re: SD corruption with READ ONLY filesystem

Tue Oct 07, 2014 2:52 pm

FM81 wrote:That's very strange?
Can you test, if number of written sectors really increases permanently during runtime? For example with:

Code: Select all

awk '{print $7}' /sys/block/mmcblk0/stat
Or is it staying at (nearly) the same value?

Greetings, FM_81
Here is the output, the number you are looking for is 520, if i reboot it does not increase.

Code: Select all

 cat /sys/block/mmcblk0/stat 
    3453     4059   223331    21090       34       31      520    34580        0    16160    55610

FM81
Posts: 518
Joined: Wed Apr 17, 2013 4:33 pm

Re: SD corruption with READ ONLY filesystem

Thu Oct 09, 2014 8:52 am

@ilpiero: As far as I can say: Then should there be no write-events, at least seen from linux-kernel-side.
But it says nothing if there is "something like internal wear-leveling in the card itself".

Best regards, FM_81
A: What does the command 'cat /dev/urandom', can you tell me please?
B: Yeah, that's very simple: It feeds your cat with radioactive material!

fruit-uk
Posts: 609
Joined: Wed Aug 06, 2014 4:19 pm
Location: Suffolk, UK

Re: SD corruption with READ ONLY filesystem

Thu Oct 09, 2014 9:22 am

A search for 'read disturb' might bring up some interesting reading.

A long time ago when I was running debian on an Alix board (OS on CF )I used an aufs system, basically running totally in RAM. It may be that the distros above are doing similar.

There are many links describing the method such as this for USB https://help.ubuntu.com/community/aufsR ... OnUsbFlash - probably not for the fainthearted!

I believe voyage linux uses a similar system.

Return to “Troubleshooting”