User avatar
Lob0426
Posts: 2198
Joined: Fri Aug 05, 2011 4:30 pm
Location: Susanville CA.
Contact: Website

Re: Prevent SD-Card Corruption

Sat Jan 10, 2015 2:51 am

Most of my "card corruptions" have been due to power "brownouts". If the power goes out, it usually does not cause corruption. Early on it was an OS issue, but that was solved long ago. I would look at your power supply setup first. It could be the cards, but there should be a way to test them.
512MB version 2.0 as WordPress Server
Motorola Lapdock with Pi2B
Modded Rev 1.0 with pin headers at USB

http://rich1.dyndns.tv/
(RS)Allied ships old stock to reward its Customers for long wait!

axkibe
Posts: 4
Joined: Wed Jan 21, 2015 12:47 pm

Re: Prevent SD-Card Corruption

Wed Jan 21, 2015 12:53 pm

I'm using one Rasperry:Pi as Samba4 Domain Controller for a small computer lab (9 clients). Just for authenticating users, the Rasp sure would be enough, right?

Was running it for a year now without any issues. Then after a power failure it refused to boot, no init found. Okay, fsck.ext4 failed really badly. I tried to rewrite the SD card with another raspian image, but it continued to fail. Okay that card is smoke.

Got myself a new card, after looking at the Wiki Sony seemed to be the only manufacturer nobody reported a bad for. So I got a sony card. Reinstalled everything, Reconnected all Domain Clients, Recreated all User accounts.

Now 2 days later, I wanted to make a backup image of the card with all users, poweroff the rasp, plugged the SD-Card into my notebook, made an image and tried to power it up again. Corruption!!!

I'm right now quite seriously pissed. As well my users are.

I'm using the powersupply RS suggested me to be used with the rasperry.

Right now I try to rewrite the card with the last image.

But eitherway, if there isn't get to be any fix. I have to throw away the rasperry and get a real computer to be DC.

johnspackman
Posts: 18
Joined: Tue Jul 16, 2013 9:29 am

Re: Prevent SD-Card Corruption

Wed Jan 21, 2015 1:53 pm

Is there a reason why you didnt choose the SD cards sold by Raspberry? The NOOBS cards are 8Gb cards and seem pretty solid, just wipe them and put your preferred distro on.

SD cards are not as reliable as good old fashioned mechanical hard drives so if you have anything really sensitive stick it on an external HDD - it's probably cheaper than a whole PC just for a domain controller.

Alternatively (or as well as a HDD) use rsync or similar to backup your server regularly

axkibe
Posts: 4
Joined: Wed Jan 21, 2015 12:47 pm

Re: Prevent SD-Card Corruption

Wed Jan 21, 2015 4:07 pm

The first card I used was one I bought from RS as bundle. For the other I explicitly looked up the wiki to see which are supposed to be good.

The way samba4 uses hard links and ACL will make rsync backups fail anyway.

Dunno why SD cards are supposed to be less reliable. The SSD hard drive in my notebook made zero issues so far.

Sorry but I'm likely gonna dump the rasperry. Using it in a production environment was maybe a bad idea. Am now in the process of setting up just some old pentium as replacement.

User avatar
RaTTuS
Posts: 10500
Joined: Tue Nov 29, 2011 11:12 am
Location: North West UK
Contact: Twitter YouTube

Re: Prevent SD-Card Corruption

Wed Jan 21, 2015 4:15 pm

calm down...!
you've been using it for a year with no issues
so grab a new SD card
write the backup
carry on...

you have had an issue with the last sdcard you used - this could be all sorts of reasons ...

was it fake ? did you test ?

has the PSU declined over time ?

what is the Voltage as measured on the test points?

has anything else changed?

are you getting a vast amount of writes to the SD card ?
How To ask Questions :- http://www.catb.org/esr/faqs/smart-questions.html
WARNING - some parts of this post may be erroneous YMMV

1QC43qbL5FySu2Pi51vGqKqxy3UiJgukSX
Covfefe

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 24139
Joined: Sat Jul 30, 2011 7:41 pm

Re: Prevent SD-Card Corruption

Wed Jan 21, 2015 4:33 pm

axkibe wrote:The first card I used was one I bought from RS as bundle. For the other I explicitly looked up the wiki to see which are supposed to be good.

The way samba4 uses hard links and ACL will make rsync backups fail anyway.

Dunno why SD cards are supposed to be less reliable. The SSD hard drive in my notebook made zero issues so far.

Sorry but I'm likely gonna dump the rasperry. Using it in a production environment was maybe a bad idea. Am now in the process of setting up just some old pentium as replacement.
Please donate any unused Raspberry's to a local school, they would be more than happy to have it.

Oh, and SD cards are much more unreliable than SSD's due to their design (it's not only the memory itself, which is cheaper, but the controller), but on the other hand I've had more SSD's fail on me than SD cards...go figure.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I think it’s wrong that only one company makes the game Monopoly.” – Steven Wright

axkibe
Posts: 4
Joined: Wed Jan 21, 2015 12:47 pm

Re: Prevent SD-Card Corruption

Wed Jan 21, 2015 5:03 pm

go figure ... what?

User avatar
DougieLawson
Posts: 36548
Joined: Sun Jun 16, 2013 11:19 pm
Location: Basingstoke, UK
Contact: Website Twitter

Re: Prevent SD-Card Corruption

Wed Jan 21, 2015 5:07 pm

axkibe wrote:go figure ... what?
I think James is trying (in his terribly English way) to say that most of the SDCard corruption problems have been fixed by firmware changes, by ensuring cards are not Chinese fakes and by ensuring that the power supply for a machine that runs 24x7 can reliably provide a steady 5V0 @ 1A (or better). He's experienced more SSD failure events than most and very few SDCard failures (if any).

I've only had one 32GB SDCard go bad on me, that was with ancient firmware.
Note: Having anything humorous in your signature is completely banned on this forum. Wear a tin-foil hat and you'll get a ban.

Any DMs sent on Twitter will be answered next month.

This is a doctor free zone.

axkibe
Posts: 4
Joined: Wed Jan 21, 2015 12:47 pm

Re: Prevent SD-Card Corruption

Wed Jan 21, 2015 5:19 pm

How to ensure cards are not chinese fakes?

I got as replacement a SONY since http://elinux.org/RPi_SD_cards listed them to be very reliable and bothered the extra travel to get them from a SONY resaler I bought in the past VAIOS from.

Anyway, I really have to replace the rasp. I had to promise my users something has to change, since it failed the third time now...

Also I learned Samba4 DC seems to use some CPU-Loading encryption, albeit its doesn't have to do more than just authentication. It was slow. A 2GHz legacy old box will be a better replacement. Albeit the coolness factor is way less and I need to get a slot in the server room rather than being able to ducktape it into the media control cabinet, like the rasp was.

mxa
Posts: 9
Joined: Mon Mar 30, 2015 6:35 am

Re: Prevent SD-Card Corruption

Sat May 23, 2015 2:53 am

I have 9 Raspberries running a synchronized video player in a museum. After only three days, two of the SD Cards (SAMSUNG, 8GB) got corrupted and failed to boot. The raspberries were set up to shut down automatically after the day and were supposed to be triggered on by removing power for a few seconds in the morning. I don't know where it failed, may be that the museum staff just cut power in the evening before the proper shutdown happened.
Anyways, it is pretty troublesome and since it is supposed to run for 4 months without any more problems I am looking for ways to rule out SD Card corruption in the future. I have seen some suggestions how to run a read only system, disable logging or to use a USB thumb drive (which seems more robust) as a home partition. I also came across this interesting approach how to hardware hack the sdcard reader but it seems that is for the old model.
Anyway, what is the current state of affairs? Which route to go? I need a solution.
Last edited by mxa on Sun May 24, 2015 3:52 am, edited 1 time in total.

paulv
Posts: 558
Joined: Tue Jan 15, 2013 12:10 pm
Location: Netherlands

Re: Prevent SD-Card Corruption

Sat May 23, 2015 7:30 am

Mxa,

The Pi was not really intended to be used in these kind of environments. It can be made to work that way, but you need to improve the conditions in which the Pi can function.

As you will have seen in this post, many problems are a result of the way the Pi is shutdown, or halted. Closely related are the infamous brown-outs that may happen with the traditional wal-warts that most people use/buy with their Pi, thinking, falsely, that these so called approved supplies are OK because they deliver 5V with enough current.

These wal-warts were never designed to supply a computer with uninterruptable power.

There are several (also commercial) solutions available to help solve your power issues, look for those that will allow the Pi to function during a brown-out, properly shutdown at a loss of power, and start the Pi when power is switched on.

I designed such a DIY supply that was intended for exactly the (embedded) application you have. Have a look here : viewtopic.php?f=37&t=102015

My 5 Pies have a solid 5V supply with various variations of a UPS design, and since then I have not seen an SD card issue in almost 2 years.
Success!
paulv

shuckle
Posts: 565
Joined: Sun Aug 26, 2012 11:49 am
Location: Finland

Re: Prevent SD-Card Corruption

Sat May 23, 2015 11:46 am

Try to minimize all sd card writes. Use memory file systems and stop all loggins.
I have been running 24/7 systems which die now and then and i have not seen corruptions for a long time.
Akso avoid overclocking if possible.

ktb
Posts: 1380
Joined: Fri Dec 26, 2014 7:53 pm

Re: Prevent SD-Card Corruption

Sat May 23, 2015 12:31 pm

mxa,

This might be useful for you:
http://wiki.alpinelinux.org/wiki/RPI_Video_Receiver

User avatar
PiModules
Posts: 96
Joined: Sun Mar 03, 2013 9:20 pm
Location: Athens -Greece
Contact: Website

Re: Prevent SD-Card Corruption

Sat May 23, 2015 12:35 pm

Hi,

You can simply use the UPS PIco that solve completely the card corruption problem, offering a plenty of other extra features. The most important is that you do not need to have any extra powering system, just put it on top of RPi, power as usual, and that is it. See here:

http://www.pimodulescart.com/shop/item.aspx?itemid=5

Warmest Regards
Pi Master

mxa
Posts: 9
Joined: Mon Mar 30, 2015 6:35 am

Re: Prevent SD-Card Corruption

Sun May 24, 2015 6:57 am

shuckle wrote:Try to minimize all sd card writes. Use memory file systems and stop all logging.
I have been running 24/7 systems which die now and then and i have not seen corruptions for a long time.
There are a lot of tutorials out there, but most of them quite old. Which way is the current and recommended to disable all log activity?
shuckle wrote:Also avoid overclocking if possible.
OK, not overckocking. RPi is fast enough for my video playback needs.

mxa
Posts: 9
Joined: Mon Mar 30, 2015 6:35 am

Re: Prevent SD-Card Corruption

Sun May 24, 2015 7:05 am

Very interesting indeed. Do you think my setup and particulary the omxplayer-sync and python dependancies will run on Alpine Linux?

ktb
Posts: 1380
Joined: Fri Dec 26, 2014 7:53 pm

Re: Prevent SD-Card Corruption

Sun May 24, 2015 7:47 am

mxa wrote:
Very interesting indeed. Do you think my setup and particulary the omxplayer-sync and python dependancies will run on Alpine Linux?
Yes, I do think it would work. However, I think you would need to set up at least one Pi (or cross-compile environment on another computer) for development so you could compile any of these extra software packages which aren't already in the Alpine repositories. It might take a bit of time/effort to get familiar with using Alpine Linux. I have used it on my B+ and really like it, but I haven't been using my B+ very much these days. I think they have recently added Pi2 support. I'm actually about to test it out on my Pi2B. I'll let you know how that goes.

mxa
Posts: 9
Joined: Mon Mar 30, 2015 6:35 am

Re: Prevent SD-Card Corruption

Fri May 29, 2015 7:32 am

Following this anwser I removed dphys

Code: Select all

sudo apt-get remove dphys-swapfile
and changed
fstab like this:

Code: Select all

/dev/mmcblk0p1 /boot vfat defaults,noatime 0 2
/dev/mmcblk0p2 / ext4 defaults,noatime 0 1
tmpfs /var/tmp/ tmpfs nodev,nosuid,size=50M 0 0
tmpfs /var/log/ tmpfsnodev,nosuid,size=10M 0 0
I have no idea if this is gonna help the corruption issue. What do you think?
I found also this different site which recommends the ro (read-only) flag instead.

Code: Select all

/dev/mmcblk0p1  /boot       vfat    defaults,ro                0       2
/dev/mmcblk0p2  /           ext4    defaults,ro,noatime        0       1
That seems a bit brutal, since all services relying on writing will fail. Never the less the poster has found it to be successful for him. Please advise which strategy is better. At the mean time I play with AlpineLinux and see how far I can go with that.

johnspackman
Posts: 18
Joined: Tue Jul 16, 2013 9:29 am

Re: Prevent SD-Card Corruption

Mon Jun 08, 2015 6:12 am

Corruption is caused by power failures at the same time as the disk is being written to. If the power fails while the disk is not being written to, you're OK (assuming that your PSU is a suitably good quality etc)

Removing dphy-swapfile stops the pi from being able to swap to disk when memory gets low, it's useful in this case because that is one less thing that needs to write to disk.

Changing /var/log and /var/tmp into tmpfs (a ram disk) means that files those directories will never write to the physical disk, and /var/log in particular is very frequently written to. The disadvantage is that they are stored in RAM, so you've just given away 60Mb of RAM that is no longer available for applications and when you reboot you will loose anything stored in those directories. /var/log can fill up too, so you need to adjust /etc/rsyslog.conf to add log rotation or reduce the amount of logging going on.

User avatar
rpdom
Posts: 15580
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: Prevent SD-Card Corruption

Mon Jun 08, 2015 6:32 am

johnspackman wrote:Corruption is caused by power failures at the same time as the disk is being written to. If the power fails while the disk is not being written to, you're OK (assuming that your PSU is a suitably good quality etc)
Yes, but the "disk" in this case is an SD card, and SD cards perform all sorts of wear-level and maintenance functions at various times - usually when they aren't busy doing something else. While performing those operations it will be shuffling data around and writing out buffers - data could still get corrupted if power is removed during those times.

johnspackman
Posts: 18
Joined: Tue Jul 16, 2013 9:29 am

Re: Prevent SD-Card Corruption

Mon Jun 08, 2015 8:18 am

Thanks I'm aware of that, and wear leveling will only occur if the disk has been written to; by reducing the number of writes to disk (the SD card) you reduce the likelyhood that the disk will be written to by wear leveling or by the OS during a power failure.

User avatar
SFX Group
Posts: 74
Joined: Thu Nov 01, 2012 3:54 am
Location: Cambridge, ON. Canada
Contact: Website

Re: Prevent SD-Card Corruption

Wed Jul 01, 2015 3:48 am

We found the Pi to be a great idea, however its storage medium was flawed. We started using the PI when it first came out, since then it has eaten memory cards like floppy disks stored next to the microwave.

The only common thing we found was the PI was at the center of it all, on all occasions it eat memory cards no matter the make, quality, speed in use, including what version of operating system, which PSU, how many APC UPS units we had, weather it was shut down or kept running, it just eats cards and kills them so they cant be used again anywhere.

At one point we were RMA 8 Kingston class 10 cards at a time, Kingston said "yes, the Pi, that eats cards" so there aware of this.
We have had it eat Sony cards wirth $60, we have used many power supplies, tested them, load tested them, tested the PIs, used APC UPS units, have HP POE switches with $8000 running them, no matter what happened the Pi within about 8 weeks eat a card.

We have had many PIs including the B (and that eat cards as well), we have had many different PSU units, POE adapters (which are way more stable as a PSU device) running from $1000+ HP POE switches which are on APC UPS units.

What we found is the most cards would pass a READ test, however where unable to WRITE to the card, we are actually thinking there is some hardware fault that is burning out the cards write capacity for what ever reason.

We had a number of PIs never shut down and what would you know it eat card for what appears to be no reason, and that was on an APC UPS unit.

Unfortunately we did position the PI in a live environment (and spent alot of money specifically designing on OS for it which is now redundant), and had to move to the Intel Atom as that didnt eat drives. We are moving to another hardware design, its a shame as we do like the PI, however it basically doesn't work, it kills its storage device, you can blame what ever you want on this, however a PC or Intel hardware doesnt do this. We are soon testing Intels new PC on a stick.

Now i am sure out of the 1000 of PSU (although i doubt it) and the 1000's of cards 1 might work, but with those specifications its not a viable hardware for every day (any?) use.

I will add, any card it fails to boot on you need to FULLY test with a write then read test, as we found once that happened it killed the card permanently, the cards would read fine, but were unable to take data and store it.
Many Thanks
Ashley

© Copyright 2015 - www.ashleygriffin.ca

johnspackman
Posts: 18
Joined: Tue Jul 16, 2013 9:29 am

Re: Prevent SD-Card Corruption

Wed Jul 01, 2015 6:44 am

We have well over 70 Pi's installed on live customer sites, with power cuts, the-cleaner-turned-it-off cuts, users deliberately turning it off overnight, etc. They are powered by a £3 (US$5) PSU from RS Components, use a reformated NOOBS SD card, have heatsinks installed, are in a plastic case, and stuck to the back of TVs with velcro.

Some of these sites must be getting on for a year old by now, and I have replaced less than 10 units. I find that the Pi B's I originally used can develop hardware faults where the Pi itself becomes unreliable, which I put down to damage to overheating (before we installed heatsinks), and the past 50 Pis are B+'s.

The only thing i do is to reduce the writes to the disk by having a read only root partition, tmpfs for /var/log and /tmp, and a data partition that is read only except when the software needs to write to it.

ronald71
Posts: 1
Joined: Sat Sep 05, 2015 5:39 pm

Re: Prevent SD-Card Corruption

Sat Sep 05, 2015 5:55 pm

I have 3 Pi boards. Pi1 and Pi2. All develop SD card errors after some time. Power is stable with battery attached to PSU so the power failure does not happen. Sometimes after some weeks when I execute shutdown -r the board simply doesn't wake up, sometimes loads to the certain point and stops. Reflashing the card with a backup by diskimager helps instantly and the card is brought back to the normal condition. Sometimes apt-get upgrade kills the card so I need to upgrade the PI twice. The card is not physically destroyed, the card has only garbage data on it.
This is very good computer but for home use, learning and for the Linux fans but not for the production environment. If you run a web or e-mail server for yourself this is OK, when you discover failure you can resurrect it from the image and it is back on track.
I had not only ext4 filesystem corruption but also the boot loader was corrupted. It happened on the oldest Pi and on the Pi2.

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5370
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: Prevent SD-Card Corruption

Mon Sep 07, 2015 11:29 am

ronald71 wrote:I have 3 Pi boards. Pi1 and Pi2. All develop SD card errors after some time. Power is stable with battery attached to PSU so the power failure does not happen. Sometimes after some weeks when I execute shutdown -r the board simply doesn't wake up, sometimes loads to the certain point and stops. Reflashing the card with a backup by diskimager helps instantly and the card is brought back to the normal condition. Sometimes apt-get upgrade kills the card so I need to upgrade the PI twice. The card is not physically destroyed, the card has only garbage data on it.
This is very good computer but for home use, learning and for the Linux fans but not for the production environment. If you run a web or e-mail server for yourself this is OK, when you discover failure you can resurrect it from the image and it is back on track.
I had not only ext4 filesystem corruption but also the boot loader was corrupted. It happened on the oldest Pi and on the Pi2.
Are you up to date? What does "uname -a" report?

Return to “Troubleshooting”