Page 1 of 1

Persistent SD Card Corruption

Posted: Tue Nov 05, 2019 11:49 am
by LONGJA
Hi there,

The problem that we are facing is SD card corruption, specifically with the Raspbian operating system. We are using the Raspberry Pis (3B+) in environments where they may lose power at any moment. We need to use them headless and have no formal process to switch them off, we just pull the power.

We knew this was a problem and likely to be causing the corruption so introduced a read-only root file system within Raspbian. This is something I found online and we tested it as thoroughly as we could:

[removed bad link]

However, after several months of no corruption related issues, recently we have had several units fail. Despite protecting the root file system, this hasn’t been enough, so I believe we need to look at protecting the rest of the partitions (boot etc).

It also appears to be a time related issue. If a Pi is restarted in a short period of time, it will normally recover and restart. If one has been running for a long period of time and it then power cycled, there's a very slim chance (almost none) that it will reboot. For me, this points at some sort of internal logging building up over time. An issue which I thought would have been removed with a read-only root file system.

I am not a programmer by trade, I’m a product designer and have done what I can but have unfortunately reached the end of my knowledge. We’re looking for an expert in Linux and Raspbian to help us build a robust, incorruptible file system. The world seems to be full of examples of people running Raspberry Pis for years but we have struggled to get ours to run consistently.

I appreciate this is slightly unorthodox but I would really appreciate any help you can provide.

Many thanks,

James

Re: Persistent SD Card Corruption

Posted: Tue Nov 05, 2019 3:26 pm
by jbudd
The URL you posted tries to download a file. No thanks!

Re: Persistent SD Card Corruption

Posted: Tue Nov 05, 2019 3:41 pm
by LONGJA
Sorry! Been using it in a batch script. I meant this website...

http://wiki.psuter.ch/doku.php?id=solve ... _partition

Re: Persistent SD Card Corruption

Posted: Tue Nov 05, 2019 4:08 pm
by jamesh
You should ensure all logging goes to a tmp filesystem - ie RAM, which means very few SD card accesses. Its possible the devices are actually filling up their SD cards, which does reduce their lifetime (less free space means less chance for wear levelling).

Re: Persistent SD Card Corruption

Posted: Tue Nov 05, 2019 4:17 pm
by rpiMike
Why not add a UPS hat that will do a clean shutdown on power loss?

https://uk.pi-supply.com/products/pijuice-standard

Re: Persistent SD Card Corruption

Posted: Tue Nov 05, 2019 4:58 pm
by LONGJA
Unfortunately, I can't use a hat as we've built our own and therefore no longer have access to the GPIO header.

I've really looking for a RAM solution - @jamesh - the read only solution we're using at the moment claims to deliver a 'read only root partition'. I assume this therefore does not cover system logging. Is there a list of directories that we need to point to a tmp filesystem?

Ideally I would like to run the whole Pi in RAM and never write to the SD card.

Re: Persistent SD Card Corruption

Posted: Tue Nov 05, 2019 6:22 pm
by rpdom
There's an option in raspi-config in Raspbian now to use a read-only root with all the log files and things moved to RAM disks.

Re: Persistent SD Card Corruption

Posted: Tue Nov 05, 2019 6:59 pm
by Heater
That's cool. Missed the memo on the read-only root option in raspi-config. Just as the need of doing it again was coming around for me.

Thanks for the heads up.

Re: Persistent SD Card Corruption

Posted: Wed Nov 06, 2019 9:10 am
by LONGJA
rpdom - that's great that there's now an option in Raspbain. Do you have any idea (or could you point me to) how this deals with the boot and other partitions?

Re: Persistent SD Card Corruption

Posted: Wed Nov 06, 2019 9:43 am
by rpdom
Have a look at what raspi-config does. It is just a bash script.

Re: Persistent SD Card Corruption

Posted: Wed Nov 06, 2019 10:36 am
by Heater
I would expect nothing more.

Re: Persistent SD Card Corruption

Posted: Wed Nov 06, 2019 5:33 pm
by LONGJA
rpdom - apologies if I'm being thick. Have had a look through raspi-config and I can't find anything that looks like 'read-only-root'.

Please could you point me in a right direction?

I'm using a 3B+, will this have any effect?

Re: Persistent SD Card Corruption

Posted: Wed Nov 06, 2019 5:43 pm
by rpdom
LONGJA wrote:
Wed Nov 06, 2019 5:33 pm
rpdom - apologies if I'm being thick. Have had a look through raspi-config and I can't find anything that looks like 'read-only-root'.

Please could you point me in a right direction?

I'm using a 3B+, will this have any effect?
The bit that looks like it is doing the work starts with

Code: Select all

enable_overlayfs() {
If you don't have that then you are not running the latest raspi-config.

Re: Persistent SD Card Corruption

Posted: Wed Nov 06, 2019 5:47 pm
by hommar
https://www.raspberrypi.org/forums/view ... 9&t=212604
Paul Hutch wrote:
Tue May 15, 2018 1:41 pm
The problem is that the power can be cut while this is happening leaving the uSD card in a non-recoverable state..

Re: Persistent SD Card Corruption

Posted: Wed Nov 06, 2019 6:54 pm
by Heater
Yesterday I had no read-only-root option on my Pi 3B

I had to update raspi-config to get it.

Re: Persistent SD Card Corruption

Posted: Fri Nov 08, 2019 9:49 am
by LONGJA
Thanks for your responses, I've upgraded and now have access to the read-only functionality. Now to start testing...

Re: Persistent SD Card Corruption

Posted: Mon Nov 11, 2019 8:55 pm
by rln
LONGJA wrote:
Tue Nov 05, 2019 4:58 pm
Ideally I would like to run the whole Pi in RAM and never write to the SD card.
Have a look at my project Nard SDK which is exactly what you're looking for. A RAM only embedded Linux distro.
http://www.nard.se/