LONGJA
Posts: 23
Joined: Wed Mar 28, 2018 4:04 pm

Persistent SD Card Corruption

Tue Nov 05, 2019 11:49 am

Hi there,

The problem that we are facing is SD card corruption, specifically with the Raspbian operating system. We are using the Raspberry Pis (3B+) in environments where they may lose power at any moment. We need to use them headless and have no formal process to switch them off, we just pull the power.

We knew this was a problem and likely to be causing the corruption so introduced a read-only root file system within Raspbian. This is something I found online and we tested it as thoroughly as we could:

[removed bad link]

However, after several months of no corruption related issues, recently we have had several units fail. Despite protecting the root file system, this hasn’t been enough, so I believe we need to look at protecting the rest of the partitions (boot etc).

It also appears to be a time related issue. If a Pi is restarted in a short period of time, it will normally recover and restart. If one has been running for a long period of time and it then power cycled, there's a very slim chance (almost none) that it will reboot. For me, this points at some sort of internal logging building up over time. An issue which I thought would have been removed with a read-only root file system.

I am not a programmer by trade, I’m a product designer and have done what I can but have unfortunately reached the end of my knowledge. We’re looking for an expert in Linux and Raspbian to help us build a robust, incorruptible file system. The world seems to be full of examples of people running Raspberry Pis for years but we have struggled to get ours to run consistently.

I appreciate this is slightly unorthodox but I would really appreciate any help you can provide.

Many thanks,

James

jbudd
Posts: 1266
Joined: Mon Dec 16, 2013 10:23 am

Re: Persistent SD Card Corruption

Tue Nov 05, 2019 3:26 pm

The URL you posted tries to download a file. No thanks!

LONGJA
Posts: 23
Joined: Wed Mar 28, 2018 4:04 pm

Re: Persistent SD Card Corruption

Tue Nov 05, 2019 3:41 pm

Sorry! Been using it in a batch script. I meant this website...

http://wiki.psuter.ch/doku.php?id=solve ... _partition

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 26015
Joined: Sat Jul 30, 2011 7:41 pm

Re: Persistent SD Card Corruption

Tue Nov 05, 2019 4:08 pm

You should ensure all logging goes to a tmp filesystem - ie RAM, which means very few SD card accesses. Its possible the devices are actually filling up their SD cards, which does reduce their lifetime (less free space means less chance for wear levelling).
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“My wife said to me `...you’re not even listening`.
I thought, that’s an odd way to start a conversation.."

User avatar
rpiMike
Posts: 1206
Joined: Fri Aug 10, 2012 12:38 pm
Location: Cumbria, UK

Re: Persistent SD Card Corruption

Tue Nov 05, 2019 4:17 pm

Why not add a UPS hat that will do a clean shutdown on power loss?

https://uk.pi-supply.com/products/pijuice-standard

LONGJA
Posts: 23
Joined: Wed Mar 28, 2018 4:04 pm

Re: Persistent SD Card Corruption

Tue Nov 05, 2019 4:58 pm

Unfortunately, I can't use a hat as we've built our own and therefore no longer have access to the GPIO header.

I've really looking for a RAM solution - @jamesh - the read only solution we're using at the moment claims to deliver a 'read only root partition'. I assume this therefore does not cover system logging. Is there a list of directories that we need to point to a tmp filesystem?

Ideally I would like to run the whole Pi in RAM and never write to the SD card.

User avatar
rpdom
Posts: 16744
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: Persistent SD Card Corruption

Tue Nov 05, 2019 6:22 pm

There's an option in raspi-config in Raspbian now to use a read-only root with all the log files and things moved to RAM disks.
Unreadable squiggle

Heater
Posts: 15438
Joined: Tue Jul 17, 2012 3:02 pm

Re: Persistent SD Card Corruption

Tue Nov 05, 2019 6:59 pm

That's cool. Missed the memo on the read-only root option in raspi-config. Just as the need of doing it again was coming around for me.

Thanks for the heads up.
Memory in C++ is a leaky abstraction .

LONGJA
Posts: 23
Joined: Wed Mar 28, 2018 4:04 pm

Re: Persistent SD Card Corruption

Wed Nov 06, 2019 9:10 am

rpdom - that's great that there's now an option in Raspbain. Do you have any idea (or could you point me to) how this deals with the boot and other partitions?

User avatar
rpdom
Posts: 16744
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: Persistent SD Card Corruption

Wed Nov 06, 2019 9:43 am

Have a look at what raspi-config does. It is just a bash script.
Unreadable squiggle

Heater
Posts: 15438
Joined: Tue Jul 17, 2012 3:02 pm

Re: Persistent SD Card Corruption

Wed Nov 06, 2019 10:36 am

I would expect nothing more.
Memory in C++ is a leaky abstraction .

LONGJA
Posts: 23
Joined: Wed Mar 28, 2018 4:04 pm

Re: Persistent SD Card Corruption

Wed Nov 06, 2019 5:33 pm

rpdom - apologies if I'm being thick. Have had a look through raspi-config and I can't find anything that looks like 'read-only-root'.

Please could you point me in a right direction?

I'm using a 3B+, will this have any effect?

User avatar
rpdom
Posts: 16744
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: Persistent SD Card Corruption

Wed Nov 06, 2019 5:43 pm

LONGJA wrote:
Wed Nov 06, 2019 5:33 pm
rpdom - apologies if I'm being thick. Have had a look through raspi-config and I can't find anything that looks like 'read-only-root'.

Please could you point me in a right direction?

I'm using a 3B+, will this have any effect?
The bit that looks like it is doing the work starts with

Code: Select all

enable_overlayfs() {
If you don't have that then you are not running the latest raspi-config.
Unreadable squiggle

hommar
Posts: 191
Joined: Sat Mar 25, 2017 1:55 pm
Location: Russia, Yekaterinburg

Re: Persistent SD Card Corruption

Wed Nov 06, 2019 5:47 pm

https://www.raspberrypi.org/forums/view ... 9&t=212604
Paul Hutch wrote:
Tue May 15, 2018 1:41 pm
The problem is that the power can be cut while this is happening leaving the uSD card in a non-recoverable state..

Heater
Posts: 15438
Joined: Tue Jul 17, 2012 3:02 pm

Re: Persistent SD Card Corruption

Wed Nov 06, 2019 6:54 pm

Yesterday I had no read-only-root option on my Pi 3B

I had to update raspi-config to get it.
Memory in C++ is a leaky abstraction .

LONGJA
Posts: 23
Joined: Wed Mar 28, 2018 4:04 pm

Re: Persistent SD Card Corruption

Fri Nov 08, 2019 9:49 am

Thanks for your responses, I've upgraded and now have access to the read-only functionality. Now to start testing...

rln
Posts: 176
Joined: Wed Apr 09, 2014 1:43 pm
Location: Sweden
Contact: Website

Re: Persistent SD Card Corruption

Mon Nov 11, 2019 8:55 pm

LONGJA wrote:
Tue Nov 05, 2019 4:58 pm
Ideally I would like to run the whole Pi in RAM and never write to the SD card.
Have a look at my project Nard SDK which is exactly what you're looking for. A RAM only embedded Linux distro.
http://www.nard.se/
Author of the robust Nard distro http://www.nard.se

Return to “General discussion”