dptdpt
Posts: 6
Joined: Sat Jun 27, 2020 11:06 pm

My pi3 keeps getting corrupted

Mon Jun 29, 2020 10:14 pm

I'm working on a project on a Pi3. It seems like at least once a week my image becomes corrupted and I have to do a rebuild. Various unrelated commands will stop working. I typically spend an hour or two trying to repair the image and then bite the bullet and create a new image and spend another couple of hours installing software and configuring. This is getting old. Two questions:

* Any suggestions as to how I can figure out why it keeps getting corrupted? I suspect it's due to the frequent reboots I do, is that not safe? Isn't the file system on raspbian capable of handling reboots without a clean shutdown?

* Is there any recommended way I can make a copy of my image to reuse in the future, so I don't have to rebuild everything? I found something online that I tried, but the image I created was corrupt. I forget what I did. I saw the sticky on this site about setting up an image, but it sounded like that wouldn't work if you needed to include a lot of software in your image, which I do.

W. H. Heydt
Posts: 12990
Joined: Fri Mar 09, 2012 7:36 pm
Location: Vallejo, CA (US)

Re: My pi3 keeps getting corrupted

Mon Jun 29, 2020 10:22 pm

When you have a good image, with all your system changes and added software on it...take a backup of it. (This is, by the way, a good practice even if you're not having problems.)

As for the corrupted images.. How are you shutting you Pi down? Do you have a *good* PSU? What is it rated to deliver? What--in general--are you doing as part of your project? Any added hardware? If so, how is it connected? What protections are you doing to prevent excessive current draw? Are you running anything as root? If so, why and do you know how to avoid doing damage to system files?

trejan
Posts: 2332
Joined: Tue Jul 02, 2019 2:28 pm

Re: My pi3 keeps getting corrupted

Mon Jun 29, 2020 10:23 pm

dptdpt wrote:
Mon Jun 29, 2020 10:14 pm
I suspect it's due to the frequent reboots I do, is that not safe? Isn't the file system on raspbian capable of handling reboots without a clean shutdown?
Rebooting with a clean shutdown is fine. If you're just unplugging it then that is bad and you will corrupt the SD card itself or the filesystem.
dptdpt wrote:
Mon Jun 29, 2020 10:14 pm
* Is there any recommended way I can make a copy of my image to reuse in the future, so I don't have to rebuild everything?
The "SD Card Copier" utility if you're using the desktop version will do it. Otherwise use something like Win32DiskImager to create a backup. The backup will be the size of your card though so you'll need a lot of spare disk space to store it.

User avatar
dickon
Posts: 1657
Joined: Sun Dec 09, 2012 3:54 pm
Location: Home, just outside Reading

Re: My pi3 keeps getting corrupted

Mon Jun 29, 2020 10:25 pm

If possible, network boot. Saves a hell of a lot of hassle.

dptdpt
Posts: 6
Joined: Sat Jun 27, 2020 11:06 pm

Re: My pi3 keeps getting corrupted

Wed Jul 01, 2020 5:08 pm

Thanks for the responses.

So yes, I am sometimes doing a hard reboot - cycling the power. I am running with a headless Pi, and sometimes it gets to a state where I can't ssh into it. Bad me. But there is such a thing as a filesystem that does not become corrupted even if you lose power in the middle of a write. So the pi actually doesn't have such a file system? Bad pi.

Regarding taking a backup - is there nothing that I can just back up the Pi filesystem, that only needs as much space as I'm using? Backing up and restoring the entire partition sounds time consuming and would use a lot of space. I'm looking for something I could use on a mac or PC, not on the pi itself...

Thanks again.

- Dave

trejan
Posts: 2332
Joined: Tue Jul 02, 2019 2:28 pm

Re: My pi3 keeps getting corrupted

Wed Jul 01, 2020 5:18 pm

dptdpt wrote:
Wed Jul 01, 2020 5:08 pm
But there is such a thing as a filesystem that does not become corrupted even if you lose power in the middle of a write. So the pi actually doesn't have such a file system? Bad pi.
The Pi isn't doing anything wrong here. The Pi does use a journaled filesystem but it defaults to protecting metadata only. You can tell it to journal metadata + data but there is a performance penalty for doing that which is why Linux distributions don't have it enabled by default. Enabling it will ensure your filesystem is still consistent for data + metadata but you'll still lose any data that was in the cache and not written out yet. There is no 100% solution for this. You're just meant to not pull the plug without shutting down first.

You're still at the mercy of the cache and wear leveling inside the SD card though where interrupting it may cause corruption of the internal card metadata that will kill the card itself. The only fix for this is to never abruptly disconnect power whilst it is still busy.
dptdpt wrote:
Wed Jul 01, 2020 5:08 pm
Regarding taking a backup - is there nothing that I can just back up the Pi filesystem, that only needs as much space as I'm using? Backing up and restoring the entire partition sounds time consuming and would use a lot of space. I'm looking for something I could use on a mac or PC, not on the pi itself...
If you have something that understands Ext4 like a Linux VM then you can mount the partition then backup the files only using tar, rsync or whatever your favourite backup tool is.

dptdpt
Posts: 6
Joined: Sat Jun 27, 2020 11:06 pm

Re: My pi3 keeps getting corrupted

Wed Jul 01, 2020 5:39 pm

Thanks. Maybe I'll look into journaling of the whole FS. But I don't think the pi is in the middle of writing anything when I do my reboots. It should be idle... How much of a performance hit would it be, roughly?

Regarding backing up, isn't this a common question? Is there a writeup anywhere about how to do this? I'd think it would be a sticky somewhere on this site...

W. H. Heydt
Posts: 12990
Joined: Fri Mar 09, 2012 7:36 pm
Location: Vallejo, CA (US)

Re: My pi3 keeps getting corrupted

Wed Jul 01, 2020 6:20 pm

dptdpt wrote:
Wed Jul 01, 2020 5:08 pm
But there is such a thing as a filesystem that does not become corrupted even if you lose power in the middle of a write. So the pi actually doesn't have such a file system? Bad pi.
No, there isn't, at that applies to *all* computers, not just Pis. What does exist are OSes that do very little writing by using a read-only root filesystem and running everything in RAM. But note that you're eventually going to have to save something somewhere or what's the point? That said, the data that gets saved doesn't have to get saved to the SD card. It could be saved to another device (such as a USB stick), or over a network connection to another system or device (Network Attached Storage or shared directory). But any time a system crashes or is taken down, anything left in caches will be lost and that may include directory structure data as well as applications or program data.

More to the point would probably be to figure out what you're doing that locks up the system and stop doing that. In the mean time, as the old unix saw has it: Save early and often. If you *have* to pull the power, wait long enough to any cache data to be written. IIRC, sync usually runs about every 30 seconds (and is configurable), so wait *at* *least* that long before pulling the power.

W. H. Heydt
Posts: 12990
Joined: Fri Mar 09, 2012 7:36 pm
Location: Vallejo, CA (US)

Re: My pi3 keeps getting corrupted

Wed Jul 01, 2020 6:27 pm

dptdpt wrote:
Wed Jul 01, 2020 5:08 pm
Regarding taking a backup - is there nothing that I can just back up the Pi filesystem, that only needs as much space as I'm using? Backing up and restoring the entire partition sounds time consuming and would use a lot of space. I'm looking for something I could use on a mac or PC, not on the pi itself...
I use Win32DiskImager (which I also use to build SD cards in the first place). Yes, the file is the size of the card, and it takes a while. However, since I'm taking that backup to a 2TB HDD, space isn't an issue. And if space becomes and issue, I can always compress the files.

On systems where I have specific backup requirements because they are "production" systems (that is, the server for a convention registration system), I burn backups of the directory trees I care about to a CD periodically. "Periodically" depending on where I am during the convention cycle. During pre-reg, every couple of weeks. At the convention, daily. However, that system is running from a 120GB SSD and imaging that would be a trifle awkward.

User avatar
dickon
Posts: 1657
Joined: Sun Dec 09, 2012 3:54 pm
Location: Home, just outside Reading

Re: My pi3 keeps getting corrupted

Wed Jul 01, 2020 6:52 pm

W. H. Heydt wrote:
Wed Jul 01, 2020 6:20 pm
dptdpt wrote:
Wed Jul 01, 2020 5:08 pm
But there is such a thing as a filesystem that does not become corrupted even if you lose power in the middle of a write. So the pi actually doesn't have such a file system? Bad pi.
No, there isn't, at that applies to *all* computers, not just Pis.
Umm, that isn't true. Any copy-on-write filesystem -- such as ZFS or ButterFS -- will be consistent on disc at all times, although of course you may lose data that hasn't been completely committed to storage before the event (partially-commited data, such as that written to a journal but not written to its final resting place may be recovered on the next mount; that depends on the filesystem).

This assertion relies on storage systems behaving as advertised, and not pulling a fast one, of course.

There is a performance hit -- quite significant at times -- to this, of course, so such filesystems aren't for everything. I use ZFS on everything because I happen to like the featureset and the data reliability guarantees, but I don't run it on Pis. I run NFS on Pis, which keep their systems and data on NFS-exported ZFS filesystems, of course. It's why I'm a bit of a fan of eatmydata.

RonR
Posts: 1420
Joined: Tue Apr 12, 2016 10:29 pm
Location: US

Re: My pi3 keeps getting corrupted

Wed Jul 01, 2020 6:58 pm

dptdpt wrote:
Wed Jul 01, 2020 5:08 pm
Regarding taking a backup - is there nothing that I can just back up the Pi filesystem, that only needs as much space as I'm using?

image-backup

Heater
Posts: 16318
Joined: Tue Jul 17, 2012 3:02 pm

Re: My pi3 keeps getting corrupted

Wed Jul 01, 2020 7:08 pm

Oddly enough for some time now I have been very cavalier about shutdown. I just yank the power whenever. In the last few years no SD corruption has happened.

It's odd because back in the days of the first Pi it seemed SD cards would become corrupt if you looked at them the wrong way. Either the file system getting corrupted, which is at least recoverable, or the SD card becoming unusable.

I always wonder if something changed in the OS or the hardware or are modern SD cards just more robust.

Anyway, if you want reliability for a system subject to random power downs then configure the root fs as read-only. See raspi-config. Store your data on a different media or off the Pi some place over the network.
Memory in C++ is a leaky abstraction .

jahboater
Posts: 5944
Joined: Wed Feb 04, 2015 6:38 pm
Location: West Dorset

Re: My pi3 keeps getting corrupted

Wed Jul 01, 2020 7:18 pm

dptdpt wrote:
Wed Jul 01, 2020 5:39 pm
But I don't think the pi is in the middle of writing anything when I do my reboots. It should be idle...
Doesn't matter. By default the system flushes the disk cache every five seconds, in between it will do little or no writes.
So put simply, up to five seconds worth of data could be in memory waiting to be written. That data is obviously going to be lost when you yank the plug out. If you really must cut the power, do a "sync" first.

I change my ext4 filesystems to flush the disk cache every ten minutes (giving improved performance, less SD card wear, and more contiguous files). Because I always do an orderly shutdown, I never have a problem (power cuts are rare around here).

I just don't get corrupt SD cards on the Pi so I have never bothered with read-only root filesystems.
Pi4 8GB running PIOS64 Lite

pidd
Posts: 931
Joined: Fri May 29, 2020 8:29 pm
Location: Birkenhead, Wirral, UK
Contact: Website

Re: My pi3 keeps getting corrupted

Wed Jul 01, 2020 9:04 pm

Heater wrote:
Wed Jul 01, 2020 7:08 pm
I always wonder if something changed in the OS or the hardware or are modern SD cards just more robust.
Modern SDs are faster so maybe just less probability of being caught in the middle of a write.

As an aside ..... HDDs are spec'd to store sufficient power to complete any current write to disk on detecting a power drop, I can't see how SDs could have the same so they are pretty vulnerable and rely on the system to mitigate the risk.

rwcooper
Posts: 41
Joined: Wed Dec 30, 2015 9:04 pm

Re: My pi3 keeps getting corrupted

Wed Jul 01, 2020 11:19 pm

Are you reusing the same SD card after each corruption? If so, try a different SD card. I use the SD card copier to backup my SD cards, and rsync to backup my hard drives. I've never experienced corruption even after power is lost. I've certainly had raspberry Pi's lock up and couldn't ssh into them to shut them down so I just unplug the power supply and hope for the best, and so far so good. That's not to say it can't happen, it's happened to me on x86 systems a few times after losing power. One strange thing that has happened to a couple of my raspberry pi's is occasionally not recovering after losing power and having to have to cycle the power manually once after which they were fine.

Return to “General discussion”