Leeloo
Posts: 45
Joined: Fri Oct 06, 2017 9:53 pm

unplug anytime ?

Wed Aug 21, 2019 4:06 am

hi,

wondering if its possible to setp a pi/linux system in a way that it can be safely/randomly unpowered at (almost) any time ?

the main idea is I want to run form an inexpensive powerbank which works fine from testing but don't really have any battery indicator (though i do know it runs for a few hours) ideally would be great to run it for an hour here and there without really worrying that if I let the power get low my sd card will be irreparably damaged and/or corrupted.

I was also sort of inspired by the old 90s era of computing, dos / amiga os neither of which had a "shut down" they didn't have constant disc io like modern systems and were intended to be switched off when no longer needed using a hard switch.

is it possible to setup this way ?

Thanks

Heater
Posts: 13064
Joined: Tue Jul 17, 2012 3:02 pm

Re: unplug anytime ?

Wed Aug 21, 2019 5:01 am

Yes it is. I have done it. It works very well.

What you need to do is set up your root file system as read only. A search should find you many discussions about doing this on this forum. There are many ways to approach it.

It might be a problem for you that with a read-only root you will have to find another place to store any locally produced data if you need to. Perhaps a USB stick or such.

It can also make applying updates to your operating system harder.

User avatar
rpdom
Posts: 14984
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: unplug anytime ?

Wed Aug 21, 2019 5:03 am

The most common way to set up a system like that is to have the main file system as read only. That means there are no updates to the card, so there is nothing that can get corrupted if you lose the power. Search "Read only root".

Any files that do get written to are stored in a RAM based file system, which gets lost when power goes.

This is fine as long as you don't want to save anything ever. You could have a separate file system for data. With less stuff happening on it, it is less likely to get badly corrupted if power is lost - but there is still the possibility of losing things.

I have had to pull power from my Raspis on occasions, and have had a couple of power outages in the last 7 years. I've never lost anything important on those occasions and not had an SD card damaged beyond simple recovery (fsck) so far. But I wouldn't recommend pulling power on a running system.

With those old DOS systems it was fine to pull the power when sitting at a command prompt (apart from one version of, I think it was, MSDOS 6 which used a disk cache to speed things up and didn't flush the write cache before returning to a command prompt - that got changed fairly quickly when people started getting corrupted filesystems and losing work), as nothing was doing anything in those single-user single task machines.

User avatar
Gavinmc42
Posts: 3605
Joined: Wed Aug 28, 2013 3:31 am

Re: unplug anytime ?

Wed Aug 21, 2019 7:18 am

I use PiCore Linux for apps like this, it just works without messing about with read only etc.
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

Leeloo
Posts: 45
Joined: Fri Oct 06, 2017 9:53 pm

Re: unplug anytime ?

Wed Aug 21, 2019 5:33 pm

rpdom wrote:
Wed Aug 21, 2019 5:03 am
The most common way to set up a system like that is to have the main file system as read only.
so if i partition a 16gb card say something like a 6gb root as read only then 10gb fat32 for read/write storage (due it it being a pretty fast fs without logging) would that mean it is safe to power down at any time so long as I am not writing files to the fat32 partition? because that would be perfect!

will look into the other suggestions too, one more concern, I would have to run without a swap right? not sure if that is a good idea but will figure it out.

Thanks

trejan
Posts: 510
Joined: Tue Jul 02, 2019 2:28 pm

Re: unplug anytime ?

Wed Aug 21, 2019 5:44 pm

Leeloo wrote:
Wed Aug 21, 2019 5:33 pm
so if i partition a 16gb card say something like a 6gb root as read only then 10gb fat32 for read/write storage (due it it being a pretty fast fs without logging) would that mean it is safe to power down at any time so long as I am not writing files to the fat32 partition?
There is still a risk of corruption to the other partition. It is much lower than if you were writing to the other partition but it isn't zero. Flash memory needs to be erased in blocks which are far bigger than the pages the OS is using so the controller inside will be combining multiple pages together into one big block. The wear leveling algorithms mean data moves around and won't always be sequential on the underlying flash. The end result is that if you interrupt a block write at the wrong time then you'll corrupt multiple pages inside and they may be pages from your read only partition. Read https://en.wikipedia.org/wiki/Write_amplification for better explanation.

I've had this happen to me once. Somebody disconnected power whilst it was busy writing and it wouldn't start up again. Comparing it with a recent backup of the card showed a few pages in /boot, the partition table and the file that was being written were corrupted even though nothing was altering it. Everything else was fine. Rewriting the card with the backup and it has been working without any problems since then.

If you want to be safe then put your read/write partition onto a USB drive and keep the entire micro SD card read only.

drgeoff
Posts: 9729
Joined: Wed Jan 25, 2012 6:39 pm

Re: unplug anytime ?

Wed Aug 21, 2019 6:43 pm

trejan wrote:
Wed Aug 21, 2019 5:44 pm
If you want to be safe then put your read/write partition onto a USB drive and keep the entire micro SD card read only.
It isn't immediately obvious to me why a USB flash drive would be less susceptible than the micro-SD card.

trejan
Posts: 510
Joined: Tue Jul 02, 2019 2:28 pm

Re: unplug anytime ?

Wed Aug 21, 2019 6:48 pm

drgeoff wrote:
Wed Aug 21, 2019 6:43 pm
It isn't immediately obvious to me why a USB flash drive would be less susceptible than the micro-SD card.
It is separate so can't affect the SD card.

Heater
Posts: 13064
Joined: Tue Jul 17, 2012 3:02 pm

Re: unplug anytime ?

Wed Aug 21, 2019 6:59 pm

The idea is to protect whatever media it is one is booting from, say the SD card.

Then you at least have a system that will boot and work with.

Data on another media, like USB stick, may well get damaged.

drgeoff
Posts: 9729
Joined: Wed Jan 25, 2012 6:39 pm

Re: unplug anytime ?

Wed Aug 21, 2019 7:07 pm

Heater wrote:
Wed Aug 21, 2019 6:59 pm
The idea is to protect whatever media it is one is booting from, say the SD card.

Then you at least have a system that will boot and work with.

Data on another media, like USB stick, may well get damaged.
To me that appears to be the inverse of safety. If the boot files are standard they can be replaced easily. Damaged user data might be either irreplaceable or only replaceable with much time and effort.

trejan
Posts: 510
Joined: Tue Jul 02, 2019 2:28 pm

Re: unplug anytime ?

Wed Aug 21, 2019 7:20 pm

drgeoff wrote:
Wed Aug 21, 2019 7:07 pm
To me that appears to be the inverse of safety. If the boot files are standard they can be replaced easily. Damaged user data might be either irreplaceable or only replaceable with much time and effort.
What are you even trying to argue here?

The user data isn't protected in either situation but having a separate USB drive would mean your OS + configuration + app will be okay even if something bad did happen to the USB drive. If you can't afford to lose any user data at all then procedures needs to be changed so a clean shutdown always happens. If you don't mind the risk or want to minimise size/cost then just do it from the micro SD with a read only OS partition and read/write data partition.

bjtheone
Posts: 129
Joined: Mon May 20, 2019 11:28 pm

Re: unplug anytime ?

Wed Aug 21, 2019 7:58 pm

drgeoff wrote:
Wed Aug 21, 2019 7:07 pm
Heater wrote:
Wed Aug 21, 2019 6:59 pm
The idea is to protect whatever media it is one is booting from, say the SD card.

Then you at least have a system that will boot and work with.

Data on another media, like USB stick, may well get damaged.
To me that appears to be the inverse of safety. If the boot files are standard they can be replaced easily. Damaged user data might be either irreplaceable or only replaceable with much time and effort.
The difference is that you can make the OS read only and still have a functional system. Making the user data read only kinda defeats the purpose in most use cases. You can solve it somewhat by switching to a journaled file system on a real hard drive for the user data. On an SD you will be at risk of both unflushed writes and wear leveling. The other aspect is that the system will boot as long as the OS in not corrupted.

You are making backups of user data you care about, right? No real point in backing up OS files unless you have a funky hard to recreate setup.

hippy
Posts: 5757
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: unplug anytime ?

Wed Aug 21, 2019 8:00 pm

trejan wrote:
Wed Aug 21, 2019 7:20 pm
drgeoff wrote:
Wed Aug 21, 2019 7:07 pm
To me that appears to be the inverse of safety. If the boot files are standard they can be replaced easily. Damaged user data might be either irreplaceable or only replaceable with much time and effort.
What are you even trying to argue here?
That data is more important than the OS. I would agree with that sentiment.

You can fix a broken OS, replace a broken computer, use a different computer, but lost data cannot be replaced unless it is backed-up somewhere else.

Of course if data is unimportant, transient, losing it doesn't matter so much.

Consider a Pi-based home webcam surveillance system, which recorded a miscreant who broke into your property before the pulled the power on your Pi. Would you prefer -

The system boots but the video recording is unreadable and unusable, or
The system doesn't boot but the video recording is intact ?
Last edited by hippy on Wed Aug 21, 2019 8:07 pm, edited 1 time in total.

trejan
Posts: 510
Joined: Tue Jul 02, 2019 2:28 pm

Re: unplug anytime ?

Wed Aug 21, 2019 8:07 pm

hippy wrote:
Wed Aug 21, 2019 8:00 pm
That data is more important than the OS. I would agree with that sentiment.
Sure but protecting the OS by separating it onto separate storage isn't detrimental to the user data. "To me that appears to be the inverse of safety." makes it out that doing this is increasing your risk.

drgeoff
Posts: 9729
Joined: Wed Jan 25, 2012 6:39 pm

Re: unplug anytime ?

Wed Aug 21, 2019 9:49 pm

trejan wrote:
Wed Aug 21, 2019 7:20 pm
drgeoff wrote:
Wed Aug 21, 2019 7:07 pm
To me that appears to be the inverse of safety. If the boot files are standard they can be replaced easily. Damaged user data might be either irreplaceable or only replaceable with much time and effort.
What are you even trying to argue here?
Not arguing anything. Merely concerned that the statement "If you want to be safe then put your read/write partition onto a USB drive and keep the entire micro SD card read only." may be leaving the OP with a dangerous false sense of security.

bjtheone
Posts: 129
Joined: Mon May 20, 2019 11:28 pm

Re: unplug anytime ?

Thu Aug 22, 2019 1:06 am

If you really care about the data, then an ext4 file system on an SD card is so not the way to go. If you really want it secure, you want some kind of redundant solution, ideally with an offsite copy/mirror/backup.

tqhien
Posts: 33
Joined: Thu Feb 02, 2012 10:07 am

Re: unplug anytime ?

Fri Aug 23, 2019 11:40 am

Hello,

For one of my project (embedded rpi on motorbike), I used the following configuration on a single SDCard :
- boot partition (fat)
- root file system read-only (ext2 or ext4)
- 2 btrfs partitions for user data, in raid mode, with 5 seconds cache (data are written to disk every 5 seconds, mirrorring is provided by btrfs). Those partitions are mounted in a directory where I can save user data.

My app doesn't use a graphical environnement : I directly write to the framebuffer.
That config goes well for me and my app.

Hien.

bjtheone
Posts: 129
Joined: Mon May 20, 2019 11:28 pm

Re: unplug anytime ?

Fri Aug 23, 2019 12:14 pm

tqhien wrote:
Fri Aug 23, 2019 11:40 am
Hello,

For one of my project (embedded rpi on motorbike), I used the following configuration on a single SDCard :
- boot partition (fat)
- root file system read-only (ext2 or ext4)
- 2 btrfs partitions for user data, in raid mode, with 5 seconds cache (data are written to disk every 5 seconds, mirrorring is provided by btrfs). Those partitions are mounted in a directory where I can save user data.

My app doesn't use a graphical environnement : I directly write to the framebuffer.
That config goes well for me and my app.

Hien.
As I understand wear leveling and SD cards, partitioning does not protect you. The wear leveling and actual writing operations occur at a low level with the partitioning being a virtual layer on top of this. The OS sees the partitioned view of the storage and behaves as if they were physical partitions. On a physical hard drive, partitions define physical portions of the disk and all writing is constrained by said partitions. The end result is that the SD will happily wear level across the entire SD thus any portion of the entire SD is at risk due to corruption if you power off mid write. If you moved your user data to a USB drive your solution would work. Please correct me if I am wrong about this.

Of course making the root fs read-only does reduce the volume of writes to the SD, which reduces the amount of wear leveling which reduces the likelihood of corruption, but it is not eliminated.

jmmec
Posts: 25
Joined: Thu Dec 31, 2015 11:13 pm

Re: unplug anytime ?

Fri Aug 23, 2019 1:14 pm

Anyone have any practical experience with the SanDisk Industrial SD cards with "power protection"?
In addition, the new SanDisk Automotive and Industrial SD cards support Power Protection capability, which prevents data from being corrupted during a write when the power supply is unstable or unexpectedly lost, which is a feature found on several industrial and enterprise SSDs as well, and is required due to the way in which flash memory is written (see here for a quick refresher)

https://www.anandtech.com/show/11889/sa ... eliability

Aydan
Posts: 688
Joined: Fri Apr 13, 2012 11:48 am
Location: Germany, near Lake Constance

Re: unplug anytime ?

Fri Aug 23, 2019 1:21 pm

Wear leveling on SD cards is normally of the dynamic kind, meaning it is only done when a block is written to, not like SSDs which will also use blocks that aren't written to for wear leveling and do wear leveling in their spare time.
Meaning as long as a block is not written to it stays untouched.
If you put a "spacer" between your partitions (or align the partitions to a block boundary, which is nigh impossible, since the SD card manufacturers usually dont tell you the erase block size) which is >= the erase block size of the SD card (e.g. 32 or 64MB) then writing to a block of one partiton will not affect a block from another partition.

Regards
Aydan

tqhien
Posts: 33
Joined: Thu Feb 02, 2012 10:07 am

Re: unplug anytime ?

Fri Aug 23, 2019 1:29 pm

Well, SDCard or USBpendrives are the same and subject to wear levelling. Only USB HDD drive would be safe. But on an embedded computer as mine, HDD USB is not possible.

Wear levelling is concerned for writing blocks. And when I create a partition, write to it, I only change a few block on it. And for that, ext2 uses less block as ext4 is journaling every writes...

BTRFS is a copy on write (cow) filesystem. That's mean there's implicit sharing (or shadowing) mecanisms, with the concept of snapshot : on change, the original data is not modified, only the new data is stored (in a new block so no wear levelling of the original block !). And with raid, both partitions get their copy. If power is loss, on startup, if one is more up to date than the other, a repair process is automatically done.

Hien.

Aydan
Posts: 688
Joined: Fri Apr 13, 2012 11:48 am
Location: Germany, near Lake Constance

Re: unplug anytime ?

Fri Aug 23, 2019 2:19 pm

What I said about erase block size still applies to BTRFS. If your BTRFS blocks are smaller than the erase block size (very likely), then the old block and the new block could be located in the same eraseblock, and be both corrupted by a power failure.

Regards
Aydan

Return to “General discussion”