User avatar
SlowBro
Posts: 164
Joined: Sat Feb 18, 2017 1:30 am

Re: Raspbian with Read-only Root

Tue Oct 24, 2017 11:18 am

Heater wrote:
Mon Oct 09, 2017 11:10 am
I think if you ever write anything to the SD card you are defeating the point of making this read-only root system. Even if you make a new partition to write data to. The SD card knows nothing about your partitions and file systems it only knows about blocks of data. If it's going to screw up, which they do, it might take down your read-only root as well.
The problem as I understand it is not that writing to flash risks corruption; it’s that writing to flash — while simultaneously powering off — risks corruption to anywhere on the flash. So I can see no problem writing things very seldomly such as wpa_supplicant.conf as long as you don’t have a coinciding power cut. It’s not as though you write one file and it turns into a pumpkin.

I likely will do this for config files on my IoT device. Likely I will create a new partition (or a sub directory under/boot) and write those seldomly-changed files there, sync and flush buffers, symlink them to the real location, and take the slight risk that there won’t be a power cut during the moment they are written. Everything else will be read only or stored in the cloud.

User avatar
TimG
Posts: 293
Joined: Tue Apr 03, 2012 12:15 am
Location: Switzerland

Re: Raspbian with Read-only Root

Tue Oct 24, 2017 1:38 pm

Heater wrote:
Mon Oct 09, 2017 11:10 am
I think if you ever write anything to the SD card you are defeating the point of making this read-only root system. Even if you make a new partition to write data to.

I'm not sure that's right. From https://www.embeddedarm.com/about/resou ... dded-linux:
If local data logging is required a read-write partition can be created on the same medium as the read only partition, but in this case it is understood that in the rare case of NAND corruption due to sudden power loss it is acceptable to lose data. The read/write partition should be created aligned with the allocation group size (typically 4MiB). With this setup, the worst-case scenario from a poorly timed failure is that the system will boot correctly, but the data it has been collecting recently will be corrupted and the filesystem with the read/write data may need to be recreated.

Heater
Posts: 13671
Joined: Tue Jul 17, 2012 3:02 pm

Re: Raspbian with Read-only Root

Tue Oct 24, 2017 1:44 pm

Sounds reasonable.

I have never found a specification of what goes on inside an SD card controller so it's not something I can work to or depend on.

If anyone has such a specification to share that would be great.
Last edited by Heater on Tue Oct 24, 2017 10:39 pm, edited 1 time in total.
Memory in C++ is a leaky abstraction .

User avatar
SlowBro
Posts: 164
Joined: Sat Feb 18, 2017 1:30 am

Re: Raspbian with Read-only Root

Tue Oct 24, 2017 9:59 pm

TimG wrote:
Tue Oct 24, 2017 1:38 pm
The read/write partition should be created aligned with the allocation group size (typically 4MiB).
Anyone know how to do that? Couldn’t find instructions in the link.

Heater
Posts: 13671
Joined: Tue Jul 17, 2012 3:02 pm

Re: Raspbian with Read-only Root

Tue Oct 24, 2017 10:50 pm

When you use a tool like fdisk or parted to create partitions on a block device you can specify exactly the starting and ending offsets of the partitions you create. So arranging your partitions on 4MByte, or whatever, boundaries is quite possible.

It's a clunky business so I prefer to take the easy route and put data on a different device.

https://linux.die.net/man/8/parted

https://www.tecmint.com/fdisk-commands- ... artitions/
Memory in C++ is a leaky abstraction .

User avatar
SlowBro
Posts: 164
Joined: Sat Feb 18, 2017 1:30 am

Re: Raspbian with Read-only Root

Wed Oct 25, 2017 9:22 am

Thanks. So when you say you put it on a different device you mean an external usb drive or something else?

Heater
Posts: 13671
Joined: Tue Jul 17, 2012 3:02 pm

Re: Raspbian with Read-only Root

Wed Oct 25, 2017 10:03 am

External, obviously. Could be a tiny USB memory stick. Whatever storage medium you like.
Memory in C++ is a leaky abstraction .

User avatar
SlowBro
Posts: 164
Joined: Sat Feb 18, 2017 1:30 am

Re: Raspbian with Read-only Root

Wed Oct 25, 2017 11:47 am

Heater wrote:
Wed Oct 25, 2017 10:03 am
External, obviously. Could be a tiny USB memory stick. Whatever storage medium you like.
Not obviously. The Pi can interface on its I2C, SPI, or SDIO interfaces :-) I've pondered adding extra storage via a custom cape using one of those interfaces, which is why I asked.

Heater
Posts: 13671
Joined: Tue Jul 17, 2012 3:02 pm

Re: Raspbian with Read-only Root

Wed Oct 25, 2017 11:59 am

SlowBro,
Not obviously. The Pi can interface on its I2C, SPI, or SDIO interfaces...
Sounds external to me.

For sure you can add external storage accessed via the GPIO.
Memory in C++ is a leaky abstraction .

FM81
Posts: 518
Joined: Wed Apr 17, 2013 4:33 pm

Re: Raspbian with Read-only Root

Thu Oct 26, 2017 4:44 am

TimG wrote:
Tue Oct 24, 2017 1:38 pm
Heater wrote:
Mon Oct 09, 2017 11:10 am
I think if you ever write anything to the SD card you are defeating the point of making this read-only root system. Even if you make a new partition to write data to.

I'm not sure that's right.
I'd assume Heater is right?
Nobody know's, if wear-leveling on SD-cards, you're typically using on a rasperry, is stopping at partition-borders?
My assumption is: NO, it doesn't ...
(Also tested on own practice, but this was few years ago with older RPi-firmware, never with newer again.)

Greetings, FM_81
A: What does the command 'cat /dev/urandom', can you tell me please?
B: Yeah, that's very simple: It feeds your cat with radioactive material!

Heater
Posts: 13671
Joined: Tue Jul 17, 2012 3:02 pm

Re: Raspbian with Read-only Root

Thu Oct 26, 2017 5:17 am

I have no idea if I'm right or not.

All I know is that nobody so far has presented us with any specification as to what wear-leveling actually goes on in SD cards. It may vary from manufacturer to manufacturer. It may change with different generations of cards.

We do know that SD cards get corrupted. That they write-protect themselves, in whole or in part. As yet no explanation of that is forthcoming.

Therefore I only suggest that if you want data on your SD to remain intact it's better to never write to any part of it.
Memory in C++ is a leaky abstraction .

User avatar
rpdom
Posts: 15418
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: Raspbian with Read-only Root

Thu Oct 26, 2017 5:41 am

Heater wrote:
Thu Oct 26, 2017 5:17 am
Therefore I only suggest that if you want data on your SD to remain intact it's better to never write to any part of it.
If you never write to it, how is the data going to get there in the first place? :lol:

Heater
Posts: 13671
Joined: Tue Jul 17, 2012 3:02 pm

Re: Raspbian with Read-only Root

Thu Oct 26, 2017 9:20 am

rpdom,
If you never write to it, how is the data going to get there in the first place?
I'm going to get somebody else to write it there. :)
Memory in C++ is a leaky abstraction .

User avatar
jojopi
Posts: 3085
Joined: Tue Oct 11, 2011 8:38 pm

Re: Raspbian with Read-only Root

Fri Oct 27, 2017 1:19 am

FM81 wrote:
Thu Oct 26, 2017 4:44 am
Nobody know's, if wear-leveling on SD-cards, you're typically using on a rasperry, is stopping at partition-borders?
My assumption is: NO, it doesn't ...
If the wear levelling did stay within partitions, you would be able to destroy a card prematurely by writing to a small partition such as /boot. It is actually a worse implementation than simply levelling over the whole card.

Heater
Posts: 13671
Joined: Tue Jul 17, 2012 3:02 pm

Re: Raspbian with Read-only Root

Fri Oct 27, 2017 3:49 am

jojppi,
If the wear leveling did stay within partitions, you would be able to destroy a card prematurely by writing to a small partition such as /boot. It is actually a worse implementation than simply leveling over the whole card.
I'm not sure I follow what you are saying.

The controller in SD cards does not know anything about your partitions or the file systems you may have in those partitions. It only knows that you want to read and write blocks to some position in the storage space.

The argument is that SD cards perform wear leveling over various areas of the storage space independently. Those areas being 4MB or whatever in size. If that is true then you can arrange your partitions to align with those area boundaries.

If you can do that then a heavily rewritten partition could not cause writes to blocks in unwritten partitions. I which case your writable partition can fail but your unwritten partition will still be intact. Which is good if that is your boot partition.

Anyway, whatever, as far as I am concerned none of this SD card behavior is specified so we cannot use it. I don't want my systems to rely on rumor and speculation. Besides, if the writable partition fails I cannot not replace it, so the fact I can still boot from an undamaged partition does not help much.
Memory in C++ is a leaky abstraction .

User avatar
jojopi
Posts: 3085
Joined: Tue Oct 11, 2011 8:38 pm

Re: Raspbian with Read-only Root

Fri Oct 27, 2017 5:08 am

Heater wrote:
Fri Oct 27, 2017 3:49 am
The argument is that SD cards perform wear leveling over various areas of the storage space independently. Those areas being 4MB or whatever in size. If that is true then you can arrange your partitions to align with those area boundaries.
No, 4MiB (or cat /sys/dev/block/179\:0/device/preferred_erase_size, or neither) is the allocation unit size, the granularity at which the hardware can erase. Erasure is the damaging operation, so wear levelling must be across a pool of allocation units.

Either way, my argument still holds. Assume the flash is rated for 1000 P/E cycles. If the wear levelling is perfect you can write up to 8TB to an 8GB card before you must be exceeding the cycle counts. (In practice the total could be a lot less, because even small writes may require erasing whole units.) At low MB/s, that is a reasonable life.

If wear levelling only works within partitions then you can write no more than 48GB total to a 48MB /boot partition, which could be done within hours. If wear levelling somehow worked only within 4MiB units, you would only be able to take a thousand pictures in a camera before you destroyed the root directory or FAT.

We do not know how the controllers work, but we should hope that they spread writes over the whole card.

Heater
Posts: 13671
Joined: Tue Jul 17, 2012 3:02 pm

Re: Raspbian with Read-only Root

Fri Oct 27, 2017 8:20 am

OK. Sounds reasonable.

So we can agree then. Writing to any logical block can result in any other physical block on the card being written as wear-leveling shuffles things around. As such having one writable partition and one write-protected partition does nothing to guarantee the write-protected partition does not get corrupted.
Memory in C++ is a leaky abstraction .

User avatar
SlowBro
Posts: 164
Joined: Sat Feb 18, 2017 1:30 am

Re: Raspbian with Read-only Root

Fri Oct 27, 2017 10:31 am

So it sounds as though you’re saying that corruption can occur even without a power loss? Just daily writing.

Heater
Posts: 13671
Joined: Tue Jul 17, 2012 3:02 pm

Re: Raspbian with Read-only Root

Fri Oct 27, 2017 11:01 am

I once had Pi running for nearly a year, taking camera snapshots to a file then serving them via a web server. In the end it failed, the SD card over heated and smoke was coming out of it!

So yes, corruption can occur even without a power loss? Just daily writing.

OK, that might have been a freak incident but given that the FLASH storage in SD cards has limited number of times it can be written to then it must fail eventually. No matter how good any wear levelling it has is.

Finally, we get a lot of reports here of cards write-protecting themselves, in whole or in part, and as yet we have no explanation as to how that happens. Which convinces me that it is better not to write to and SD card used as a boot media.
Memory in C++ is a leaky abstraction .

User avatar
SlowBro
Posts: 164
Joined: Sat Feb 18, 2017 1:30 am

Re: Raspbian with Read-only Root

Fri Oct 27, 2017 5:33 pm

Well eventual failure is one thing but the concerns are of the rapid decline of relatively new cards.

Mounting read-only clearly must improve the odds, as has been reported in various places.

On a hunch I checked the TinyCore forums to see what they see on corruption. As you probably know their OS is in memory by default. I didn’t see that they have this issue severely. And yes, they have to write to the flash occasionally for updates.

I’m going to risk it. Going to run read only most of the time and reboot for updates and config for my product. If I start seeing too much corruption I’ll consider alternatives, but this seems to be an 80/20 solution; 20% of effort for 80% of the results. If my product meant life or death that would be different.

Heater
Posts: 13671
Joined: Tue Jul 17, 2012 3:02 pm

Re: Raspbian with Read-only Root

Fri Oct 27, 2017 6:20 pm

It all depends on what you want to do and what failure rate you can tolerate.

The 80/20 rule sounds great. Until you have a 100 units installed in remote locations and 20 of them are going to fail prematurely and it costs you a thousand bucks to visit each one and fix it.
Memory in C++ is a leaky abstraction .

User avatar
AllanGH
Posts: 29
Joined: Wed Oct 25, 2017 8:09 am
Location: 34.033909, -117.313616

Re: Raspbian with Read-only Root

Sat Oct 28, 2017 4:47 am

Greetings all.

In reading through the contents of the thread, I came to understand that this method of making / RO will work with Stretch Lite, but would be unsuccessful on a Stretch-GUI based installation; or have I come to the wrong conclusion?
##########################

http://www.catb.org/~esr/faqs/smart-questions.html

Heater
Posts: 13671
Joined: Tue Jul 17, 2012 3:02 pm

Re: Raspbian with Read-only Root

Sat Oct 28, 2017 8:13 am

I have only used on on Stretch Lite.

I see no reason why it it would not work with full Stretch except....

The technique described in this thread involves using an overlay file system. This basically makes the rootfs look writable to the system by keeping any writes in RAM. The writes never make it down to the write-protected fs on the SD card. Writes are of course lost on a power cycle and you are back to the pristine state.

This of course requires using up RAM. We only have 1GB of RAM so running applications that require use of a lot of RAM and/or file writing may not be happy. For example a web browser requires a lot of RAM anyway and then does a lot of caching of downloaded things to files. Which is more RAM usage in the overlay.

I'm hoping you try it and report back how you get on. What works, what does not.
Memory in C++ is a leaky abstraction .

User avatar
AllanGH
Posts: 29
Joined: Wed Oct 25, 2017 8:09 am
Location: 34.033909, -117.313616

Re: Raspbian with Read-only Root

Sat Oct 28, 2017 8:30 pm

Thanks for the reply, Heater.

I'm one of those guys who has never used windows or Mac, sticking with Red Hat and Debian, for the most part; but have only just now started looking at the Pi for projects that PIC or Atmel can't handle. It seems that--exclusive of it being an ARM architecture--I can approach it from the perspective of a VERY LIMITED resource Atom processor netbook type of system, but I am very much not up to speed with this particular environment.

My initial foray into the Pi pecking order has been cobbling-together a Video Kiosk Controller for our local Seniors Center, so that they can display their calendar events and announcements without having to license windows, Office, and have a full-blown PC operating all the time, below the video display. So far, for me, everything worked with Stretch Lite, but I did have to leave that particular installation at risk from sudden power loss, and I don't like that.

After exploring options to replace the MicroSD card with a ROM that can mimic the flash media, I ran across this thread, and thought that it might be the shorter path to guarding that installation from corruption, should their power suddenly tank. I will definitely try the script posted by spock, since it looks as though I can avoid taking the Pi out of its installed location, and just SSH into it from a netbook and run the script locally. Just knowing that this will get me going in the right direction with the existing Stretch Lite installation is enough for my immediate needs; seeing as satisfying the needs of our Seniors Center has consumed far more time than I initially thought it would.

My mind, of course, tends to take things to the next level, in anticipation of a request for something that sports a WM or DE, so I will definitely explore the option of increasing R/W memory availability and try things with a full Stretch installation.

I'll start testing this on a Pi3B today.

Again, many thanks for your reply.
##########################

http://www.catb.org/~esr/faqs/smart-questions.html

User avatar
SlowBro
Posts: 164
Joined: Sat Feb 18, 2017 1:30 am

Re: Raspbian with Read-only Root

Sun Oct 29, 2017 2:34 am

Heater wrote:
Fri Oct 27, 2017 6:20 pm
It all depends on what you want to do and what failure rate you can tolerate.

The 80/20 rule sounds great. Until you have a 100 units installed in remote locations and 20 of them are going to fail prematurely and it costs you a thousand bucks to visit each one and fix it.
I don't have that situation, but thinking about this more and more bothers me. I may instead write my few transient config files to EEPROM. I can't think how to avoid having /boot read-write unless I can also somehow write a flag to the EEPROM that gets checked before /boot/config.txt gets loaded. Very doubtful.

Return to “General discussion”