Page 5 of 8

Re: Raspbian with Read-only Root

Posted: Tue Oct 24, 2017 11:18 am
by SlowBro
Heater wrote:
Mon Oct 09, 2017 11:10 am
I think if you ever write anything to the SD card you are defeating the point of making this read-only root system. Even if you make a new partition to write data to. The SD card knows nothing about your partitions and file systems it only knows about blocks of data. If it's going to screw up, which they do, it might take down your read-only root as well.
The problem as I understand it is not that writing to flash risks corruption; it’s that writing to flash — while simultaneously powering off — risks corruption to anywhere on the flash. So I can see no problem writing things very seldomly such as wpa_supplicant.conf as long as you don’t have a coinciding power cut. It’s not as though you write one file and it turns into a pumpkin.

I likely will do this for config files on my IoT device. Likely I will create a new partition (or a sub directory under/boot) and write those seldomly-changed files there, sync and flush buffers, symlink them to the real location, and take the slight risk that there won’t be a power cut during the moment they are written. Everything else will be read only or stored in the cloud.

Re: Raspbian with Read-only Root

Posted: Tue Oct 24, 2017 1:38 pm
by TimG
Heater wrote:
Mon Oct 09, 2017 11:10 am
I think if you ever write anything to the SD card you are defeating the point of making this read-only root system. Even if you make a new partition to write data to.

I'm not sure that's right. From https://www.embeddedarm.com/about/resou ... dded-linux:
If local data logging is required a read-write partition can be created on the same medium as the read only partition, but in this case it is understood that in the rare case of NAND corruption due to sudden power loss it is acceptable to lose data. The read/write partition should be created aligned with the allocation group size (typically 4MiB). With this setup, the worst-case scenario from a poorly timed failure is that the system will boot correctly, but the data it has been collecting recently will be corrupted and the filesystem with the read/write data may need to be recreated.

Re: Raspbian with Read-only Root

Posted: Tue Oct 24, 2017 1:44 pm
by Heater
Sounds reasonable.

I have never found a specification of what goes on inside an SD card controller so it's not something I can work to or depend on.

If anyone has such a specification to share that would be great.

Re: Raspbian with Read-only Root

Posted: Tue Oct 24, 2017 9:59 pm
by SlowBro
TimG wrote:
Tue Oct 24, 2017 1:38 pm
The read/write partition should be created aligned with the allocation group size (typically 4MiB).
Anyone know how to do that? Couldn’t find instructions in the link.

Re: Raspbian with Read-only Root

Posted: Tue Oct 24, 2017 10:50 pm
by Heater
When you use a tool like fdisk or parted to create partitions on a block device you can specify exactly the starting and ending offsets of the partitions you create. So arranging your partitions on 4MByte, or whatever, boundaries is quite possible.

It's a clunky business so I prefer to take the easy route and put data on a different device.

https://linux.die.net/man/8/parted

https://www.tecmint.com/fdisk-commands- ... artitions/

Re: Raspbian with Read-only Root

Posted: Wed Oct 25, 2017 9:22 am
by SlowBro
Thanks. So when you say you put it on a different device you mean an external usb drive or something else?

Re: Raspbian with Read-only Root

Posted: Wed Oct 25, 2017 10:03 am
by Heater
External, obviously. Could be a tiny USB memory stick. Whatever storage medium you like.

Re: Raspbian with Read-only Root

Posted: Wed Oct 25, 2017 11:47 am
by SlowBro
Heater wrote:
Wed Oct 25, 2017 10:03 am
External, obviously. Could be a tiny USB memory stick. Whatever storage medium you like.
Not obviously. The Pi can interface on its I2C, SPI, or SDIO interfaces :-) I've pondered adding extra storage via a custom cape using one of those interfaces, which is why I asked.

Re: Raspbian with Read-only Root

Posted: Wed Oct 25, 2017 11:59 am
by Heater
SlowBro,
Not obviously. The Pi can interface on its I2C, SPI, or SDIO interfaces...
Sounds external to me.

For sure you can add external storage accessed via the GPIO.

Re: Raspbian with Read-only Root

Posted: Thu Oct 26, 2017 4:44 am
by FM81
TimG wrote:
Tue Oct 24, 2017 1:38 pm
Heater wrote:
Mon Oct 09, 2017 11:10 am
I think if you ever write anything to the SD card you are defeating the point of making this read-only root system. Even if you make a new partition to write data to.

I'm not sure that's right.
I'd assume Heater is right?
Nobody know's, if wear-leveling on SD-cards, you're typically using on a rasperry, is stopping at partition-borders?
My assumption is: NO, it doesn't ...
(Also tested on own practice, but this was few years ago with older RPi-firmware, never with newer again.)

Greetings, FM_81

Re: Raspbian with Read-only Root

Posted: Thu Oct 26, 2017 5:17 am
by Heater
I have no idea if I'm right or not.

All I know is that nobody so far has presented us with any specification as to what wear-leveling actually goes on in SD cards. It may vary from manufacturer to manufacturer. It may change with different generations of cards.

We do know that SD cards get corrupted. That they write-protect themselves, in whole or in part. As yet no explanation of that is forthcoming.

Therefore I only suggest that if you want data on your SD to remain intact it's better to never write to any part of it.

Re: Raspbian with Read-only Root

Posted: Thu Oct 26, 2017 5:41 am
by rpdom
Heater wrote:
Thu Oct 26, 2017 5:17 am
Therefore I only suggest that if you want data on your SD to remain intact it's better to never write to any part of it.
If you never write to it, how is the data going to get there in the first place? :lol:

Re: Raspbian with Read-only Root

Posted: Thu Oct 26, 2017 9:20 am
by Heater
rpdom,
If you never write to it, how is the data going to get there in the first place?
I'm going to get somebody else to write it there. :)

Re: Raspbian with Read-only Root

Posted: Fri Oct 27, 2017 1:19 am
by jojopi
FM81 wrote:
Thu Oct 26, 2017 4:44 am
Nobody know's, if wear-leveling on SD-cards, you're typically using on a rasperry, is stopping at partition-borders?
My assumption is: NO, it doesn't ...
If the wear levelling did stay within partitions, you would be able to destroy a card prematurely by writing to a small partition such as /boot. It is actually a worse implementation than simply levelling over the whole card.

Re: Raspbian with Read-only Root

Posted: Fri Oct 27, 2017 3:49 am
by Heater
jojppi,
If the wear leveling did stay within partitions, you would be able to destroy a card prematurely by writing to a small partition such as /boot. It is actually a worse implementation than simply leveling over the whole card.
I'm not sure I follow what you are saying.

The controller in SD cards does not know anything about your partitions or the file systems you may have in those partitions. It only knows that you want to read and write blocks to some position in the storage space.

The argument is that SD cards perform wear leveling over various areas of the storage space independently. Those areas being 4MB or whatever in size. If that is true then you can arrange your partitions to align with those area boundaries.

If you can do that then a heavily rewritten partition could not cause writes to blocks in unwritten partitions. I which case your writable partition can fail but your unwritten partition will still be intact. Which is good if that is your boot partition.

Anyway, whatever, as far as I am concerned none of this SD card behavior is specified so we cannot use it. I don't want my systems to rely on rumor and speculation. Besides, if the writable partition fails I cannot not replace it, so the fact I can still boot from an undamaged partition does not help much.

Re: Raspbian with Read-only Root

Posted: Fri Oct 27, 2017 5:08 am
by jojopi
Heater wrote:
Fri Oct 27, 2017 3:49 am
The argument is that SD cards perform wear leveling over various areas of the storage space independently. Those areas being 4MB or whatever in size. If that is true then you can arrange your partitions to align with those area boundaries.
No, 4MiB (or cat /sys/dev/block/179\:0/device/preferred_erase_size, or neither) is the allocation unit size, the granularity at which the hardware can erase. Erasure is the damaging operation, so wear levelling must be across a pool of allocation units.

Either way, my argument still holds. Assume the flash is rated for 1000 P/E cycles. If the wear levelling is perfect you can write up to 8TB to an 8GB card before you must be exceeding the cycle counts. (In practice the total could be a lot less, because even small writes may require erasing whole units.) At low MB/s, that is a reasonable life.

If wear levelling only works within partitions then you can write no more than 48GB total to a 48MB /boot partition, which could be done within hours. If wear levelling somehow worked only within 4MiB units, you would only be able to take a thousand pictures in a camera before you destroyed the root directory or FAT.

We do not know how the controllers work, but we should hope that they spread writes over the whole card.

Re: Raspbian with Read-only Root

Posted: Fri Oct 27, 2017 8:20 am
by Heater
OK. Sounds reasonable.

So we can agree then. Writing to any logical block can result in any other physical block on the card being written as wear-leveling shuffles things around. As such having one writable partition and one write-protected partition does nothing to guarantee the write-protected partition does not get corrupted.

Re: Raspbian with Read-only Root

Posted: Fri Oct 27, 2017 10:31 am
by SlowBro
So it sounds as though you’re saying that corruption can occur even without a power loss? Just daily writing.

Re: Raspbian with Read-only Root

Posted: Fri Oct 27, 2017 11:01 am
by Heater
I once had Pi running for nearly a year, taking camera snapshots to a file then serving them via a web server. In the end it failed, the SD card over heated and smoke was coming out of it!

So yes, corruption can occur even without a power loss? Just daily writing.

OK, that might have been a freak incident but given that the FLASH storage in SD cards has limited number of times it can be written to then it must fail eventually. No matter how good any wear levelling it has is.

Finally, we get a lot of reports here of cards write-protecting themselves, in whole or in part, and as yet we have no explanation as to how that happens. Which convinces me that it is better not to write to and SD card used as a boot media.

Re: Raspbian with Read-only Root

Posted: Fri Oct 27, 2017 5:33 pm
by SlowBro
Well eventual failure is one thing but the concerns are of the rapid decline of relatively new cards.

Mounting read-only clearly must improve the odds, as has been reported in various places.

On a hunch I checked the TinyCore forums to see what they see on corruption. As you probably know their OS is in memory by default. I didn’t see that they have this issue severely. And yes, they have to write to the flash occasionally for updates.

I’m going to risk it. Going to run read only most of the time and reboot for updates and config for my product. If I start seeing too much corruption I’ll consider alternatives, but this seems to be an 80/20 solution; 20% of effort for 80% of the results. If my product meant life or death that would be different.

Re: Raspbian with Read-only Root

Posted: Fri Oct 27, 2017 6:20 pm
by Heater
It all depends on what you want to do and what failure rate you can tolerate.

The 80/20 rule sounds great. Until you have a 100 units installed in remote locations and 20 of them are going to fail prematurely and it costs you a thousand bucks to visit each one and fix it.

Re: Raspbian with Read-only Root

Posted: Sat Oct 28, 2017 4:47 am
by AllanGH
Greetings all.

In reading through the contents of the thread, I came to understand that this method of making / RO will work with Stretch Lite, but would be unsuccessful on a Stretch-GUI based installation; or have I come to the wrong conclusion?

Re: Raspbian with Read-only Root

Posted: Sat Oct 28, 2017 8:13 am
by Heater
I have only used on on Stretch Lite.

I see no reason why it it would not work with full Stretch except....

The technique described in this thread involves using an overlay file system. This basically makes the rootfs look writable to the system by keeping any writes in RAM. The writes never make it down to the write-protected fs on the SD card. Writes are of course lost on a power cycle and you are back to the pristine state.

This of course requires using up RAM. We only have 1GB of RAM so running applications that require use of a lot of RAM and/or file writing may not be happy. For example a web browser requires a lot of RAM anyway and then does a lot of caching of downloaded things to files. Which is more RAM usage in the overlay.

I'm hoping you try it and report back how you get on. What works, what does not.

Re: Raspbian with Read-only Root

Posted: Sat Oct 28, 2017 8:30 pm
by AllanGH
Thanks for the reply, Heater.

I'm one of those guys who has never used windows or Mac, sticking with Red Hat and Debian, for the most part; but have only just now started looking at the Pi for projects that PIC or Atmel can't handle. It seems that--exclusive of it being an ARM architecture--I can approach it from the perspective of a VERY LIMITED resource Atom processor netbook type of system, but I am very much not up to speed with this particular environment.

My initial foray into the Pi pecking order has been cobbling-together a Video Kiosk Controller for our local Seniors Center, so that they can display their calendar events and announcements without having to license windows, Office, and have a full-blown PC operating all the time, below the video display. So far, for me, everything worked with Stretch Lite, but I did have to leave that particular installation at risk from sudden power loss, and I don't like that.

After exploring options to replace the MicroSD card with a ROM that can mimic the flash media, I ran across this thread, and thought that it might be the shorter path to guarding that installation from corruption, should their power suddenly tank. I will definitely try the script posted by spock, since it looks as though I can avoid taking the Pi out of its installed location, and just SSH into it from a netbook and run the script locally. Just knowing that this will get me going in the right direction with the existing Stretch Lite installation is enough for my immediate needs; seeing as satisfying the needs of our Seniors Center has consumed far more time than I initially thought it would.

My mind, of course, tends to take things to the next level, in anticipation of a request for something that sports a WM or DE, so I will definitely explore the option of increasing R/W memory availability and try things with a full Stretch installation.

I'll start testing this on a Pi3B today.

Again, many thanks for your reply.

Re: Raspbian with Read-only Root

Posted: Sun Oct 29, 2017 2:34 am
by SlowBro
Heater wrote:
Fri Oct 27, 2017 6:20 pm
It all depends on what you want to do and what failure rate you can tolerate.

The 80/20 rule sounds great. Until you have a 100 units installed in remote locations and 20 of them are going to fail prematurely and it costs you a thousand bucks to visit each one and fix it.
I don't have that situation, but thinking about this more and more bothers me. I may instead write my few transient config files to EEPROM. I can't think how to avoid having /boot read-write unless I can also somehow write a flag to the EEPROM that gets checked before /boot/config.txt gets loaded. Very doubtful.