Heater
Posts: 13689
Joined: Tue Jul 17, 2012 3:02 pm

Re: Raspbian with Read-only Root

Sun Oct 29, 2017 6:14 am

In my setups /boot remains writable.

That flies in the face of what I have said recently here but then again in normal running /boot is never written to.

It's convenient to have it remain writable such that configurations can be changed. But when that happens the thing is on the bench in front of me and if anything goes bad it's not a disaster.
Memory in C++ is a leaky abstraction .

User avatar
SlowBro
Posts: 165
Joined: Sat Feb 18, 2017 1:30 am

Re: Raspbian with Read-only Root

Sun Oct 29, 2017 3:38 pm

Right so for a device deployed in the field it sounds as though you would use another device such as the C.H.I.P. which boots from onboard NAND. I see that the Pi can boot from NAND in theory but not in practice.

Another consideration is to use industrial strength and/or rugged SDs such as which have SLC or MLC and ECC. Check out this discussion.

Heater
Posts: 13689
Joined: Tue Jul 17, 2012 3:02 pm

Re: Raspbian with Read-only Root

Sun Oct 29, 2017 4:25 pm

For the remote applications we mostly use industrial strength boards like the IGEPv2 https://www.isee.biz/products/igep-proc ... pv2-dm3730 but they run at 200 odd dollars each.
Memory in C++ is a leaky abstraction .

Paul Hutch
Posts: 399
Joined: Fri Aug 25, 2017 2:58 pm
Location: Blackstone River Valley, MA, USA
Contact: Website

Re: Raspbian with Read-only Root

Sun Oct 29, 2017 5:43 pm

SlowBro wrote:
Sun Oct 29, 2017 3:38 pm
Right so for a device deployed in the field it sounds as though you would use another device such as the C.H.I.P. which boots from onboard NAND.
The on board NAND Flash of a C.H.I.P. has all the same failure modes as an SD card on a RasPi (corruption from power loss, wear out from too many writes, etc.) Although I run 4x as many RasPi's as I do C.H.I.P.'s I have had the same number of Flash related failures on each platform. The big advantage to RasPi is that it's easy to just swap in a backup copy of the card preserving the settings. With CHIP I have to re-flash it with the factory supplied OS image and then re-configure it and restore files since there is no method available to back up and restore the on board flash directly.

User avatar
SlowBro
Posts: 165
Joined: Sat Feb 18, 2017 1:30 am

Re: Raspbian with Read-only Root

Sun Oct 29, 2017 10:24 pm

So using SD cards is a feature not a bug. Grin. I will have my config on EEPROM and if the customer’s card fails I’ll overnight them a new one and if they swap it, it should come right up.

@Heater have you ever had a failure after editing boot files on your bench?

I found some relatively inexpensive higher reliability higher quality SD cards. This combined with read-only should give us a fighting chance.
https://www.amazon.com/gp/aw/d/B01DOFCPNW?th=1&psc=1

https://www.digikey.com/product-detail/ ... ND/5119434

https://www.mouser.com/ProductDetail/Ap ... bVxA%3d%3d

This looks like the best one:
https://www.amazon.com/gp/product/B01BD ... th=1&psc=1

rln
Posts: 175
Joined: Wed Apr 09, 2014 1:43 pm
Location: Sweden
Contact: Website

Re: Raspbian with Read-only Root

Fri Nov 03, 2017 3:49 pm

As an alternative to overlay I would to recommend my Nard SDK. It's a distro built from ground up to leave the SD-card alone as much as possible. Everything runs from RAM and it's designed for industrial embedded systems.
http://www.nard.se/
Author of the robust Nard distro http://www.nard.se

User avatar
SlowBro
Posts: 165
Joined: Sat Feb 18, 2017 1:30 am

Re: Raspbian with Read-only Root

Sat Nov 04, 2017 2:27 am

It’s a very cool SDK. Does it work out of the box with Amazon AWS, especially their IoT and server management capabilities? I appreciate that their AWS server management for example can directly manage my Raspbian instance as though it were one of their cloud servers.

Also I don’t see how Nard completely eliminates risk. We had been speaking above how even writing a little bit, as you must do when upgrading Nard, puts you at risk.

What are your stats for SD failures?

rln
Posts: 175
Joined: Wed Apr 09, 2014 1:43 pm
Location: Sweden
Contact: Website

Re: Raspbian with Read-only Root

Sat Nov 04, 2017 1:13 pm

Nard tries to eliminate the risk as far as theoretically possible having only a singe, cheap, non-volatile storage. During upgrade the filesystem is kept in a bootable state at all times (including power-cuts). However, would the filesystem anyhow become corrupt there is a safety feature which can reformat and rewrite the entire SD-card from scratch in a live system.

There is no AWS IoT support but this is an SDK. It means you are supposed to customize it your way to suit your needs.
Author of the robust Nard distro http://www.nard.se

User avatar
SlowBro
Posts: 165
Joined: Sat Feb 18, 2017 1:30 am

Re: Raspbian with Read-only Root

Sat Nov 04, 2017 2:05 pm

What are your stats for SD failures?

rln
Posts: 175
Joined: Wed Apr 09, 2014 1:43 pm
Location: Sweden
Contact: Website

Re: Raspbian with Read-only Root

Sat Nov 04, 2017 3:35 pm

About 0.5% get filesystem corruption sometimes for no obvious reason. But all of those has been reformated and reused successfully (one time issue).
Author of the robust Nard distro http://www.nard.se

User avatar
AllanGH
Posts: 29
Joined: Wed Oct 25, 2017 8:09 am
Location: 34.033909, -117.313616

Re: Raspbian with Read-only Root

Sat Nov 04, 2017 6:33 pm

Interesting. I'll definitely look at this in depth to see if it is appropriate for my some of my use cases.

Thank you for letting us know about this.
##########################

http://www.catb.org/~esr/faqs/smart-questions.html

Powerpuffdude
Posts: 2
Joined: Wed Apr 09, 2014 6:29 pm

Re: Raspbian with Read-only Root

Sun Nov 05, 2017 4:29 pm

ejolson,

Thanks so much for the easy to follow tutorial!
I have a follow-up question that perhaps you could address, namely: is there a way to configure initrd such that the system won't default to allocating 50% of the RAM for the overlay? It reminds me of SquashFS, that also defaults to bite half of the system RAM in Live USB distros.
I find that for my application even 20-30% would suffice, leaving 70-80% of the Raspi's 1GB of RAM for memory hungry-applications, like opening multiple browser tabs. Googling around I stumbled on a parameter called TMPFS_SIZE within the file /etc/default/tmpfs, but changing it there didn’t do the trick. I presume that setting is buried deeper inside the initrd.img* file. Any ideas where I could find/set and regenerate it to recover more Raspi RAM goodness?
Many thanks!

ejolson
Posts: 3724
Joined: Tue Mar 18, 2014 11:47 am

Re: Raspbian with Read-only Root

Mon Nov 06, 2017 6:17 am

Powerpuffdude wrote:
Sun Nov 05, 2017 4:29 pm
I have a follow-up question that perhaps you could address, namely: is there a way to configure initrd such that the system won't default to allocating 50% of the RAM for the overlay?
The tmpfs is mounted in the file /usr/share/initramfs-tools/scripts/overlay using the command

mount -t tmpfs tmpfs /upper

which uses only the default settings. Although the default is to limit the maximum size to 50% of available memory, the actual RAM is used only as needed up to this limit. Thus, much less than 50% is usually used.

If you want to limit the maximum size of the tmpfs for some reason to 25% of available memory, try adding the option size=256M to this command as in

mount -t tmpfs -o size=256M tmpfs /upper

and regenerating the initramfs.

Note that I haven't tried the above modification so it might not work exactly as I've written it. If you try it, it would be interesting to know the results.

User avatar
rpdom
Posts: 15430
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: Raspbian with Read-only Root

Mon Nov 06, 2017 6:44 am

tmpfs is clever. The memory size specified is the maximum it will allocate - but only if it is needed. What it does is use the minimum amount of memory needed to store the files that have been put on it. If more files are stored it will use more memory - up to that limit. If files are deleted it will free up memory.

You can prove this by creating four tmpfs file systems on one system - each will be allocated half of the total memory. That's 4 x 50% which is 200% of the available memory. Obviously that is not possible, and that is not the way it works.

I'm just looking at my Ubuntu laptop. It has five tmpfs file systems. Two of those have sizes of 7.7GB from a system with 15GB of useable RAM. Checking with free -h shows that I have 11GB of RAM available, and no swap being used, which wouldn't be possible if tmpfs actually reserved all the memory it was allocated.

Powerpuffdude
Posts: 2
Joined: Wed Apr 09, 2014 6:29 pm

Re: Raspbian with Read-only Root

Tue Nov 07, 2017 6:29 am

ejolson,

Thanks for the suggestion.
mount -t tmpfs -o size=256M tmpfs /upper worked to the extend that it changed the "apparent" amount of free available RAM with no upper limit, but like you and rpdom mentioned, tmpfs took what it needed and when it had too little left, it crashed...
Another question if I may:
I noted that the 'local' file in usr/share/initramfs-tools/scripts/ in another Debian Distro (Kali) is very similar in structure to the one in Raspbian. So I tried to replicate this procedure on an amd64 architecture but when it came to updating the initramfs it failed. Any tips on adopting the overlay idea to other Linux distros (if possible). I think I like it better than squashFS (neither of which I yet understand...) for making a system Read-only.
Many thanks!

fromJPN
Posts: 14
Joined: Wed Dec 13, 2017 3:40 am

Re: Raspbian with Read-only Root

Wed Dec 13, 2017 4:38 am

Hello.
Please tell me.

A system is 2017-07-05 raspbian-jessie-lite.

1) when it's made setting in read only,but I could write on the "/boot" directory in. Everyone,it a same result?

2) I have used SD card used in setting as read only. I have An image on a SD card is read and compared at occasionally. then 1 byte of decided place has changes.
I tried several times, but a result is written different datam each time.
why? Is this the specification of the SD card? or "initramFS" ?

Please tell me.

User avatar
SlowBro
Posts: 165
Joined: Sat Feb 18, 2017 1:30 am

Re: Raspbian with Read-only Root

Wed Dec 13, 2017 11:53 am

The answer to the first question is that is expected behavior. You must be able to write to /boot to alter the boot configuration to be able to change whether the system boots in read only mode or read/write.

I didn’t understand your second question.

derykmarl
Posts: 9
Joined: Wed Dec 13, 2017 3:33 pm

Re: Raspbian with Read-only Root

Wed Dec 13, 2017 4:46 pm

Excellent, thanks for the guide. This helps protect our custom digital signage setup, which gets turned off overnight at the socket.

Ours is not internet-serving at all, but as I'm running this in a corporate environment it's still important that we update the Pi regularly to keep it relatively secure (as much of a pain as it is to remove the overlay, reboot, update, replace overlay, reboot again - at least once a month is generally fine for Linux) which of course will break the initramfs if a new kernel goes in and has the potential to overwrite some of the script customisation.

As such I've done it slightly differently to try and protect it from the updates. Probably not relevant to everyone, but some may find the custom scripts further down useful....

Firstly, to try and preserve the changes, I don't edit hook-functions per the first block of code in the OP, but instead create /etc/initramfs-tools/hooks/overlay with the contents:

Code: Select all

#!/bin/sh
set -e
. /usr/share/initramfs-tools/hook-functions

manual_add_modules overlay
To be honest I'm not sure (m)any of those other than overlay are needed but I'm just copying the list from the OP and they do no harm.
EDIT: Only the overlay module was needed to I've removed the others.

Hooks need to be executable so I then sudo chmod +x /etc/initramfs-tools/hooks/overlay

I also make any other changes in /etc rather than /usr/share (as conventionally /usr/share tends to be stock configuration that can be overwritten in updates, whilst customisations are a bit safer in /etc), and don't have a 'local-bottom' in my version of Raspbian so it goes:

cd /etc/initramfs-tools
sudo cp /usr/share/initramfs-tools/scripts/local overlay
sudo cp -rp /usr/share/initramfs-tools/scripts/local-premount overlay-premount


Then I follow the rest of the instructions as written in the OP, but have also found the kernel= line to be unnecessary in config.txt.

To make the update process a bit easier I created some scripts which I put into /usr/local/bin (which is in the path). I've been a bit unconventional and put sudo commands inside the script so they can be run as the pi user - this is probably not ideal security wise (particularly seeing as raspbian has no password requirement by default for sudo) so it may be necessary to tighten up security later.

/usr/local/bin/romode - sets Pi to read only (including boot partition)

Code: Select all

#!/bin/bash
sudo -v
sudo mount -o remount,rw /boot
if grep -q boot=overlay /boot/cmdline.txt; then
    echo "ERROR: 'boot=overlay' is already in /boot/cmdline.txt"
    echo "This means the system should already be configured to boot read only"
    echo "Run the command 'rwstatus' to see what the current session is - perhaps you just need to reboot?"
    echo "No changes made."
else
    sudo sed -i 's/^/boot=overlay /g' /boot/cmdline.txt
    echo "If there were no errors the next boot will be in read-only mode."
    echo "Please reboot as soon as possible to engage read-only mode and protect the system."
fi
/usr/local/bin/rwmode - sets it to read/write

Code: Select all

#!/bin/bash
sudo -v
sudo mount -o remount,rw /boot
if grep -q boot=overlay /boot/cmdline.txt; then
    sudo sed -i 's/boot=overlay //g' /boot/cmdline.txt
    echo "If there were no errors the next boot will be read/write."
    echo "Reboot when ready, make the changes, then run romode and reboot again to protect the system."
else
    echo "ERROR: 'boot=overlay' not found in /boot/cmdline.txt"
    echo "This means the system should already be configured to boot read/write"
    echo "Run the command 'rwstatus' to see what the current session is - perhaps you just need to reboot?"
    echo "No changes made."
fi
/usr/local/bin/rwstatus - looks at the mounting status to see if there's an overlay or if root is mounted as normal, and reports back

Code: Select all

#!/bin/bash
if $(mount | grep -q overlay); then
    printf "\033[1;32mREAD-ONLY SYSTEM.\033[0m Changes (such as updates or wifi network passwords)\n"
    echo "are NOT persistent and will be lost on reboot."
    echo " "
    echo "To change this, run the 'rwmode' command and reboot BEFORE making changes."
elif $(mount | grep -q "on / type ext4 (rw,noatime,data=ordered)"); then
    printf "\033[1;31mREAD-WRITE SYSTEM.\033[0m Changes made in this session are permanent. Modify with care!\n"
    echo " "
    printf "\033[1;31mPlease use 'do-updates' command to run updates\033[0m, rather than apt update.\n"
    echo " "
    echo "When finished please run 'romode' and reboot to protect the system"
    echo "from issues such as power failure."
else
    echo "Read/Write status could not be determined."
    echo "Changes may or may not be permanent..."
    echo " "
    echo "Please check the /usr/local/bin/rwstatus script for errors at some point."
fi
I added a call of this command to ~/.profile so that I'm reminded of the Pi's current read/write status and the correct update command when SSH'ing into it. This doesn't affect the local terminal, I think you need to put it into something like .bashrc for that.

/usr/local/bin/do-updates - For use in place of apt update and apt upgrade. First it has a check in place to warn you if there's an overlay (i.e. if the SD is read-only) so that you don't waste your time updating an overlay and then cursing yourself when you reboot :) Then it mounts /boot read-write (seeing as I make ours read-only), does the updates, then updates the initramfs for you so that hopefully any kernel updates don't break the overlay support.

EDIT: this won't work if there's a kernel update. As 'uname -r' returns the running kernel, but the modules it needs are for the new kernel. I ended up outputting some text at the end of the script that says basically "if there are a bunch of errors up above, look in /lib/modules and write the upadte-initramfs command for yourself"...

Code: Select all

#!/bin/bash
if $(mount | grep -q overlay); then
    printf "\033[1;31mREAD-ONLY SYSTEM DETECTED.\033[0m Updates will not be permanent in this state.\n"
    echo "If you intended to update the system permanently, run 'rwmode' and reboot first."
    echo " "
    echo "Press enter to continue or Ctrl-C to abort."
    read
fi
sudo -v
sudo mount -o remount,rw /boot
sudo apt update
sudo apt upgrade
sudo rm -f /boot/initrd7.img
sudo update-initramfs -c -k $(uname -r)
echo "Renaming /boot/initrd.img-$(uname -r) to /boot/initrd7.img"
sudo mv /boot/initrd.img-$(uname -r) /boot/initrd7.img
echo "If all is well you can now reboot.  After rebooting remember to run 'romode' and reboot a second time to protect the system."
Don't forget to make them executable with sudo chmod +x /usr/local/bin/*

Edit: Tweaked last script, which filled the /boot drive as it didn't delete the image first, and also the rename was wrong.
Edit 2: The /boot fstab change in romode and rwmode was irrelevant as if you're in read-only mode it won't remember being changed back to read/write! So just pick a mode for /boot and stick with it.
Last edited by derykmarl on Wed Apr 18, 2018 11:15 am, edited 8 times in total.

fromJPN
Posts: 14
Joined: Wed Dec 13, 2017 3:40 am

Re: Raspbian with Read-only Root

Thu Dec 14, 2017 1:01 am

to SlowBro

Thank you very much for your answer.
I found out that "initramfs" is in operation successfully about my "first question". Thank you. :D


to everybody

Next, It's will explained conversantly about my "second question".

A system is 2017-07-05 raspbian-jessie-lite.
SD card is SanDisk Ulta A1 16GB

(1) I made a SD card as the read only as I was telling by this thread.

(2) I put this SD card in "raspberryPi", power on and used "raspberryPi" for a while.

(3) I power off of "raspberryPi" and took a SD card out.

(4) I took the contents on this SD card out as "DiskImageFile" using "Win32DiskImager" on the "WindowsPC".

(5) I have repeat (4) from (2) above-mentioned and save some "DiskImageFile"s.

(6)When several "DiskImageFile"s collected, I executed the next command from a command line on "WindowsPC"...I have compare the each "DiskImageFile"s .

FC /B DiskImageFile1st.img DiskImageFile2nd.img

Then, result was indicated. :o

02E00578: 61 65

The results have the difference at address " 02E00578:" in the each "DiskImageFile"s .

(7) My question... :?:
what is the cause of this phenomenon?
Is this the specification of the SD card?
Is this the specification of the "initramfs"?
Is this phenomenon normal?

please tell me.

ejolson
Posts: 3724
Joined: Tue Mar 18, 2014 11:47 am

Re: Raspbian with Read-only Root

Thu Dec 14, 2017 2:36 am

fromJPN wrote:
Thu Dec 14, 2017 1:01 am
Then, result was indicated. :o

02E00578: 61 65

The results have the difference at address " 02E00578:" in the each "DiskImageFile"s.
This is a very interesting result. Is that address part of the boot partition? As has been pointed out, the boot partition is mounted rw for convenience. It is possible that mounting it updates a date and time field. While this is not likely a big risk, it would be a good idea to track down where the write occurs.

fromJPN
Posts: 14
Joined: Wed Dec 13, 2017 3:40 am

Re: Raspbian with Read-only Root

Mon Dec 18, 2017 5:27 am

to ejolson


Thank you very much for your answer to me. :P


" /boot" for "read / write" in most cases, it's very convenient.
But only a little, this state has a possibility that a system destroys.

I wish also to make the "/boot" for "read only" and protect a system perfectly.


I tried on some and discovered next thing. :shock:

When a shutdown didn't do on a right procedure, "00400041:" changes to "01".

When a shutdown does on a right procedure, "00400041:" is "00".

I thought,
Maybe, This system was writing information about a shutdown at here, each time for power on. :!:


How to change the "/boot" made "read only" ? :?:

It'll be recently, and I have begun to use Linux. I have experience too little for use Linux. :oops:

User avatar
SlowBro
Posts: 165
Joined: Sat Feb 18, 2017 1:30 am

Re: Raspbian with Read-only Root

Mon Dec 18, 2017 10:16 am

If you make /boot read-only how will you tell the OS when you want to reboot into read-write mode for updates and such?

User avatar
TimG
Posts: 293
Joined: Tue Apr 03, 2012 12:15 am
Location: Switzerland

Re: Raspbian with Read-only Root

Mon Dec 18, 2017 7:59 pm

The boot partition doesn't need to be mounted *at* *all*. All the needed files are read by the bootloader and access is no longer required once the kernel is running.

Add "noauto" to the mount options in /etc/fstab:

Code: Select all

PARTUUID=8858c9fc-01  /boot           vfat    defaults,noauto          0       2
Then if you need to alter the boot config just do a "sudo mount /boot".

ejolson
Posts: 3724
Joined: Tue Mar 18, 2014 11:47 am

Re: Raspbian with Read-only Root

Tue Dec 19, 2017 2:22 am

TimG wrote:
Mon Dec 18, 2017 7:59 pm
The boot partition doesn't need to be mounted *at* *all*. All the needed files are read by the bootloader and access is no longer required once the kernel is running.

Add "noauto" to the mount options in /etc/fstab:

Code: Select all

PARTUUID=8858c9fc-01  /boot           vfat    defaults,noauto          0       2
Then if you need to alter the boot config just do a "sudo mount /boot".
This is an excellent recommendation. I should have put this in the original guide. Next year when I have the time, I'll add the experiences reported here together to make a new guide.

fromJPN
Posts: 14
Joined: Wed Dec 13, 2017 3:40 am

Re: Raspbian with Read-only Root

Fri Dec 22, 2017 9:32 am

to SlowBro
to TimG
to ejolson

Thank you very much for your response to me.
Thank you very much for your important hint to me. :D

I was inspecting some about "initramfs".
And I gathered in the following about the phenomenon on a SD card data changes.

About "00400041:" on the "DiskImageFile" read from a SD card... :roll:

This will be "00" by a power off of a proper procedure.
This will be "01" by a power off of an unjust procedure.

When adding option "noauto" to setting of "/boot" in "/etc/fstab", (When "/boot" didn't mount.) "00400041:" doesn't change.
I could not catch this cause yet.


About "02E00578:" on the "DiskImageFile" read from a SD card... :roll:

This is power supply on or reboot, and inclement (+4) is done.
When value in the "02E00578:" is overflow, the next "02E00579:" be inclement (+1).

When deleted the line "checkfs ${ROOT} root" on the process "local_mount_root () " in the file "/usr/share/initramfs-tools/scripts/overlay", the datam in the "02E00578:" doesn't change. This "checkfs $ {ROOT} root" is the cause.
And Finally, this "checkfs" was calling the command "e2fsck".


And this is my incomplete suggestion to tell now. :|

For example ,the following commands instead of the "checkfs $ {ROOT} root" ...

(ex.1) The next command makes the force-checks each time without rewriting the contents on a DS card. The datam in the "02E00578:" doesn't change by this command.

e2fsck -n -f /dev/mmcblk0p2
echo $?

(ex.2) The next command makes the force-checks the contents on the DS card, and write the data that errors corrected to the DS card. Even if there is no correction of an error, the datam in the "02E00578:" changes by this command.

e2fsck -p -f /dev/mmcblk0p2
echo $?


I wrote "makes the force-checks" on the above. :(
Because ,when the "e2fsck" has not the option "-f" isn't put, and a file system is in the "clean" state, " e2fsck" do not check the file system as carefulness.

The commands (ex.1) (ex.2) above-mentioned is putting option "-f", so even when a file system is in the "clean" state, it's will checked carefulness.

The command "echo $?" will shows the result when execute the command "e2fsck". These are written in logfile "/var/log/boot.log".

I hope that I can choose one of them from which one is the commands above-mentioned (ex.1) (ex.2) as the need arises.
And, now, I'll think how to establish these choice specifically. :shock:

Return to “General discussion”