User avatar
SlowBro
Posts: 164
Joined: Sat Feb 18, 2017 1:30 am

Re: Raspbian with Read-only Root

Tue Nov 20, 2018 1:56 pm

I have my doubts that ejolson's workarounds will work for me as I will be doing 100% of updates remotely. When I do apt-get dist-upgrade and there is a new kernel I cannot yet create the initrd until the system reboots to the new kernel, whereupon the new kernel will lack critical modules and cannot connect to the internet, I2C, etc. I possibly could hack together a post-update script to rebuild the initrd and reboot but I don't have confidence in that.

I did find these notes suggesting the use of the followkernel flag which sounds like it would automagically maintain the initrd file with upgrades. But it comes with the same feared caveat, the possibility of a failed upgrade causing a loss of drivers. So I think next I will try a script that does not use initrd or initramfs such as this one.
Initramfs for custom loading of modules at boot does work, but you must update it everytime you use rpi-update, this way:

mkinitramfs -o /boot/initrd
In the Raspberry Pi 2 you must also include this in /boot/config.txt:

initramfs initrd followkernel
ramfsfile=initrd
ramfsaddr=-1
And this is /boot/cdmline.txt:

root=/dev/mmcblk0p2 initrd=-1 dwc_otg.lpm_enable=0 console=ttyAMA0,115200 console=tty1 elevator=deadline rootwait
Because rpi-update is not the official way that Debian updates the Initramfs, you may find weird things happening when you update rpi, so if your system becomes unbootable because there is a needed module in the initrd that you need (like encryption), you must boot with another up to date sdcard, chroot into the new environment, mounting first /boot and /dev, and then update the initrd manually. This just happens because of the privative special way of booting of the Raspberry Pi, out of any reasonable standard.
https://wiki.debian.org/RaspberryPi

ejolson
Posts: 3565
Joined: Tue Mar 18, 2014 11:47 am

Re: Raspbian with Read-only Root

Tue Nov 20, 2018 5:57 pm

SlowBro wrote:
Tue Nov 20, 2018 1:56 pm
I have my doubts that ejolson's workarounds will work for me as I will be doing 100% of updates remotely. When I do apt-get dist-upgrade and there is a new kernel I cannot yet create the initrd until the system reboots to the new kernel, whereupon the new kernel will lack critical modules and cannot connect to the internet, I2C, etc. I possibly could hack together a post-update script to rebuild the initrd and reboot but I don't have confidence in that.

I did find these notes suggesting the use of the followkernel flag which sounds like it would automagically maintain the initrd file with upgrades. But it comes with the same feared caveat, the possibility of a failed upgrade causing a loss of drivers. So I think next I will try a script that does not use initrd or initramfs such as this one.
Unfortunately the followkernel option does not keep the initial RAM filesystem synchronised with the current kernel. Instead it affects the location where the image is placed in memory by the bootloader. As far as I can tell, followkernel is no longer needed as the defaults are reasonable.

The script you mention is nice because it sets up an overlayfs before running the system init process, while at the same time does not use an initial RAM filesystem. In this case the trouble of updating the same is avoided. I'm not sure mounting the overlays under the /mnt directory is such a good idea and I haven't tried that other script. Still, it looks straightforward enough to work well.

I would caution against any sort of updates in situations where recovering from failure to boot is difficult. In particular, there is no remote management engine available on the Raspberry Pi, so fixing a failure to boot in remotely deployed systems usually involves swapping SD cards. Depending on how remote the system is, this could be a significant difficulty. As a precaution, I would recommend keeping an identically configured system on your bench to verify each update before it is deployed to the remote system.

User avatar
SlowBro
Posts: 164
Joined: Sat Feb 18, 2017 1:30 am

Re: Raspbian with Read-only Root

Tue Nov 20, 2018 6:56 pm

ejolson wrote:
Tue Nov 20, 2018 5:57 pm
As a precaution, I would recommend keeping an identically configured system on your bench to verify each update before it is deployed to the remote system.
That and only roll it out to 10% of users at a time. Have you not encountered any issue updating the kernel ejolson -- or anyone?

ejolson
Posts: 3565
Joined: Tue Mar 18, 2014 11:47 am

Re: Raspbian with Read-only Root

Tue Nov 20, 2018 7:30 pm

SlowBro wrote:
Tue Nov 20, 2018 6:56 pm
ejolson wrote:
Tue Nov 20, 2018 5:57 pm
As a precaution, I would recommend keeping an identically configured system on your bench to verify each update before it is deployed to the remote system.
That and only roll it out to 10% of users at a time. Have you not encountered any issue updating the kernel ejolson -- or anyone?
When using the method described in this thread, updating the kernel requires creating a new initial RAM filesystem. I've definitely made the mistake of forgetting to do this (or not having space on the /boot partition to do this) and ended up with a non-booting system.

Figuring out a reliable update mechanism for IOT devices in the field is a difficult problem. Many large companies don't do it correctly if at all. A Pi has the advantage that if something goes wrong with the flash update you can always swap the SD card. That generally can't be done with the flash memory in a Blu-ray player, WiFi router, mobile phone or lightbulb. Still, it is important that updates work for the Pi.

While updated Raspbian packages are tested before release to ensure apt-get upgrade doesn't break things, there are still reports on this forum of things breaking on default installs. The more an installation has been customised, the more likely updates will break things. Fortunately, the overlay filesystem approach to a read-only root does not involve many modifications; therefore, updates are usually still okay.

User avatar
SlowBro
Posts: 164
Joined: Sat Feb 18, 2017 1:30 am

Re: Raspbian with Read-only Root

Wed Nov 21, 2018 12:07 am

ejolson wrote:
Tue Nov 20, 2018 7:30 pm
When using the method described in this thread, updating the kernel requires creating a new initial RAM filesystem. I've definitely made the mistake of forgetting to do this (or not having space on the /boot partition to do this) and ended up with a non-booting system.
So I'm guessing you always update the initrd with your Pi attached to your PC via serial or monitor via HDMI, and only after first rebooting to the new kernel?

ejolson
Posts: 3565
Joined: Tue Mar 18, 2014 11:47 am

Re: Raspbian with Read-only Root

Wed Nov 21, 2018 1:32 am

SlowBro wrote:
Wed Nov 21, 2018 12:07 am
ejolson wrote:
Tue Nov 20, 2018 7:30 pm
When using the method described in this thread, updating the kernel requires creating a new initial RAM filesystem. I've definitely made the mistake of forgetting to do this (or not having space on the /boot partition to do this) and ended up with a non-booting system.
So I'm guessing you always update the initrd with your Pi attached to your PC via serial or monitor via HDMI, and only after first rebooting to the new kernel?
I'm generally running my Pi computers no monitor no keyboard and connect through ssh. Luckily it is possible to generate the initial RAM filesystem before booting into the kernel, otherwise one would have to solve the problem about the chicken or the egg and which came first. Simply specify the new kernel using

# update-initramfs -c -k ZZZZZZ

where ZZZZZZ is the suffix of the new kernel even if the old kernel is still running.

User avatar
TimG
Posts: 291
Joined: Tue Apr 03, 2012 12:15 am
Location: Switzerland

Re: Raspbian with Read-only Root

Wed Nov 21, 2018 5:15 am

This may be relevant -- it's the script I use to create a read-only root, based on the recipe by ejolson. It detects the correct kernel version even after an apt-get upgrade.

There are a few other alterations: It automatically deals with ARM-v6/v7; it puts the overlay scripts in /etc rather than /usr/share/; and it disables auto-mount of /boot (don't forget to re-mount it before doing a kernel upgrade).

Code: Select all

  # configure initramfs
  echo overlay >>/etc/initramfs-tools/modules
  cp -rp /usr/share/initramfs-tools/scripts/local* /etc/initramfs-tools/scripts/
  rename s/local/overlay/ /etc/initramfs-tools/scripts/local*
  patch -d/ -p0 <overlay.patch
  # create initramfs
  # hack for use after apt-get upgrade: get currently installed kernel, even if upgraded but not rebooted
  kver=$(dpkg -L raspberrypi-kernel |perl -ne 'if (/(\d+\.\d+\.\d+)(-v7)?\+$/) { print "$1"; exit; }')
  kver=$kver$(uname -r |perl -pe 's/\d+\.\d+\.\d+//')
  update-initramfs -c -k $kver
  mv /boot/initrd.img{-$kver,}
  # command line parameter
  perl -i -pe '$_="boot=overlay $_" unless /\bboot=overlay\b/' /boot/cmdline.txt
  # don't mount /boot
  perl -i -pe 's"^(.+\s/boot\s+vfat\s+defaults)(.+)"#$_$1,noauto$2"' /etc/fstab

The patch used by the script above:

Code: Select all

--- /etc/initramfs-tools/scripts/overlay	2017-12-04 01:04:03.327799400 +0100
+++ /etc/initramfs-tools/scripts/overlay	2017-12-04 01:17:25.099966600 +0100
@@ -167,11 +167,11 @@
 
 	local_premount
 
-	if [ "${readonly}" = "y" ]; then
+#	if [ "${readonly}" = "y" ]; then
 		roflag=-r
-	else
-		roflag=-w
-	fi
+#	else
+#		roflag=-w
+#	fi
 
 	# FIXME This has no error checking
 	modprobe ${FSTYPE}
@@ -180,11 +180,18 @@
 
 	# FIXME This has no error checking
 	# Mount root
+	mkdir /upper /lower
 	if [ "${FSTYPE}" != "unknown" ]; then
-		mount ${roflag} -t ${FSTYPE} ${ROOTFLAGS} ${ROOT} ${rootmnt}
+		mount ${roflag} -t ${FSTYPE} ${ROOTFLAGS} ${ROOT} /lower
 	else
-		mount ${roflag} ${ROOTFLAGS} ${ROOT} ${rootmnt}
+		mount ${roflag} ${ROOTFLAGS} ${ROOT} /lower
 	fi
+	modprobe overlay
+	mount -t tmpfs tmpfs /upper
+	mkdir /upper/data /upper/work
+	mount -t overlay \
+		-olowerdir=/lower,upperdir=/upper/data,workdir=/upper/work \
+		overlay ${rootmnt}
 }
 
 local_mount_fs()

Scripts to enable or disable the read-only root:

Code: Select all

#!/bin/bash
# enable RO config
[ $(id -u) = 0 ] || { echo "$0 needs to run as root"; exit; }

mount /boot
# enable initramfs
perl -i -pe 'next if $ok; if (s/^#?(initramfs.+)/$1/) {$ok=1;} elsif (eof) {$_.="# --- read only ---\ninitramfs initrd.img\n";}' /boot/config.txt

# truncate logs to reduce overlay ram usage
:|tee /var/log/{daemon.log,syslog,lastlog,kern.log,messages,auth.log}

# reboot
echo -e "\nNB. READ-ONLY ROOT !!!\n" >/etc/motd
shutdown -r now

Code: Select all

#!/bin/bash
# restore RW config
[ $(id -u) = 0 ] || { echo "$0 needs to run as root"; exit; }

mount /boot
# disable initramfs
perl -i -pe 's/^(initramfs.+)/#$1/;' /boot/config.txt

# reboot
echo -e "\nNB. READ-WRITE ROOT !!!\n" >/etc/motd
shutdown -r now
Last edited by TimG on Wed Jan 02, 2019 1:01 pm, edited 1 time in total.

User avatar
SlowBro
Posts: 164
Joined: Sat Feb 18, 2017 1:30 am

Re: Raspbian with Read-only Root

Wed Nov 21, 2018 10:39 am

Good stuff.

Another way to handle this may be to have a bootup script check if certain modules are loaded. If not, reboot to read/write mode, rebuild the initrd, reboot again to read only.

Yet another way may be to wrap apt update commands in a script (that checks if the file system is read only) and if after updating a new kernel is installed and detected, build the initrd for the new one before rebooting.

I think I like the bootup script option best. Should be the most robust but would be the harder of the two to implement.

wksu
Posts: 1
Joined: Sun Dec 30, 2018 2:12 am

Re: Raspbian with Read-only Root

Sun Dec 30, 2018 2:25 am

> perl -i -pe 's"^(.+\s/boot\s+vfat\s+defaults)(.+)"#$_$1,noauto$2"' /etc/stab

This should have been for /etc/fstab.

Also the procedure is missing the part about adding "overlay" to "hook-functions".

There may still be other differences. Even accounting for these I could not get
it to work for my Raspberry Pi Zero W. I am doing headless setup with no visibilty
so I can't debug. I would leave it here for now.

User avatar
TimG
Posts: 291
Joined: Tue Apr 03, 2012 12:15 am
Location: Switzerland

Re: Raspbian with Read-only Root

Wed Jan 02, 2019 1:20 pm

wksu wrote:
Sun Dec 30, 2018 2:25 am
This should have been for /etc/fstab.
Good catch. That seems to be a copy-paste error. I've corrected the code in the post above. (In any case that should not prevent the root partition being made read-only.)
Also the procedure is missing the part about adding "overlay" to "hook-functions".
This is taken care of on the second line, echo overlay >>/etc/initramfs-tools/modules, and saves editing hook-functions.
Even accounting for these I could not get
it to work for my Raspberry Pi Zero W. I am doing headless setup with no visibilty
so I can't debug. I would leave it here for now.
I'm using this script as part of my standard Pi setup procedure, so it definitely ought to work. Are you running the script in an ssh terminal? What output does it give?

bouwew
Posts: 3
Joined: Tue Jan 22, 2019 1:48 pm

Re: Raspbian with Read-only Root

Tue Jan 22, 2019 1:54 pm

@TimG:

I am using you scripts in my read-only Pi-system.
Today, I performed an update: switch to RW-mode, mount /boot and run the upgrade.

During the update I see these messages:

Code: Select all

Processing triggers for mime-support (3.60) ...
Processing triggers for libc-bin (2.24-11+deb9u3) ...
Setting up udev (232-25+deb9u8) ...
addgroup: The group `input' already exists as a system group. Exiting.
update-initramfs: deferring update (trigger activated)
Setting up systemd (232-25+deb9u8) ...
addgroup: The group `systemd-journal' already exists as a system group. Exiting.
Processing triggers for man-db (2.7.6.1-2) ...
Processing triggers for shared-mime-info (1.8-1+deb9u1) ...
Processing triggers for dbus (1.10.26-0+deb9u1) ...
Processing triggers for hicolor-icon-theme (0.15-1) ...
Setting up systemd-sysv (232-25+deb9u8) ...
Setting up oracle-java8-set-default (8u201-1~webupd8~1) ...
Setting up libpam-systemd:armhf (232-25+deb9u8) ...
Processing triggers for initramfs-tools (0.130) ...
/boot/initrd.img-4.14.79-v7+ does not exist. Cannot update.
Are the errors related to initramfs real errors, or can I ignore these messages?
Last edited by bouwew on Wed Jan 23, 2019 7:48 am, edited 1 time in total.

bouwew
Posts: 3
Joined: Tue Jan 22, 2019 1:48 pm

Re: Raspbian with Read-only Root

Wed Jan 23, 2019 7:47 am

@TimG:

As a follow-up on my email below, I looked in detail at your scripts to understand what they do.

I think I have an idea to improve these:

1) in the first script, instead of

Code: Select all

mv /boot/initrd.img{-$kver,}
use

Code: Select all

cp /boot/initrd.img{-$kver,}
This leave the original in place, which is then present when the regular upgrade-process requires it.

2) in the enable ro script add:

Code: Select all

  kver=$(dpkg -L raspberrypi-kernel |perl -ne 'if (/(\d+\.\d+\.\d+)(-v7)?\+$/) { print "$1"; exit; }')
  kver=$kver$(uname -r |perl -pe 's/\d+\.\d+\.\d+//')
  update-initramfs -c -k $kver
  cp /boot/initrd.img{-$kver,}
By adding this a fresh initrd.img file is created when turning to the RO-state. Good to do after the original img-file has been updated by the upgrade-process.
Maybe the 3rd line is not needed, as the regular upgrade-process already does that?

Does this make some sense? Or do I misunderstand the original code?

User avatar
TimG
Posts: 291
Joined: Tue Apr 03, 2012 12:15 am
Location: Switzerland

Re: Raspbian with Read-only Root

Wed Jan 23, 2019 12:24 pm

Hi bouwew,
I think the "/boot/initrd.img-4.14.79-v7+ does not exist. Cannot update." message is a hang-over from Debian which expects a different kernel file naming scheme. Use it as a reminder that initrd.img needs to be updated by hand.

Probably if you had a copy of initrd.img named with the appropriate version the .deb scripts would update it automatically, but I haven't tested this. You'd still need to rename it by hand, or update /boot/config.txt. In any case my /boot partition does not have enough free space for that (not sure why it's made so small by default).

My suggestion would be wrap up "apt-get upgrade" and "update-initramfs -c -k $kver" in a script which can be run separately from the RO/RW scripts (don't forget to mount /boot and check for free space).

Aside: the line "echo -e "\nNB. READ-WRITE ROOT !!!\n" >/etc/motd" in the RW script fails because the filesystem is still RO at that point. It would be better to add something to /etc/profile to produce these messages.

bouwew
Posts: 3
Joined: Tue Jan 22, 2019 1:48 pm

Re: Raspbian with Read-only Root

Fri Jan 25, 2019 2:46 pm

Hi TimG,

Thanks for answering!

Ok, I understand what you mean.
I will try your suggestion, see if that works.
But, maybe better to update (-u) instead if creating a new one (-c): "update-initramfs -u -k $kver"?

kytkija
Posts: 13
Joined: Fri Feb 15, 2019 1:47 pm
Location: Finland

Re: Raspbian with Read-only Root

Fri Feb 15, 2019 10:37 pm

Has anyone got the overlayfs working with 3B+?

I am total beginner, but it would be crucial for my project. I cant find exact/fresh information anywhere. Is it possible with newest stretch, anyone got tips where to find script? It should be ready script or well documented, because i dont have much understanding on linux (yet).

I would appreciate any help :) I will test atleast the adafruit script.

amirfh
Posts: 3
Joined: Sat Mar 30, 2013 11:33 pm

Re: Raspbian with Read-only Root

Wed Mar 20, 2019 10:38 pm

check this for rpi3b+ :
https://github.com/JasperE84/root-ro

duren
Posts: 2
Joined: Wed Jul 31, 2019 4:54 am

Re: Raspbian with Read-only Root

Wed Jul 31, 2019 5:09 am

I'm currently testing out the following instance of the overlay solution on an RPI1B with Raspbian Buster Lite and it's looking quite promising.

https://github.com/JasperE84/root-ro

Had to install initramfs-tools, also installed busybox and bindfs but not sure if they are required.

Even though tmp and log are now in ram by default, to ensure they don't eat all the ram used by the overlay, I have explicit tmpfs settings in /etc/fstab

tmpfs /tmp tmpfs size=8m,defaults,relatime,mode=0755 0 0
tmpfs /var/log tmpfs size=16m,defaults,relatime,mode=0755 0 0

I think this is a good idea because it also keeps tmp and log in ram when / is temporarily or semi-permanently mounted as rw.

One good addition would be partitioning a part of the card as permanently rw that can be written to on a cron schedule to archive logs. In the meantime, one could set up remote logging or rsync to push or pull logs.

Return to “General discussion”