Raspberry pi x64 root is now ro read only!

[whiteshepherd]

Been working on a web server on Pi OS 64 (on Pi4B 8GB) and all has been well. I tried a new USB hub which corrupted the boot SSD. No worried as I have a second SSD that I weekly rsync my boot drive too as a emergency backup with the command: rsync -avxHAX --progress --delete / /mnt/drive2/

On trying to boot my 2nd drive all seems fine except the root boots as read only. I can issue the command mount -o remount,rw /dev/sda2 / and the root becomes read-write. However I have to restart my services manually which seem to fail because of the read only. I suspect I may of get this problem by not excluding specific folders /dev/ etc. But I am not certain what caused this. So question for the experts. Am I borked and have to reinstall from scratch or can I fix this?

I've tried fixing this in fstab but it remains read only on boot (I have a long commit/writeback as my Pi runs off batteries 24/7).
fstab
----------
proc /proc proc defaults 0 0
PARTUUID=a4924ed2-01 /boot vfat defaults 0 2
UUID=9dceb626-04b3-4ee6-bcbe-48394bdaa4c1 / ext4 defaults,rw,noatime,barrier=0,data=writeback,commit=1800,acl,errors=remount-rw 0 1

[pidd]

I'd give up on this, even if you patch it together and get it going, you can never have confidence that some permissions or links are not working correctly, either a backup is complete and perfect or it isn't.

[mahjongg]

Raspberry PI OS-64 is NOT YET READY!!! its not out of beta yet, wait for the release post on the blog.

[jojopi]

The rsync is fine. The -x correctly excludes /boot /dev /proc /run /sys, which you do not want to back up. (You may want to check that these directories do exist on drive2.)

UUID=9dceb626-04b3-4ee6-bcbe-48394bdaa4c1 / ext4 defaults,rw,noatime,barrier=0,data=writeback,commit=1800,acl,errors=remount-rw 0 1

Is that the UUID for drive1 or drive2? They will not be the same unless you copied using dd before switching to rsync. Run blkid to check.

The wrong UUID in /etc/fstab gives exactly the result you descibe: kernel mounts root read only and userspace fails to remount read write.

What are you passing as root= in /boot/cmdline.txt?

errors=remount-rw is not a valid option. The choice is continue|remount-ro|panic.

[whiteshepherd]

jojopi:
"Is that the UUID for drive1 or drive2?"

I ran blkid when I switched. the UUID in fstab is for my backup drive2. cmdline.txt has the PARTUUID for my backup drive. I even double checked and unplugged drive1 when I red your message. It's on drive2.

"errors=remount-rw is not a valid option. The choice is continue|remount-ro|panic"

Thanks and fixed. set to "continue" though will change to remount-ro if I get this fixed.

"The rsync is fine. The -x correctly excludes /boot /dev /proc /run /sys, which you do not want to back up. (You may want to check that these directories do exist on drive2.)"


All folders are there. So my "rsync -avxHAX --progress --delete / /mnt/drive2/" is OK to backup Pi OS to a new drive copying root-drive1 to root-drive2? Any idea how that may cause read only on booting the new drive? I also copied drive2 to a new SSD to test for a bad drive. But the behaves exactly the same. Thanks for any input. I have a ton of work in this installation if I can hopefully save it.

Some other odd effects. When I type "apt-get update" I get:
Get:1 http://deb.debian.org/debian buster InRelease [122 kB]
Get:2 http://deb.debian.org/debian-security buster/updates InRelease [65.4 kB]
Get:3 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]
Get:4 http://archive.raspberrypi.org/debian buster InRelease [32.6 kB]
Reading package lists... Done
E: Release file for http://deb.debian.org/debian/dists/buster/InRelease is not valid yet (invalid for another 10d 1h 23min 3s). Updates for this repository will not be applied.
E: Release file for http://deb.debian.org/debian-security/d ... /InRelease is not valid yet (invalid for another 13d 6h 39min 58s). Updates for this repository will not be applied.
E: Release file for http://deb.debian.org/debian/dists/bust ... /InRelease is not valid yet (invalid for another 13d 10h 29min 40s). Updates for this repository will not be applied.
E: Release file for http://archive.raspberrypi.org/debian/d ... /InRelease is not valid yet (invalid for another 12d 10h 27min 4s). Updates for this repository will not be applied.

My pearl script that reads my gpio (battery voltages) seems to not be working as well.

[jojopi]

You are using PARTUUID in cmdline and for /boot, but UUID for /, why?

Are you able to read the scrolling messages at boot time? There must be errors; not least for the services that need to write, and probably for the problem that prevents writing.

Your system clock thinks it is early July, so repositories are in the future. Do not worry about that until you have a writable filesystem.

What do you get from:

Code: Select all

systemctl status systemd-remount-fs

[whiteshepherd]

systemctl status systemd-remount-fs gives me:

root@inlightofus:/mnt# systemctl status systemd-remount-fs
● systemd-remount-fs.service - Remount Root and Kernel File Systems
Loaded: loaded (/lib/systemd/system/systemd-remount-fs.service; static; vendor preset: enabled)
Active: failed (Result: exit-code) since Wed 2020-07-22 10:06:47 EDT; 30min ago
Docs: man:systemd-remount-fs.service(8)
https://www.freedesktop.org/wiki/Softwa ... ileSystems
Process: 308 ExecStart=/lib/systemd/systemd-remount-fs (code=exited, status=1/FAILURE)
Main PID: 308 (code=exited, status=1/FAILURE)

Jul 22 10:06:47 inlightofus systemd[1]: Starting Remount Root and Kernel File Systems...
Jul 22 10:06:47 inlightofus systemd-remount-fs[308]: mount: /: mount point not mounted or bad option.
Jul 22 10:06:47 inlightofus systemd-remount-fs[308]: /bin/mount for / exited with exit status 32.
Jul 22 10:06:47 inlightofus systemd[1]: systemd-remount-fs.service: Main process exited, code=exited, status=1/FAILURE
Jul 22 10:06:47 inlightofus systemd[1]: systemd-remount-fs.service: Failed with result 'exit-code'.
Jul 22 10:06:47 inlightofus systemd[1]: Failed to start Remount Root and Kernel File Systems.

Based on the error I simplified my fstab and removed all extra options. Rebooted and root is writable and updates are working. Phew thanks!

As to the cause here is my theory. When my sdcard got corrupted I used a backup which had a new cmdline.txt I suspect there was a kernel option that was required for a option I had in fstab. I'll have to track that down. In the mean time thanks again for the help!

[DougieLawson]

Is your rootfs clean? Is it in needs of sudo fsck -f -y /dev/....

I usually use a USB reader and second RPi (running any version of Linux) to run that filesystem check.