Strange behavior on Pi OS x64 {update?}

[whiteshepherd]

I have a Pi OS x64 web server that has been running good for some time. I decided to switch powered USB hubs so I did a upgrade of the OS then shutdown. I tried the new hub but it wouldn't boot. So I switched back to the old hub and still no boot. Here is where things get strange. As I explain my debugging process I will list hardware I have by letters to help you keep track.

My setup has a Raspberry Pi 4B 8GB with a USB3 powered hub. A 1TB SSD (A) has the OS installed. I boot off the "boot" of SDcard (A) and load from cmdline.txt the root on the SSD with it's PARTUUID. I also have a identical 1TB SSD (B) which I weekly rsync {rsync -avxHAX --progress --delete /drive(A) /drive(B)}. I do this so if I ever had a crash on drive A I would have an older backup on drive B I could fall back on without redoing everything. I also have a SDcard B which has a basic install of Pi OS x64 that boots of the SDcard itself for basic diagnostics.

I first modify SDcard A to boot off SSD B. I modify fstab on SSD B to load it's own UUID as root. It just won't boot. I wonder if the SDcard A is bad.

Next I boot SDcard B and fsck SSD A and B. Drive A has a few errors fixed. Drive B is clean. I modify cmdline.txt on SDcard B to load SSD A and fstab on SSD A. Still no boot. I then do the same switching SDcard B to load SSD B and it works! All my files are there except root is loaded as read only! The drive is clean. So issuing the command "mount -o remount,rw /dev/sda1 /" switches it back to "rw". So at least I have a working system again.

From this point I decide to make my SSD A mirror SSD B so I can restore things and keep B as a backup. I issue the command "rsync -avxHAX --progress --delete /drive(B) /drive(A)" and many files are copied finishing in about 45 seconds (these are fast drives). I switch SDcard B cmdline.txt to boot SSD A (along with modifying it's fstab). SSD A will not boot at all. So switch back to booting off SSD B which works though read only.

I don't want to make to many changes on SSD B as it's my only backup. Would rather get SSD A running. I' thinking now rsync only copies files that are different. Perhaps some of the files are corrupt and rsync can tell that? I could do a DD partition copy. By my drive is %6 full and I don't want to waste 1TB of writes on a mostly empty drive.

Not sure if the update or new USB hub corrupted things (leaning towards USB power issue may of corrupted things?). What would be the best way to mirror files on B to A preserving ACLs which I use? Any other thoughts to fix issues happening like read only on root?

[pidd]

I've been trying to use rsync to backup the complete system and it doesn't work but it stands a better chance in your case because you are copying / to / whereas i'm copying / to /backup/ and trying to restore to root.

"I first modify SDcard A to boot off SSD B. I modify fstab on SSD B to load it's own UUID as root. It just won't boot. I wonder if the SDcard A is bad."

You would need to modify cmdline.txt on SDcard A to use the partuuid of SSD B unless you set SSD A and SSD B partuuid's the same.

[whiteshepherd]

I am MOSTLY back. I used clonezilla to copy my backup partition to the primary drive and now the primary drive is booting. However just like the backup drive the root file system is booting as read only until I manually issue the command: "mount -o remount,rw /dev/sda1 /"

Any idea how to fix this so it boots as read+write? I've already scanned the drives with fsck and they are clean. I suspect this happened the original backup is a copy (root drive A -> root drive B) with rsync?