Raspberry Pi Forums

Posted: **Sat Oct 19, 2019 11:13 am**

This post is a continuation of my search to find out why the upgrade of the raspberrypi-kernel and bootloader packages on one of my remotely running Pi's is failing regularly.

The search was initially started under a different topic: upgrade of kernel and/or bootloader going wrong (https://www.raspberrypi.org/forums/view ... 8&t=250039), but that path did not provide any clues.

With the next kernel & bootloader upgrade pending, I started looking for other reasons for the problem and I stumbled over something interesting.

Because the pi is at a remote location, what I did is, make a 'perfect' clone of the SD card of the running pi for (remote) recovery purposes.

In order to be able to make configuration adaptations to the clone, I used a mini USB SD-cardreader with the clone inserted and put it in one of the free USB ports, causing it to be automagically mounted.

However, because it is a 'perfect' clone, the UUID's of the clone's partitions are identical to those of the master's partitions:

Code: Select all

 lsblk -o name,mountpoint,size,ptuuid,partuuid
NAME        MOUNTPOINT  SIZE PTUUID           PARTUUID
sda                    14.9G bcb7c562
├─sda1      /boot       257M bcb7c562         bcb7c562-01
└─sda2                 14.7G bcb7c562         bcb7c562-02
mmcblk0                14.9G bcb7c562
├─mmcblk0p1             257M bcb7c562         bcb7c562-01
└─mmcblk0p2 /          14.7G bcb7c562         bcb7c562-02

and looking at /etc/fstab

Code: Select all

proc            /proc           proc    defaults          0       0
PARTUUID=bcb7c562-01  /boot           vfat    defaults          0       2
PARTUUID=bcb7c562-02  /               ext4    defaults,noatime  0       1

reveals that the requested PARTUUID's are available on both /dev/mmcblk0 and /dev/sda

This means, that, during boot there is more than one device to choose from to get the data/code from.

Hence my question: How does the pi select the device to boot from?

I thought it was preset to boot from the SD card, which I believe is /dev/mmcblk0 (Note: I have not touched the usb_boot_mode bit)

If so, why is the boot partition taken from /dev/sda, while the root partition is taken from /dev/mmcblk0 ? (Note: I have not modified /etc/fstab)

If it would be the other way around, i.e. boot from /dev/mmcblk0, I could somehow understand that the root partition is taken from /dev/sda, in case the boot process searches for the requested PARTUUID and finds the first one on /dev/sda.

I could modify /etc/fstab to point to the /dev/mmcblk0 partitions to work around this, but I wonder whether /etc/fstab is consulted during the boot process?

Can somebody shed some light on this? I know (now) that it probably would be better to adapt the PARTUUID and fstab of the clone, but I am nevertheless trying to find an explanation of this issue.

Posted: **Sat Oct 19, 2019 11:24 am**

I can't shed much light on that as I haven't investigated it, but I will say that it is likely that your (unspecified model of) Pi is booting using the VFAT partition on the SD card, but once the kernel is loaded and it accesses the fstab to see what should be mounted, it is using the sda version for /boot instead. That is bad, because during an update the /boot partition on the USB device will be updated and the one on the card (complete with kernel) won't be and you will end up with mismatched kernel and modules.

I prefer to use device names for my partitions when I can trust the device names not to change, otherwise I use labels, with UUIDs and PARTUUIDs as a last resort.

Posted: **Sat Oct 19, 2019 11:25 am**

epposan wrote: ↑
Sat Oct 19, 2019 11:13 am
Hence my question: How does the pi select the device to boot from?

The boot partition it chooses from a device: the first available of SD/USB/network (TFTP)
The root partition it chooses from cmdline.txt (as found on the boot partition)

UUID means Universally Unique IDentifier. If you have the same UUIDs for 2 devices on the same computer lord knows what happens.

Posted: **Sat Oct 19, 2019 11:34 am**

epposan wrote: ↑
Sat Oct 19, 2019 11:13 am
I thought it was preset to boot from the SD card, which I believe is /dev/mmcblk0 (Note: I have not touched the usb_boot_mode bit)

If so, why is the boot partition taken from /dev/sda, while the root partition is taken from /dev/mmcblk0 ? (Note: I have not modified /etc/fstab)

Your confusion is from the boot process being in two stages. The boot ROM loads the firmware which then loads the Linux kernel into RAM. The default for this is the SD card first but you can override it using OTP config bits. The Linux kernel then later on reads /etc/fstab to mount /boot but nothing actually says what the kernel mounts and what the boot ROM uses is the same. Having the same PARTUUID is making Linux pick the wrong partition to mount to /boot.

/boot not being the same as what the boot ROM is using causes problems with upgrades as the upgrade process running inside Linux will be updating the wrong partition. You'll get a mismatch between the kernel in the boot partition and the modules in the root partition.

Change the UUID on your clone or change your configuration to use a specific device file instead.

Read https://www.raspberrypi.org/documentati ... ootflow.md for the exact boot sequence.

Posted: **Sat Oct 19, 2019 11:43 am**

Thanks for all (very quick) replies !

The boot process being performed in two stages really sheds a (bright) light on my issue. It would explain my problems.

I feel confident (now) to change /etc/fstab on the remote pi to the device's names instead of the UUID's.

Posted: **Sat Oct 19, 2019 12:06 pm**

After adapting /etc/fstab, lsblk --fs now reports the mount points as expected:

Code: Select all

NAME        FSTYPE LABEL  UUID                                 FSAVAIL FSUSE% MOUNTPOINT
sda
├─sda1      vfat   boot   0C61-73F5
└─sda2      ext4   rootfs 43f2d0bb-83be-464f-94d0-9a751f376c64
mmcblk0
├─mmcblk0p1 vfat   boot   0C61-73F5                             213.9M    15% /boot
└─mmcblk0p2 ext4   rootfs 43f2d0bb-83be-464f-94d0-9a751f376c64    9.5G    29% /

Posted: **Sat Oct 19, 2019 12:27 pm**

W.r.t. generating a clone using new UUIDS, if using Raspbian's provided "SD card copier" tool the current, Buster, version has that option:
http://www.cpmspectrepi.uk/raspberry_pi ... OnP4B.html
(don't recall if that was the case for Stretch or Jessie)
Trev.

Posted: **Sat Oct 19, 2019 1:10 pm**

The nature of the problem is the UUID (universally unique identifier) not being unique, Linux will use the first one it finds in a naïve search of /dev/disk/by-uuid.

Code: Select all

root@intrepid:/dev/disk/by-uuid # ls -la
total 0
drwxr-xr-x 2 root root 140 Oct 13 23:41 .
drwxr-xr-x 7 root root 140 Oct 13 23:41 ..
lrwxrwxrwx 1 root root  10 Oct 13 23:41 0062acee-6c4f-48e9-97cb-496118d6a63f -> ../../sda8
lrwxrwxrwx 1 root root  10 Oct 13 23:41 7c731579-fad4-42a3-829a-9a0fcf33c9cd -> ../../sda6
lrwxrwxrwx 1 root root  10 Oct 13 23:40 9a9ffb1b-8a51-4678-9f56-1a89f7977e78 -> ../../sda5
lrwxrwxrwx 1 root root  10 Oct 13 23:40 a5ddd0ce-3568-46e7-a45b-15f65d4d12ca -> ../../sda7
lrwxrwxrwx 1 root root  10 Oct 13 23:40 E6C2-1C12 -> ../../sda1
root@intrepid:/dev/disk/by-uuid #

It's trivial to rewrite a UUID (/dev/whateverdevicehere needs to be umount'ed).

sudo apt install uuid-runtime
sudo tune2fs /dev/whateverdevicehere -U $(uuidgen)

Posted: **Sat Oct 19, 2019 2:02 pm**

I am happy to report that the remote update of raspberrypi-kernel and bootloader worked out as expected.

Thanks to all of you for the tips and directions. I learned a great deal today !

Posted: **Sat Oct 19, 2019 2:08 pm**

DougieLawson wrote: ↑
Sat Oct 19, 2019 1:10 pm
It's trivial to rewrite a UUID (/dev/whateverdevicehere needs to be umount'ed).

sudo apt install uuid-runtime
sudo tune2fs /dev/whateverdevicehere -U $(uuidgen)

Apparantly the default is to use UUID's. Would it be better to stick to UUID's in this case (with the cloned SD card in /dev/sda)

Posted: **Sat Oct 19, 2019 2:29 pm**

If your UUID are really UNIQUE then using them guarantees the system can ONLY mount the partition you want to mount.

Personally, I use PARTUUID as they're easier to work with. I start with the output from a blkid > /boot/blkid command and my favourite editor.

Code: Select all

pi@intrepid:~$ sudo -s
root@intrepid:/home/pi_i # cd /boot
root@intrepid:/boot # blkid > blkid
root@intrepid:/boot # cat blkid
/dev/sda1: UUID="E6C2-1C12" TYPE="vfat" PARTUUID="02532899-01"
/dev/sda5: LABEL="root" UUID="9a9ffb1b-8a51-4678-9f56-1a89f7977e78" TYPE="ext4" PARTUUID="02532899-05"
/dev/sda6: UUID="7c731579-fad4-42a3-829a-9a0fcf33c9cd" TYPE="ext4" PARTUUID="02532899-06"
/dev/sda7: UUID="a5ddd0ce-3568-46e7-a45b-15f65d4d12ca" TYPE="ext4" PARTUUID="02532899-07"
/dev/sda8: UUID="0062acee-6c4f-48e9-97cb-496118d6a63f" TYPE="ext4" PARTUUID="02532899-08"
root@intrepid:/boot # cat cmdline.txt
dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 root=PARTUUID=02532899-05 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait
root@intrepid:/boot # cat /etc/fstab
proc /proc proc defaults 0 0
PARTUUID=02532899-01 /boot vfat defaults 0 2
PARTUUID=02532899-05 / ext4 defaults,noatime 0 1
root@intrepid:/boot #

Posted: **Sat Oct 19, 2019 3:22 pm**

I see. Thanks for the clarification. I guess you have a larger external USB drive attached.

With more than the two basic partitions on the same external drive that makes a lot of sense, even more, if one is using an external partition manager (like gparted) to manipulate the drive.

Posted: **Sat Oct 19, 2019 7:04 pm**

epposan wrote: ↑
Sat Oct 19, 2019 3:22 pm
I see. Thanks for the clarification. I guess you have a larger external USB drive attached.

That RPi 3B boots from a WD PiDrive hard drive (in their very pretty WD case). No SDCard in that one.

Raspberry Pi Forums

How does the pi select the device to boot from?

How does the pi select the device to boot from?

Re: How does the pi select the device to boot from?

Re: How does the pi select the device to boot from?

Re: How does the pi select the device to boot from?

Re: How does the pi select the device to boot from?

Re: How does the pi select the device to boot from?

Re: How does the pi select the device to boot from?

Re: How does the pi select the device to boot from?

Re: How does the pi select the device to boot from?

Re: How does the pi select the device to boot from?

Re: How does the pi select the device to boot from?

Re: How does the pi select the device to boot from?

Re: How does the pi select the device to boot from?