upgrade of kernel and/or bootloader going wrong

[epposan]

I am trying to troubleshoot a problem with upgrading the kernel and/or bootloader on a Raspberry Pi 2 Model B

I remotely manage several of these older (headless) raspberry-pi's as OpenVPN servers for several years now. They still are up to their task.

However, the upgrade on ONE of these pi's is repeatedly problematic: it somehow gets misconfigured after almost every raspberrypi-kernel or -bootloader update.

Since the problem until now happened using Stretch, I was hoping that a fresh install of Buster on another sd card would fix it, but unfortunately, already the first (remote) upgrade after the fresh install failed similarly.

There were NO related errors/warnings during the upgrade, and /var/log/apt/term.log shows NO evidence of any problems

The upgrade was supposed to bring the kernel from 4.19.58 to 4.19.66, but after the necessary reboot, uname -a still returns 4.19.58 This preventing the openvpn server from starting up properly, and conseuently I have to go through several hoops to get access to the problem pi over ssl (the ssl server fortunately still starts up)

The problem is NOT related to this specific version of the kernel/bootloader: it happened several times before with previous versions as well, however only on ONE specific raspberry pi

Steps taken so far:

Checked /lib/modules: the (new) 4.19.66 subfolders are present

Compared md5sum of the following files/folders with a similar pi:

/boot/*
/boot/overlays/*
/lib/modules/*
/var/cache/apt/archives/raspberrypi-*
/var/cache/apt/archives/libraspberrypi-*

Ran debsums on libraspberrypi-(bin dev) and raspberrypi-(bootloader kernel sys-mods net-mods)

However, no differences show up.

The only difference found (apart from uname -a) is, when I run

sudo /opt/vc/bin/vcgencmd version

The problem pi returns:
Jul 9 2019 14:40:53
Copyright (c) 2012 Broadcom
version 6c3fe3f096a93de3b34252ad98cdccadeb534be2 (clean) (release) (start)

The working pi shows
Aug 15 2019 12:06:42
Copyright (c) 2012 Broadcom
version 0e6daa5106dd4164474616408e0dc24f997ffcf3 (clean) (release) (start)

Could it be a hardware problem? It happens on both on Stretch and Buster, and I have used another sd card for the Buster installation. However, as far as I know, all 'code' comes from the sd card and nothing is written to the board...

Any hints on how to proceed would be welcome.

[epoch1970]

Did you try switching PSUs?

There were multiple revisions of Pi2, and OTP bits might vary from one Pi to another.
No idea how that could affect the version of vcgencmd.

[epposan]

The remote pi's all have their own small UPS that would shutdown the system gracefully if there would be PS or Mains issues.

This particular pi is located somehwere in France, more than 800 km from my place, so I took some extra measures to prevent issues caused by power glitches.

Nevertheless, I will follow your advise and replace the PS (and in the process the Pi and the UPS, to be able to troubleshoot the issue locally).

From what I have been able to determine up to this point is, that all files (that I know of) related to kernel and bootloader are correct for the 4.19.66 version, nevertheless the system shows 4.19.58 when running uname -a

I have been in this situation before, and have managed to get everything working again, usually by reinstalling the raspberrypi-(kernel bootloader) pkgs, however, this time I wanted to ask the community for some guidance.

I have a list of checks that I currently perform after any update of kernel and/or bootloader but before the first reboot thereafter, but the checklist still is missing something crucial, since I still keep running into this problem.

[epoch1970]

If uname differs it means the kernel.img file is different.
Yet you wrote above nothing in /boot/ differed from one machine to another?

[rpdom]

Did you look at the checksums and sizes of the files in /boot before and after the upgrade?

Is the /boot partition that is mounted when the Pi is running the same one it boots from?

[epposan]

Ok. I must admit that the checksum statement is incorrect at this point in time.

I am sure that I checked the md5sums before the reboot (otherwise I would not have dared to reboot, as I cannot easily access it due to distance), but after the reboot I lost contact with the pi (because openvpn did not startup correctly)

if I run dpkg --list | grep raspberrypi on both pi's, they both report the same pkg versions:

ii libraspberrypi-bin 1.20190819-1
ii libraspberrypi-dev 1.20190819-1
ii libraspberrypi-doc 1.20190819-1
ii libraspberrypi0 1.20190819-1
ii raspberrypi-artwork 20150921
ii raspberrypi-bootloader 1.20190819-1
ii raspberrypi-kernel 1.20190819-1
ii raspberrypi-net-mods 1.3.0
ii raspberrypi-sys-mods 20190730
ii raspberrypi-ui-mods 1.20190717

However, when I now run debsums, both the kernel and bootloader pkgs show changed files, so I will need to reinstall the pkgs again.

Any idea what additional checks I could perform to make sure that the files after reboot will be the same as during the installation?

Possibly it has to do with the diversion to /usr/share/rpikernelhack/...

[epposan]

As far as the /boot partition is concerned: I did not make any changes to the system. but I will explicitly check before and after the next install.

All the Pi's that I manage are setup as similar as possible, but only one is showing this issue.

My problem is, that up to now, I have not found a way to predict that the upgrade will go wrong.

[epposan]

I have performed a re-install of raspberrypi-(kernel bootloader) and this time it worked.

I keep hoping that someone has a bright idea on: how to verify that after an apt-get upgrade of kernel and/or bootloader, but before the next reboot, that the system is setup properly to run the expected version of kernel and/or bootloader after the reboot.

Currently I check the following before the reboot:

debsums -c raspberrypi-kernel
debsums -c raspberrypi-bootloader
check dates and md5sums in /boot
check dates and md5sums in /boot/overlays
check dates and md5sums in /var/cache/apt/archives/raspberrypi-*
check for errors in /var/log/apt/term.log

Thanks for your help !

[epoch1970]

I don’t understand what is going on.
Especially I can’t understand why a package would fail to install yet there would be no error in apt logs.

You could use the brute force method, run the upgrade commands twice without trying to check anything...

[epposan]

Well, neither could I, that's why I ended up here to ask the community.

Next time, I will try your suggestion to install twice before rebooting.

[epposan]

For the record:
I have posted the (adapted) question at Raspberrypi StackExchange
https://raspberrypi.stackexchange.com/q ... nal-reboot

[DougieLawson]

You are over thinking a simple, trivial and well tested upgrade.

[epposan]

I fully agree with you !

I maintain several local and remote Pi2 and Pi3 installations and have performed many kernel upgrades over time without any hick-ups

However one particular remote Pi2 has been giving kernel upgrade problems several times lately, and after each failing upgrade I try to come up with additional checks to hopefully find the real cause of the problem.

If the Pi would be local, I would not even bother to ask any questions and simply repeat/reinstall the upgrade (or even use rpi-update), however this Pi is at a remote location, where I need to get access through another pc using Teamviewer and/or VNC with access from the internet enabled (which I try to limit as much as possible)

Also, being fairly familiar with electronics and programming, I always have tried to find 'bugs' instead of working around them.

I hope you can bear with me ...

[Ernst]

However one particular remote Pi2 has been giving kernel upgrade problems several times lately, and after each failing upgrade I try to come up with additional checks to hopefully find the real cause of the problem.

Have you changed the memory card ?

[epposan]

Yes, I have changed the sd card.

[epposan]

Having found something that could shed some light on this issue, I have opened a new topic: How does the pi select the device to boot from?
(https://www.raspberrypi.org/forums/view ... 8&t=254754)

[epposan]

This problem has been solved. It was caused by a USB card reader containing a 'perfect' clone of the SD card from which the pi booted, having identical UUID's for its partitions.

For more information see: How does the pi select the device to boot from? (https://www.raspberrypi.org/forums/view ... 8&t=254754)

In my case, I worked around the issue by not using the UUID's as partition identifiers, but instead using the partition names (/dev/mmcblk0p1 and /dev/mmcblk0p2) as identifiers in /etc/fstab and /boot/cmdline.txt

It would also have been also possible to simply change the UUID's of the cloned SD card. Note however, that the UUID's to be used as identifiers are the partition UUID's, which can be displayed using: lsblk -o name,partuuid

Thanks to all participants for your replies.