dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5709
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Common firmware testing

Tue Mar 10, 2020 8:37 pm

Currently the firmware releases for Pi4 and earlier Pi models are built from separate git trees.
That means when common bug fixes or new features are added they need to be cherry-picked to each tree.
This can get overlooked and bugs we believe were fixed on Pi4 can be found to be present on earlier Pi models.

We've long had a goal to switch to the newer tree for everything and would like to get moving forward.
We've been running for a while regression tests on older Pi models using the new tree and they are passing.
But it's basic testing - we'd like to get more users helping.

The "master" branch of rpi-update is now being built from the new tree. You can update to this with:

Code: Select all

sudo rpi-update
If you are on a Pi4, then it should be identical to the standard rpi-update firmware.
If you are on an earlier model, then you'll now be running different firmware code. Ideally everything that used to work will continue to do so.

We'd like volunteers to try this. As there could be bugs, then only run on a non-critical install, or one with a recent backup.
The main areas of change relate to clocks and voltages, hardware (usb/ethernet/HATs), the fkms driver.

Testing on older models is most useful (Pi0, Pi1, Pi2, Pi3, CM). A sanity check that Pi4 still works is also helpful.
Just test what you normally use the Pi for. Let us know if you have any custom hardware (HATs) or custom config.txt settings (e.g. overclock).
If it all works, let us know. If you spot any regression compared to your previous firmware, then let us know.

As usual, if testing is positive the firmware will move from next branch to master branch of rpi-update and later to apt.

EDIT: Common firmware is now on the master branch of rpi-update. next branch is still being used for 5.4 kernel testing.

User avatar
DougieLawson
Posts: 40542
Joined: Sun Jun 16, 2013 11:19 pm
Location: A small cave in deepest darkest Basingstoke, UK
Contact: Website Twitter

Re: Common firmware testing

Tue Mar 10, 2020 11:18 pm

https://github.com/Hexxeh/rpi-firmware/ ... /README.md
needs an update, we're some way away from the 4.14 kernel version.
Any language using left-hand whitespace for syntax is ridiculous

Any DMs sent on Twitter will be answered next month.
Fake doctors - are all on my foes list.

Any requirement to use a crystal ball or mind reading will result in me ignoring your question.

User avatar
DougieLawson
Posts: 40542
Joined: Sun Jun 16, 2013 11:19 pm
Location: A small cave in deepest darkest Basingstoke, UK
Contact: Website Twitter

Re: Common firmware testing

Tue Mar 10, 2020 11:41 pm

Wow, rpi-update certainly runs faster - so that's a bonus.
Any language using left-hand whitespace for syntax is ridiculous

Any DMs sent on Twitter will be answered next month.
Fake doctors - are all on my foes list.

Any requirement to use a crystal ball or mind reading will result in me ignoring your question.

fik
Posts: 41
Joined: Thu Jan 17, 2013 1:34 pm

Re: Common firmware testing

Wed Mar 11, 2020 5:37 pm

What is the difference with just sudo rpi-update? It seems to install kernel 4.19.108 too

DirkS
Posts: 10512
Joined: Tue Jun 19, 2012 9:46 pm
Location: Essex, UK

Re: Common firmware testing

Wed Mar 11, 2020 5:51 pm

fik wrote:
Wed Mar 11, 2020 5:37 pm
What is the difference with just sudo rpi-update? It seems to install kernel 4.19.108 too
Please read Dom's post carefully.
Firmware is more than just the kernel...

User avatar
PeterO
Posts: 6068
Joined: Sun Jul 22, 2012 4:14 pm

Re: Common firmware testing

Wed Mar 11, 2020 5:52 pm

Confusing messages,,,

Code: Select all

pi@raspberrypi:~ $ sudo BRANCH=next rpi-update
 *** Raspberry Pi firmware updater by Hexxeh, enhanced by AndrewS and Dom
 *** Performing self-update
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 18058  100 18058    0     0  29267      0 --:--:-- --:--:-- --:--:-- 29314
 *** Relaunching after update
 *** Raspberry Pi firmware updater by Hexxeh, enhanced by AndrewS and Dom
 *** We're running for the first time
 *** Backing up files (this will take a few minutes)
 *** Backing up firmware
 *** Backing up modules 4.19.97-v7+       
#############################################################
This update bumps to rpi-4.14.y linux tree                    <<<<<<<<<<<<<<<<  Errrr, NO ! 
Be aware there could be compatibility issues with some drivers
Discussion here:
https://www.raspberrypi.org/forums/viewtopic.php?f=29&t=197689
##############################################################
Note it says "This update bumps to rpi-4.14.y linux tree " when it doesn't.
PeterO
Discoverer of the PI2 XENON DEATH FLASH!
Interests: C,Python,PIC,Electronics,Ham Radio (G0DZB),1960s British Computers.
"The primary requirement (as we've always seen in your examples) is that the code is readable. " Dougie Lawson

User avatar
PeterO
Posts: 6068
Joined: Sun Jul 22, 2012 4:14 pm

Re: Common firmware testing

Wed Mar 11, 2020 6:00 pm

I've updated the 3B+ I use with a Pimoroni Unicorn Hat HD (an SPI device) and it seems to be working fine.

PeterO

PS: Noting later comments about WiFI, mine is on a wired network.
Last edited by PeterO on Wed Mar 11, 2020 6:05 pm, edited 1 time in total.
Discoverer of the PI2 XENON DEATH FLASH!
Interests: C,Python,PIC,Electronics,Ham Radio (G0DZB),1960s British Computers.
"The primary requirement (as we've always seen in your examples) is that the code is readable. " Dougie Lawson

User avatar
DougieLawson
Posts: 40542
Joined: Sun Jun 16, 2013 11:19 pm
Location: A small cave in deepest darkest Basingstoke, UK
Contact: Website Twitter

Re: Common firmware testing

Wed Mar 11, 2020 6:02 pm

fik wrote:
Wed Mar 11, 2020 5:37 pm
What is the difference with just sudo rpi-update? It seems to install kernel 4.19.108 too
If you have to ask that question you should exclude yourself from the group that want to alpha test this stuff. Most users do NOT need to run rpi-update and certainly not BRANCH=next rpi-update .
Any language using left-hand whitespace for syntax is ridiculous

Any DMs sent on Twitter will be answered next month.
Fake doctors - are all on my foes list.

Any requirement to use a crystal ball or mind reading will result in me ignoring your question.

fik
Posts: 41
Joined: Thu Jan 17, 2013 1:34 pm

Re: Common firmware testing

Wed Mar 11, 2020 6:02 pm

On RPi3B+ the WiFi is unstable with the new firmware, it periodically disconnects and connects again. I get more than 70% of lost packets. Previous firmware works without problems. It's on 5 GHz.

Code: Select all

...
From 192.168.1.150 (192.168.1.150) icmp_seq=117 Destination Host Unreachable
From 192.168.1.150 (192.168.1.150) icmp_seq=118 Destination Host Unreachable
From 192.168.1.150 (192.168.1.150) icmp_seq=120 Destination Host Unreachable
From 192.168.1.150 (192.168.1.150) icmp_seq=121 Destination Host Unreachable
From 192.168.1.150 (192.168.1.150) icmp_seq=122 Destination Host Unreachable
From 192.168.1.150 (192.168.1.150) icmp_seq=123 Destination Host Unreachable
64 bytes from 192.168.1.180 (192.168.1.180): icmp_seq=126 ttl=64 time=929 ms
64 bytes from 192.168.1.180 (192.168.1.180): icmp_seq=127 ttl=64 time=1.81 ms
64 bytes from 192.168.1.180 (192.168.1.180): icmp_seq=128 ttl=64 time=2.40 ms
64 bytes from 192.168.1.180 (192.168.1.180): icmp_seq=129 ttl=64 time=1.60 ms
64 bytes from 192.168.1.180 (192.168.1.180): icmp_seq=130 ttl=64 time=1.100 ms
64 bytes from 192.168.1.180 (192.168.1.180): icmp_seq=131 ttl=64 time=1.79 ms
64 bytes from 192.168.1.180 (192.168.1.180): icmp_seq=132 ttl=64 time=1.70 ms
64 bytes from 192.168.1.180 (192.168.1.180): icmp_seq=133 ttl=64 time=2.16 ms
64 bytes from 192.168.1.180 (192.168.1.180): icmp_seq=134 ttl=64 time=1.95 ms
64 bytes from 192.168.1.180 (192.168.1.180): icmp_seq=135 ttl=64 time=2.19 ms
64 bytes from 192.168.1.180 (192.168.1.180): icmp_seq=136 ttl=64 time=2.30 ms
64 bytes from 192.168.1.180 (192.168.1.180): icmp_seq=137 ttl=64 time=1.96 ms
From 192.168.1.150 (192.168.1.150) icmp_seq=165 Destination Host Unreachable
From 192.168.1.150 (192.168.1.150) icmp_seq=166 Destination Host Unreachable
From 192.168.1.150 (192.168.1.150) icmp_seq=167 Destination Host Unreachable
From 192.168.1.150 (192.168.1.150) icmp_seq=169 Destination Host Unreachable
From 192.168.1.150 (192.168.1.150) icmp_seq=170 Destination Host Unreachable
64 bytes from 192.168.1.180 (192.168.1.180): icmp_seq=172 ttl=64 time=1.84 ms
64 bytes from 192.168.1.180 (192.168.1.180): icmp_seq=173 ttl=64 time=2.13 ms
64 bytes from 192.168.1.180 (192.168.1.180): icmp_seq=174 ttl=64 time=2.70 ms
64 bytes from 192.168.1.180 (192.168.1.180): icmp_seq=175 ttl=64 time=1.85 ms
64 bytes from 192.168.1.180 (192.168.1.180): icmp_seq=176 ttl=64 time=1.66 ms
64 bytes from 192.168.1.180 (192.168.1.180): icmp_seq=177 ttl=64 time=1.73 ms
64 bytes from 192.168.1.180 (192.168.1.180): icmp_seq=178 ttl=64 time=1.90 ms
64 bytes from 192.168.1.180 (192.168.1.180): icmp_seq=179 ttl=64 time=1.84 ms
64 bytes from 192.168.1.180 (192.168.1.180): icmp_seq=180 ttl=64 time=2.07 ms
64 bytes from 192.168.1.180 (192.168.1.180): icmp_seq=181 ttl=64 time=1.96 ms
64 bytes from 192.168.1.180 (192.168.1.180): icmp_seq=182 ttl=64 time=1.85 ms
64 bytes from 192.168.1.180 (192.168.1.180): icmp_seq=183 ttl=64 time=4.84 ms
64 bytes from 192.168.1.180 (192.168.1.180): icmp_seq=184 ttl=64 time=2.04 ms
...
EDIT:
Forgot to mention: it is 64bit kernel, i.e. 4.19.108-v8+, arm_64bit=1 in /boot/config.txt. The 32bit kernel 4.19.108-v7+ seems fine. Previous firmware with the 64bit kernel is OK (0% packets lost).
Last edited by fik on Wed Mar 11, 2020 6:27 pm, edited 1 time in total.

fik
Posts: 41
Joined: Thu Jan 17, 2013 1:34 pm

Re: Common firmware testing

Wed Mar 11, 2020 6:09 pm

DougieLawson wrote:
Wed Mar 11, 2020 6:02 pm
If you have to ask that question you should exclude yourself from the group that want to alpha test this stuff. Most users do NOT need to run rpi-update and certainly not BRANCH=next rpi-update .
Well, I was asking because to me the BRANCH sounded like the BRANCH we have for kernels, see https://github.com/raspberrypi/linux/branches

But it is not the case, it is still rpi-4.19.y, i.e. the default kernel branch.

User avatar
jahboater
Posts: 6517
Joined: Wed Feb 04, 2015 6:38 pm
Location: Wonderful West Dorset

Re: Common firmware testing

Wed Mar 11, 2020 6:24 pm

No regression on my Pi4,
Modest overclock:

over_voltage=2
arm_freq=1800
arm_64bit=1
Pi4 8GB and Pi4 4GB running Raspberry Pi OS 64-bit

andrum99
Posts: 1357
Joined: Fri Jul 20, 2012 2:41 pm

Re: Common firmware testing

Wed Mar 11, 2020 8:51 pm

DougieLawson wrote:
Tue Mar 10, 2020 11:18 pm
https://github.com/Hexxeh/rpi-firmware/ ... /README.md
needs an update, we're some way away from the 4.14 kernel version.
Actually the warning comes from https://github.com/Hexxeh/rpi-firmware/ ... /NOTICE.md. PR submitted (https://github.com/Hexxeh/rpi-firmware/pull/217) ;)

User avatar
DougieLawson
Posts: 40542
Joined: Sun Jun 16, 2013 11:19 pm
Location: A small cave in deepest darkest Basingstoke, UK
Contact: Website Twitter

Re: Common firmware testing

Wed Mar 11, 2020 11:04 pm

fik wrote:
Wed Mar 11, 2020 6:09 pm
DougieLawson wrote:
Wed Mar 11, 2020 6:02 pm
If you have to ask that question you should exclude yourself from the group that want to alpha test this stuff. Most users do NOT need to run rpi-update and certainly not BRANCH=next rpi-update .
Well, I was asking because to me the BRANCH sounded like the BRANCH we have for kernels, see https://github.com/raspberrypi/linux/branches

But it is not the case, it is still rpi-4.19.y, i.e. the default kernel branch.
It is the same as a branch for kernels. It's just with the HexxeH repo on Github the RPF folks have two supported kernel/firnware versions BRANCH=master and BRANCH=next. We should see a version 5 kernel be made available that way when the RPF/RPT are ready.
Any language using left-hand whitespace for syntax is ridiculous

Any DMs sent on Twitter will be answered next month.
Fake doctors - are all on my foes list.

Any requirement to use a crystal ball or mind reading will result in me ignoring your question.

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5709
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: Common firmware testing

Thu Mar 12, 2020 11:18 am

DougieLawson wrote:
Wed Mar 11, 2020 11:04 pm
We should see a version 5 kernel be made available that way when the RPF/RPT are ready.
Once we are happy that common firmware is largely good, we'll start testing 5.4 kernel on BRANCH=next.
We're introducing the two updates separately to narrow down the causes of regressions.

User avatar
DougieLawson
Posts: 40542
Joined: Sun Jun 16, 2013 11:19 pm
Location: A small cave in deepest darkest Basingstoke, UK
Contact: Website Twitter

Re: Common firmware testing

Thu Mar 12, 2020 7:55 pm

dom wrote:
Thu Mar 12, 2020 11:18 am
DougieLawson wrote:
Wed Mar 11, 2020 11:04 pm
We should see a version 5 kernel be made available that way when the RPF/RPT are ready.
Once we are happy that common firmware is largely good, we'll start testing 5.4 kernel on BRANCH=next.
We're introducing the two updates separately to narrow down the causes of regressions.
There's something badly wrong with this build. All of my V8 kernel systems aren't sync'ing their clocks with NTP.

I don't have enough time to investigate tonight, but my ADS-B receiver wasn't getting the multilateral sync with other ADS-B users as the clock is miles off. I'm reverting that system to BRANCH=master before I head out to the pub.
Any language using left-hand whitespace for syntax is ridiculous

Any DMs sent on Twitter will be answered next month.
Fake doctors - are all on my foes list.

Any requirement to use a crystal ball or mind reading will result in me ignoring your question.

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5709
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: Common firmware testing

Fri Mar 13, 2020 12:08 pm

DougieLawson wrote:
Thu Mar 12, 2020 7:55 pm
There's something badly wrong with this build. All of my V8 kernel systems aren't sync'ing their clocks with NTP.
Ethernet or wifi?

User avatar
DougieLawson
Posts: 40542
Joined: Sun Jun 16, 2013 11:19 pm
Location: A small cave in deepest darkest Basingstoke, UK
Contact: Website Twitter

Re: Common firmware testing

Fri Mar 13, 2020 8:16 pm

dom wrote:
Fri Mar 13, 2020 12:08 pm
DougieLawson wrote:
Thu Mar 12, 2020 7:55 pm
There's something badly wrong with this build. All of my V8 kernel systems aren't sync'ing their clocks with NTP.
Ethernet or wifi?
They all use WiFi. I've only got two of my old 1Bs that are wired.
Rolling back to BRANCH=master rpi-update got them all sync'd normally.
Any language using left-hand whitespace for syntax is ridiculous

Any DMs sent on Twitter will be answered next month.
Fake doctors - are all on my foes list.

Any requirement to use a crystal ball or mind reading will result in me ignoring your question.

mby
Posts: 45
Joined: Sat Dec 15, 2018 3:05 pm

Re: Common firmware testing

Fri Mar 13, 2020 8:19 pm

Running an NTP/NTS-server on a Pi 4 via Ethernet works great, no problems, neither w/ IPv4 nor w/IPv6.

User avatar
DougieLawson
Posts: 40542
Joined: Sun Jun 16, 2013 11:19 pm
Location: A small cave in deepest darkest Basingstoke, UK
Contact: Website Twitter

Re: Common firmware testing

Fri Mar 13, 2020 8:39 pm

mby wrote:
Fri Mar 13, 2020 8:19 pm
Running an NTP/NTS-server on a Pi 4 via Ethernet works great, no problems, neither w/ IPv4 nor w/IPv6.
You seem to have completely missed the point.

My network has NTP (not systemd-timesyncd) with IPv4 and IPv6 everywhere. The NTP server runs on a RPi1B. The other machines are mostly connected with WiFi and BRANCH=next rpi-update (which is alpha testing the firmware breaks system timekeeping).
Any language using left-hand whitespace for syntax is ridiculous

Any DMs sent on Twitter will be answered next month.
Fake doctors - are all on my foes list.

Any requirement to use a crystal ball or mind reading will result in me ignoring your question.

mby
Posts: 45
Joined: Sat Dec 15, 2018 3:05 pm

Re: Common firmware testing

Fri Mar 13, 2020 9:08 pm

Thank you Dougie for your feedback, but my posting was not a reply to yours, just a statement; my apologies if I missed your points.

cjan
Posts: 915
Joined: Sun May 06, 2012 12:00 am

Re: Common firmware testing

Sat Mar 14, 2020 5:37 am

system shutdown randomly when chromium play youtube.
ps. P2-v1.1

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5709
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: Common firmware testing

Fri Mar 27, 2020 11:12 am

There is an update to next branch which should fix wifi regressions.

User avatar
DougieLawson
Posts: 40542
Joined: Sun Jun 16, 2013 11:19 pm
Location: A small cave in deepest darkest Basingstoke, UK
Contact: Website Twitter

Re: Common firmware testing

Fri Mar 27, 2020 11:24 am

dom wrote:
Fri Mar 27, 2020 11:12 am
There is an update to next branch which should fix wifi regressions.
Have you fixed the system clock regression in that version?
Are you going to merge in the missing 766 commits?
Any language using left-hand whitespace for syntax is ridiculous

Any DMs sent on Twitter will be answered next month.
Fake doctors - are all on my foes list.

Any requirement to use a crystal ball or mind reading will result in me ignoring your question.

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5709
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: Common firmware testing

Fri Mar 27, 2020 11:50 am

DougieLawson wrote:
Fri Mar 27, 2020 11:24 am
Have you fixed the system clock regression in that version?
If you are talking about an NTP issue over wifi, then it may well be fixed.
Are you going to merge in the missing 766 commits?
You may need to explain this comment.

User avatar
DougieLawson
Posts: 40542
Joined: Sun Jun 16, 2013 11:19 pm
Location: A small cave in deepest darkest Basingstoke, UK
Contact: Website Twitter

Re: Common firmware testing

Fri Mar 27, 2020 11:52 am

dom wrote:
Fri Mar 27, 2020 11:50 am
DougieLawson wrote:
Fri Mar 27, 2020 11:24 am
Have you fixed the system clock regression in that version?
If you are talking about an NTP issue over wifi, then it may well be fixed.
Are you going to merge in the missing 766 commits?
You may need to explain this comment.
I'm just relaying what Github tells me.
https://githhub.com/hexxeh/rpi-firmware wrote:This branch is 198 commits ahead, 766 commits behind master.
for your next branch. So it needs some synchronisation.
Any language using left-hand whitespace for syntax is ridiculous

Any DMs sent on Twitter will be answered next month.
Fake doctors - are all on my foes list.

Any requirement to use a crystal ball or mind reading will result in me ignoring your question.

Return to “Advanced users”