Mainline Kernel building -- problems and issues

[rimrunner]

I haven't got linux-next (the unreleased mainline kernel) working yet. It doesn't get past the initial colorful screen on the boot.

I have a 32-bit Raspbian install on my RPi4 and I try get the linux-next kernel running on it. I have done compiling on RPi3 (because I'm busy using my RPi4).

Here is what I do step by step. Can somebody notice what might go wrong?

First I get the sources (being at /home/user):
git clone --depth=1 https://git.kernel.org/pub/scm/linux/ke ... x-next.git

Then...
cd linux-next
export ARCH=arm
make multi_v7_defconfig
make menuconfig ...and set on CONFIG_ARM_LPAE and CONFIG_PCIE_BRCMSTB
make -j4
(I have also tried make -j4 zImage modules dtbs)

After compiling I replace RPi4's /boot directory's kernel7l.img and bcm2711-rpi-4-b.dtb with /arch/arm/boot/zImage (renaming it to kernel7l.img) and arch/arm/boot/dts/bcm2711-rpi-4-b.dtb

[pyavitz]

I added a mainline Linux branch to the Image Builder I'm working on for those interested.

Currently it only supports bcm2711/aarch64, but I could easily adjust the packaging patch to drop in all built dtbs.
All images created now use UUID and PARTUUID, so kernels can be built and installed on existing images
found in the release section of the hub.

Edit:
I've added a Raspberry Pi 4B image to the release section that comes with mainline Linux 5.8-RC2.

Edit:
Mainline branch has been removed and integrated into master.

[swahren]

I haven't got linux-next (the unreleased mainline kernel) working yet. It doesn't get past the initial colorful screen on the boot.

What do you expect from linux-next in comparison from much stable stuff like linux-5.7.x or at least linux-5.8rc2?
Do you have your screen connected via HDMI?
Did you copied the kernel modules to the SD card?

[rimrunner]

What do you expect from linux-next in comparison from much stable stuff like linux-5.7.x or at least linux-5.8rc2?
Do you have your screen connected via HDMI?
Did you copied the kernel modules to the SD card?

I expect to be able to test the linux-next. Nothing else.

Yes, I have HDMI.

Okay, it seems I missed the modules part of the tutorial. I don't know how this is possible but may have to do something with it being the very last part of the tutorial.

[pyavitz]

Does the Raspberry Pi 4Bs 5GHz wifi work on mainline or is it just me having a problem?

Edit:
Never mind I figured it. I made a patch in case anyone else is interested.

[swahren]

Does the Raspberry Pi 4Bs 5GHz wifi work on mainline or is it just me having a problem?

Edit:
Never mind I figured it. I made a patch in case anyone else is interested.

Which Kernel version is affected?
Which Wifi mode is affected (AP / STA)?
Which Wifi firmware do you use?
Could you describe the problem more in detail?

This patch looks more like a workaround.

[pyavitz]

Does the Raspberry Pi 4Bs 5GHz wifi work on mainline or is it just me having a problem?

Edit:
Never mind I figured it. I made a patch in case anyone else is interested.

Which Kernel version is affected?
Which Wifi mode is affected (AP / STA)?
Which Wifi firmware do you use?
Could you describe the problem more in detail?

This patch looks more like a workaround.

I simply diffed the foundations /drivers/net/wireless/broadcom against mainlines. I'm currently on 5.8.13, but I see no reason why it wouldn't work in 5.9-rc (On 5.9-rc it patches dirty, but does patch). I'm on the most current firmware I could find in the foundation hubs. As for the problem, whilst on mainline the Pi couldn't find 5GHz. I messed about with changing channels and diff firmware, but got no where. That's when I decided to build the foundations broadcom module and install it, at which point everything worked as it should. So I created the patch and here we are.

As for which kernel version/revision is affected? As far as I can tell, all of them? At least all the ones I've tried off and on. I originally thought maybe it had something to do with the eeprom, but in 5.8.y and up, eeprom seems to be working.

I'm by no means totting this as an end all solution, but for the time being at least I'm not stuck on 2.4GHz.

[swahren]

This patch contains a lot of and different changes (ugly hacks). Since the issue affects only 5 GHz, i guess it's related to the region settings.

What's the output of

Code: Select all

iw reg get

[pyavitz]

This patch contains a lot of and different changes (ugly hacks). Since the issue affects only 5 GHz, i guess it's related to the region settings.

What's the output of

Code: Select all

iw reg get

I tried all that. Different freqs, channels and even changed country codes just to see what would happen. It would appear to me that the foundation is making changes to the broadcom module in order for it to play nice with the Pi's. Hence the reason why the default module in the mainline kernel doesn't work entirely as is.

Has anyone here tried mainline or stable lately and does 5GHz work for you? I had someone else test it for me as well and they also were unable to get 5GHz working. I then had him use the patch and all was well afterwards.

The most recent tests were on 5.8.13 and 5.9-rc7, using Debian Buster (aarch64).

Code: Select all

sudo iw reg get
global
country US: DFS-FCC
	(2400 - 2483 @ 40), (N/A, 30), (N/A)
	(5150 - 5250 @ 80), (N/A, 23), (N/A), AUTO-BW
	(5250 - 5350 @ 80), (N/A, 23), (0 ms), DFS, AUTO-BW
	(5470 - 5730 @ 160), (N/A, 23), (0 ms), DFS
	(5730 - 5850 @ 80), (N/A, 30), (N/A)
	(57240 - 71000 @ 2160), (N/A, 40), (N/A)

[swahren]

Just a quick test with Mainline 5.9-rc2 + KMS patches on RPi 4 with Raspberry Pi OS (32 bit) showed that i was able to connect to my Wifi access point via 5 GHz. I tested with Kernel config: multi_v7_defconfig and enable ARM_LPAE + BRCMSTB_PCIE manually.

dmesg output

Code: Select all

[    6.434190] cfg80211: Loading compiled-in X.509 certificates for regulatory database
[    6.510652] uart-pl011 fe201000.serial: no DMA platform data
[    6.517496] cfg80211: Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7'
[    6.580423] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6
[    6.596244] brcmfmac mmc0:0001:1: Direct firmware load for brcm/brcmfmac43455-sdio.raspberrypi,4-model-b.txt failed with error -2
[    6.706952] random: crng init done
[    6.706957] random: 7 urandom warning(s) missed due to ratelimiting
[    6.745900] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6
[    6.755081] brcmfmac: brcmf_c_preinit_dcmds: Firmware: BCM4345/6 wl0: Mar  2 2020 23:30:41 version 7.45.202 (r724630 CY) FWID 01-72f6ece2
[    6.761403] Bluetooth: hci0: BCM: chip id 107
[    6.761674] Bluetooth: hci0: BCM: features 0x2f
[    6.763184] Bluetooth: hci0: BCM4345C0
[    6.763193] Bluetooth: hci0: BCM4345C0 (003.001.025) build 0000
[    6.769207] Bluetooth: hci0: BCM4345C0 'brcm/BCM4345C0.hcd' Patch
[    6.910378] libphy: bcmgenet MII bus: probed
[    7.000468] unimac-mdio unimac-mdio.-19: Broadcom UniMAC MDIO bus
[    7.602429] Bluetooth: hci0: BCM43455 37.4MHz Raspberry Pi 3+-0159
[    7.602438] Bluetooth: hci0: BCM4345C0 (003.001.025) build 0290
[    8.968752] swapon: swapfile has holes
[    9.304758] bcmgenet fd580000.ethernet: configuring instance for external RGMII (RX delay)
[    9.305223] bcmgenet fd580000.ethernet eth0: Link is Down
[   14.460172] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready

md5sum of the relevant firmware

Code: Select all

9d5d7f1953769b5ded866217c9afc437  brcmfmac43455-sdio.bin
c5aeca0e33de4ae870986c517963fef7  brcmfmac43455-sdio.clm_blob
0ed2738fb42c392c60e34dedb74d0510  brcmfmac43455-sdio.txt

In case it is really a kernel issue, you should be able to reproduce this with Raspberry Pi OS. Did you use the arm64/defconfig from Mainline or a different one?

[pyavitz]

Just a quick test with Mainline 5.9-rc2 + KMS patches on RPi 4 with Raspberry Pi OS (32 bit) showed that i was able to connect to my Wifi access point via 5 GHz. I tested with Kernel config: multi_v7_defconfig and enable ARM_LPAE + BRCMSTB_PCIE manually.

In case it is really a kernel issue, you should be able to reproduce this with Raspberry Pi OS. Did you use the arm64/defconfig from Mainline or a different one?

I'm currently using the bcm2711_defconfig from the raspberrypi/linux github, lightly modded with additions. if I recall correctly?

output:

Code: Select all

[    4.052613] cfg80211: Loaded X.509 cert 'sforshee: 00b28ddf47aef9cea7'
[    4.114488] uart-pl011 fe201000.serial: no DMA platform data
[    4.121386] brcmfmac: F1 signature read @0x18000000=0x15264345
[    4.127780] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6
[    4.129736] usbcore: registered new interface driver brcmfmac
[    4.356295] Bluetooth: hci0: BCM: chip id 107
[    4.356682] Bluetooth: hci0: BCM: features 0x2f
[    4.358302] Bluetooth: hci0: BCM4345C0
[    4.358319] Bluetooth: hci0: BCM4345C0 (003.001.025) build 0000
[    4.360081] brcmfmac: brcmf_fw_alloc_request: using brcm/brcmfmac43455-sdio for chip BCM4345/6
[    4.360590] Bluetooth: hci0: BCM4345C0 'brcm/BCM4345C0.hcd' Patch
[    4.372528] brcmfmac: brcmf_c_preinit_dcmds: Firmware: BCM4345/6 wl0: Feb 27 2018 03:15:32 version 7.45.154 (r684107 CY) FWID 01-4fbe0b04
[    4.724008] random: crng init done
[    4.724024] random: 7 urandom warning(s) missed due to ratelimiting
[    5.200521] Bluetooth: hci0: BCM43455 37.4MHz Raspberry Pi 3+
[    5.200556] Bluetooth: hci0: BCM4345C0 (003.001.025) build 0315
[    5.243191] brcmfmac: brcmf_cfg80211_set_power_mgmt: power save disabled
[    5.253462] bcmgenet fd580000.ethernet: configuring instance for external RGMII (RX delay)
[    5.253794] bcmgenet fd580000.ethernet eth0: Link is Down
[    5.414588] zram: Added device: zram0
[    5.417967] zram: Added device: zram1
[    5.419853] zram: Added device: zram2
[    5.426612] zram: Added device: zram3
[    5.456748] zram0: detected capacity change from 0 to 268435456
[    5.622170] Adding 262140k swap on /dev/zram0.  Priority:100 extents:1 across:262140k SSFS
[    5.633298] zram1: detected capacity change from 0 to 268435456
[    5.682235] Adding 262140k swap on /dev/zram1.  Priority:100 extents:1 across:262140k SSFS
[    5.684928] zram2: detected capacity change from 0 to 268435456
[    5.723321] Bluetooth: BNEP (Ethernet Emulation) ver 1.3
[    5.723330] Bluetooth: BNEP filters: protocol multicast
[    5.723344] Bluetooth: BNEP socket layer initialized
[    5.730211] Adding 262140k swap on /dev/zram2.  Priority:100 extents:1 across:262140k SSFS
[    5.736285] zram3: detected capacity change from 0 to 268435456
[    5.776276] NET: Registered protocol family 38
[    5.782952] Adding 262140k swap on /dev/zram3.  Priority:100 extents:1 across:262140k SSFS
[    5.828778] cryptd: max_cpu_qlen set to 1000
[   11.333915] IPv6: ADDRCONF(NETDEV_CHANGE): wlan0: link becomes ready

Code: Select all

ls /lib/firmware/brcm/{brcmfmac43455-sdio.bin,brcmfmac43455-sdio.clm_blob,brcmfmac43455-sdio.txt}
/lib/firmware/brcm/brcmfmac43455-sdio.bin
/lib/firmware/brcm/brcmfmac43455-sdio.clm_blob
/lib/firmware/brcm/brcmfmac43455-sdio.txt

I haven't compiled or tested mainline @ armv7l on the Pi4 yet. But not sure why using Raspi OS would make any difference, if all the correct firmware is in place? Unless there is something on the software side of things I'm not taking into account. And then there is the patch and I don't see why that would work if the firmware and defconfig was off? I'll do some more testing and check out armv7l and see what happens.

Thanx

[swahren]

I won't recommend using a downstream defconfig with a Mainline kernel, but this shouldn't cause this issue.

Another idea is to preserve the downstream commits to the brcmfmac driver and bitsect to the fixing patch(es).

[agners]

I tried running mainline on a RPi 4 8GB and came across this thread. What does work for me to get serial console working is using `console=ttyS1,115200n8` in `cmdline.txt`. This worked for me with and without `enable_uart=1`.

[rimrunner]

Trying linux-next (an unreleased kernel version distributed solely for testing purposes) with Raspberry Pi OS seems to work otherwise but it doesn't find nor use sound hardware. Even if I explicitly tell the system to load snd_bcm2835 module on the boot, it won't be taken into use (though it at least gets loaded that way).

While compiling kernel I've taken the kernel configuration from an upgraded official OS and the device tree file is from the latest official OS as well (I've also tried a device tree file compiled from the latest rpi-kernel's git tree). Overlays are from the official OS too.

I have tried to compile the latest released mainline kernel (it was 5.9.0 then) and there was no problems with audio there. But I've tried several - maybe four - linux-next kernels and they all have this issue.

Now these linux-next kernels have this error message at logs (followed by vc4 related stuff):
Unable to handle kernel NULL pointer dereference at virtual address 00000004
pgd = (ptrval)
[00000004] *pgd=03922003, *pmd=00000000

Here is the whole OOPS message copy-pasted:
https://pastebin.com/raw/GDrwkmY0

It shows in a backtrace that crash (which kills that process but not the whole kernel) happens at the function vc4_platform_drm_probe which is located in a file linux-next/drivers/gpu/drm/vc4/vc4_drv.c

Code: Select all

static int vc4_platform_drm_probe(struct platform_device *pdev)
{
	struct component_match *match = NULL;
	struct device *dev = &pdev->dev;

	printk("debug: dev1: init_name: %s\n", dev->init_name);

	vc4_match_add_drivers(dev, &match,
			      component_drivers, ARRAY_SIZE(component_drivers));

	printk("debug: dev2: init_name: %s\n", dev->init_name);
	printk("debug: match-> size_t alloc: %zu\n", match->alloc);
	printk("debug: match-> size_t num: %zu\n", match->num);
	printk("debug: match->compare->data: %u\n", (match->compare)->data);
	printk("debug: vc4_drm_ops.bind: %u\n", vc4_drm_ops.bind);

	return component_master_add_with_match(dev, &vc4_drm_ops, match);
}

Those printk calls I've inserted to vc4_drv.c seem to reveal that dev->init_name is empty, so the actual problem may have its origin in earlier functions. The crash seems to happen (on this printk'ed version) at the point when "match" pointer is read by printk (since that printk is never outputed). This suggests that vc4_match_add_drivers function didn't manage to fill that match pointer successfully (it's probably the NULL pointer mentioned in dmesg error).

I'm not sure if it makes sense to investigate this issue further. I guess it's an error related to how RPi's firmware is dealt with (instead of being an actual kernel bug) but I'd like to hear if somebody understands this issue better, what happens here and why a released mainline kernel seems to work while unreleased kernels don't.

[swahren]

@rimrunner Thanks for your analyze. Could you please provide more information about your setup:

Which Raspberry Pi did you use?
Did you tried Linux 5.10-rcX=?
Which Kernel config did you use (multi_v7_defconfig, arm64/defconfig)?

[rimrunner]

@rimrunner Thanks for your analyze. Could you please provide more information about your setup:

Which Raspberry Pi did you use?

Raspberry Pi 4 model B, 4 GB

Did you tried Linux 5.10-rcX=?

Yes, one of the kernels I've tried is 5.10.0-rc2-next-20201109-v7l+

Which Kernel config did you use (multi_v7_defconfig, arm64/defconfig)?

I extracted (recently upgraded, 5.4.51-v7l+) Raspberry Pi OS's kernel config by using a method involving modprobe configs and getting the config file from /proc/config.gz. Then I used menu oldconfig at the kernel tree (after renaming to .config) and gave default answer for every question asked. I've assumed that I get the right settings peculiar for RPi this way.

Now I also recall, that I've considered using a config file taken from Raspberry Pi's own kernel tree (https://github.com/raspberrypi/linux/) instead of my current running RPi OS's kernel config. And I think one of my earlier linux-next builds I tried was made by using that (with similar results). But I'm not totally sure.

I recall I have had bad experiences with kernel building when using mainline's default config files (modules didn't load or so) though I don't remember which device tree (mainline kernel's or RPi OS's) I tried with them.

So I have been a bit confused about which config files are reasonable to use when trying to do some kernel testing with Raspberry Pi and it's easily possible that errors I reported have resulted from making some wrong decisions here.

[swahren]

Thanks for the information.

With Linux 5.10-rcX i meant the torvalds tree aka mainline. 5.10.0-rc2-next-20201109-v7l+ is not mainline.

linux-next = torvalds tree + all planned changes for the next kernel release

So linux-next is currently a very early version of Linux 5.11.

Btw please don't try to mix vendor stuff (config or device tree) with mainline kernel. You will get strange behavior. But the mentioned crash is still an issue.

[raun0]

I just cross compile mainline 5.10.0-rc6 succesfully for pi4b. For some reason system is truly slow and following is showing every five secods:

Code: Select all

Nov 30 12:59:05 ubuntu kernel: [   78.558896] mmc1: Timeout waiting for hardware cmd interrupt.
Nov 30 12:59:05 ubuntu kernel: [   78.561108] mmc1: sdhci: ============ SDHCI REGISTER DUMP ===========
Nov 30 12:59:05 ubuntu kernel: [   78.563309] mmc1: sdhci: Sys addr:  0x00000000 | Version:  0x00001002
Nov 30 12:59:05 ubuntu kernel: [   78.565520] mmc1: sdhci: Blk size:  0x00000000 | Blk cnt:  0x00000000
Nov 30 12:59:05 ubuntu kernel: [   78.567717] mmc1: sdhci: Argument:  0x00000000 | Trn mode: 0x00000000
Nov 30 12:59:05 ubuntu kernel: [   78.569910] mmc1: sdhci: Present:   0x1fff0001 | Host ctl: 0x00000001
Nov 30 12:59:05 ubuntu kernel: [   78.572122] mmc1: sdhci: Power:     0x0000000f | Blk gap:  0x00000080
Nov 30 12:59:05 ubuntu kernel: [   78.574329] mmc1: sdhci: Wake-up:   0x00000000 | Clock:    0x0000f447
Nov 30 12:59:05 ubuntu kernel: [   78.576532] mmc1: sdhci: Timeout:   0x00000000 | Int stat: 0x00000000
Nov 30 12:59:05 ubuntu kernel: [   78.578717] mmc1: sdhci: Int enab:  0x00ff1003 | Sig enab: 0x00ff1003
Nov 30 12:59:05 ubuntu kernel: [   78.580863] mmc1: sdhci: ACmd stat: 0x00000000 | Slot int: 0x00000000
Nov 30 12:59:05 ubuntu kernel: [   78.583031] mmc1: sdhci: Caps:      0x45ee6432 | Caps_1:   0x0000a525
Nov 30 12:59:05 ubuntu kernel: [   78.585226] mmc1: sdhci: Cmd:       0x0000371a | Max curr: 0x00080008
Nov 30 12:59:05 ubuntu kernel: [   78.587382] mmc1: sdhci: Resp[0]:   0x00000000 | Resp[1]:  0x00000000
Nov 30 12:59:05 ubuntu kernel: [   78.589520] mmc1: sdhci: Resp[2]:   0x00000000 | Resp[3]:  0x00000000
Nov 30 12:59:05 ubuntu kernel: [   78.591630] mmc1: sdhci: Host ctl2: 0x00000000
Nov 30 12:59:05 ubuntu kernel: [   78.593731] mmc1: sdhci: ADMA Err:  0x00000000 | ADMA Ptr: 0x00000000
Nov 30 12:59:05 ubuntu kernel: [   78.595860] mmc1: sdhci: ============================================