Pi4 64-bit raspbian kernel for testing - Focus on Pi4

[MrQ]

I don't cross compile. Just building directly on my RPI4.

That can work in theory. Raspbian doesn't provide a gcc-aarch64-linux-gnu package, so you would likely compile under a 64-bit userland (e.g. Gentoo) or chroot (see above).

And I think you should set KERNEL=kernel8 (not kernel7 or kernel7l).

CROSS_COMPILE=aarch64-linux-gnu- rather

BTW., there's no such command inside tools folder...

Sounds like you decided to cross-compile. Highly recommended if you might have to debug things and iterate. Get a toolchain via sudo apt install gcc-aarch64-linux-gnu on your Debian or Ubuntu x86_64 system. The toolchains under https://github.com/raspberrypi/tools are all rather old anyhow.

I managed to cross-compile on my pi4 by following this guide https://www.tal.org/tutorials/raspberry ... bit-kernel + PI4 specific commands from official manual https://www.raspberrypi.org/documentati ... uilding.md , just used the lastest binutils and gcc versions. Build was successful modules installed, dtbs and kernel copied to /boot .... but I get black screen after rainbow at boot. Do I need something more such as

Code: Select all

echo "arm_64bit=1" >> /boot/config.txt

?

[dom]

Since we are talking about `rpi-update` and the 64-bit kernel... I suppose I should ask something I’ve yet to find an answer for with all my searching... how does one get the kernel headers for the kernel install via rpi-update? I would assume those people willing to something closer to bleeding edge might also be the type of person who might make use of the headers. So why not also install the headers?

The kernel headers are not wanted by most and are not trivial in size (around 8000 header files) so we are not going to include them by default with rpi-update. Stable kernels have kernel header files available from apt.

You might be interested in https://github.com/notro/rpi-source/wiki

[dom]

I managed to cross-compile on my pi4 by following this guide https://www.tal.org/tutorials/raspberry ... bit-kernel + PI4 specific commands from official manual https://www.raspberrypi.org/documentati ... uilding.md , just used the lastest binutils and gcc versions. Build was successful modules installed, dtbs and kernel copied to /boot .... but I get black screen after rainbow at boot. Do I need something more such as

Code: Select all

echo "arm_64bit=1" >> /boot/config.txt

?

Yes you need arm_64bit=1 if you have both 32-bit (kernel7l.img) and 64-bit (kernel8.img) kenels installed.
How did you install the kernel to boot? Which file did you copy from and what did you call it?

[Kendek]

You might be interested in https://github.com/notro/rpi-source/wiki

The rpi-source script doesn't work with 64-bit kernels (Unrecognised CPU), only armv7l and armv6l.

[MrQ]

I managed to cross-compile on my pi4 by following this guide https://www.tal.org/tutorials/raspberry ... bit-kernel + PI4 specific commands from official manual https://www.raspberrypi.org/documentati ... uilding.md , just used the lastest binutils and gcc versions. Build was successful modules installed, dtbs and kernel copied to /boot .... but I get black screen after rainbow at boot. Do I need something more such as

Code: Select all

echo "arm_64bit=1" >> /boot/config.txt

?

Yes you need arm_64bit=1 if you have both 32-bit (kernel7l.img) and 64-bit (kernel8.img) kenels installed.
How did you install the kernel to boot? Which file did you copy from and what did you call it?

Code: Select all

KERNEL=kernel7l
make bcm2711_defconfig
make -j4 Image modules dtbs
sudo make modules_install
sudo cp arch/arm64/boot/dts/broadcom/*.dtb /boot/
sudo cp arch/arm64/boot/dts/overlays/*.dtb* /boot/overlays/
sudo cp arch/arm64/boot/dts/overlays/README /boot/overlays/
sudo cp arch/arm64/boot/Image /boot/$KERNEL.img

So I replaced kernel7l.img with new 64-bit kernel.

--------------------------------------------

Ok, I made it. Now 64 bit kernel boots without issues. All you have to do to make 64-bit kernel on 32-bit Raspbian is:

1. Follow this guide to compile binutils and gcc for aarch64 support https://www.tal.org/tutorials/raspberry ... bit-kernel
1a . Use lastest binutils and gcc (binutils-2.32.tar.xz and gcc-9.2.0.tar.xz in my case)

2. Backup /boot

3.Add thece CFLAGS to Makefile

Code: Select all

-Wno-error=int-to-pointer-cast -Wno-implicit-function-declaration -Wno-pointer-to-int-cast

4 Compile kernel

Code: Select all

KERNEL=kernel8
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu-  bcm2711_defconfig
make -j4 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- Image modules dtbs
sudo make modules_install
sudo cp arch/arm64/boot/dts/broadcom/*.dtb /boot/
sudo cp arch/arm64/boot/dts/overlays/*.dtb* /boot/overlays/
sudo cp arch/arm64/boot/dts/overlays/README /boot/overlays/
sudo cp arch/arm64/boot/Image /boot/$KERNEL.img

if you want to change kernel config use

Code: Select all

make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- menuconfig

5. Add to /boot/config.txt

Code: Select all

kernel=kernel8.img
arm_64bit=1

6. Reboot

[dom]

Code: Select all

kernel=kernel8.img
arm_64bit=1

Shouldn't need the kernel=kernel8.img line. That is the default kernel name with arm_64bit=1.

[MrQ]

What's the correct way of installing custom headers and make them system-wide accessible? For 32-bit I used

Code: Select all

make-kpkg kernel_headers

then installed the deb file as usual, but this method doesn't work for cross-compiled kernel

[arthurnorman]

Two questions:
(1) Have I missed something or does the G2 GL (Fake KMS) option limit video resolution to HD such that 4K is not supported. This arises when installing the 64-bit kernel as per here and then following the January 29/Sakaki recipe for using systemd/nspawn to get a 64-bit userland to go with the kernel. Eg maybe the video stuff is closed enough that 4K is not in the 64bit world, or it may be that I have just not spotted explanations about how to activate it.

(2) With the 64-bit kernel active "uname -m" reports aarch64 even though one has a 32-but userland. I believe I have a case where this really muddles autoconf in libffi as well as my own configure.ac scripts that try to put built files somewhere related to the architecture in use. I am working to patch that for myself by making a /usr/local/bin/uname to take precedence over /bin/uname and return what I expected on a 32-bit platform, but does anybody have a better suggestion?

An rpi4 with the root filesytem on an SSD, a 64-bit kernel and then both 32 and 64-bit Buster user spaces via nspawn is looking good so far, but I need longer to assess reliability as and sort out my own confusions!

Thanks. Arthur

[andrum99]

I'm getting the following messages in the log with the 64-bit kernel:

Code: Select all

Sep 10 04:53:25 livingroom kernel: [42196.434063] perf: interrupt took too long (2521 > 2500), lowering kernel.perf_event_max_sample_rate to 79250
Sep 10 06:20:46 livingroom kernel: [47437.778387] perf: interrupt took too long (3154 > 3151), lowering kernel.perf_event_max_sample_rate to 63250
Sep 10 06:25:53 livingroom kernel: [47744.652490] perf: interrupt took too long (3950 > 3942), lowering kernel.perf_event_max_sample_rate to 50500
Sep 10 06:27:45 livingroom kernel: [47856.451975] perf: interrupt took too long (4945 > 4937), lowering kernel.perf_event_max_sample_rate to 40250
Sep 10 06:44:17 livingroom kernel: [48849.052058] perf: interrupt took too long (6182 > 6181), lowering kernel.perf_event_max_sample_rate to 32250
Sep 10 07:59:01 livingroom kernel: [53332.825991] perf: interrupt took too long (7733 > 7727), lowering kernel.perf_event_max_sample_rate to 25750

There's nothing much going in at 04:53 in the morning, but from 06:20 onwards is when the daily backup runs and it takes several hours. The backup is using pigz and tar to create a tarball which is then sent to Google Drive, from the local USB 3.0 attached hard disk.

[dom]

(1) Have I missed something or does the G2 GL (Fake KMS) option limit video resolution to HD such that 4K is not supported. This arises when installing the 64-bit kernel as per here and then following the January 29/Sakaki recipe for using systemd/nspawn to get a 64-bit userland to go with the kernel. Eg maybe the video stuff is closed enough that 4K is not in the 64bit world, or it may be that I have just not spotted explanations about how to activate it.

4K is supported out of the box with fkms with or without 64-bit kernel.
You are limited to 4kp30 by default, but adding

Code: Select all

hdmi_enable_4kp60=1

to config.txt enables 4kp60 too.
But as this is unrelated to 64-bit kernel then discussion of this is best handled elsewhere.

[ShiftPlusOne]

However, I'm skeptical as to outputting instructions specific to higher-end CPU cores not making a difference.

Having played around with a debian arm64 container a bit, I've got to eat crow.

Going from hardfp armv6 to Debian's armv7 binaries may not have been much of an improvement, but debian's arm64 binaries makes a much bigger difference than I would've expected - double-digit in some cases. Using debootstrap to install to tmpfs (to avoid IO bottlenecks) is 20% faster and xz is about 8% faster.

[paul433]

Is Onboard Audio working for others? Headphone Jack. USB based Audio works fine, I have not checked i2s yet or Audio out over HDMI. The BCM2835 device is showing as a valid Alsa device, but getting errors trying to open the device for playback.

I'm using a custom compiled kernel using the arm64 BMC2711 defconfig.

BTW: There are quite a few device omissions in the 64bit defconfig over the 32bit.

[6by9]

BTW: There are quite a few device omissions in the 64bit defconfig over the 32bit.

I'd noticed the same. It should get fixed in the next few days.

[mentaluproar]

In testing, I've found arduino is broken on 64 bit. avrdude throws "mmap error -1" in 64 bit, but works fine in 32 bit. I've also found Konsole will crash during apt installs for some things, but other consoles work fine.

[BRX7]

However, I'm skeptical as to outputting instructions specific to higher-end CPU cores not making a difference.

Having played around with a debian arm64 container a bit, I've got to eat crow.

Going from hardfp armv6 to Debian's armv7 binaries may not have been much of an improvement, but debian's arm64 binaries makes a much bigger difference than I would've expected - double-digit in some cases. Using debootstrap to install to tmpfs (to avoid IO bottlenecks) is 20% faster and xz is about 8% faster.

does a 64bit raspbian buster image exist yet? using 64bit kernel in raspbian means its still using 32bit binaries of libraries and other binaries yes?
also can I use docker 64bit version within raspbian now with the new kernel? or can a arm 64bit container work like https://hub.docker.com/layers/ownyourbi ... e55c0a761b

[DougieLawson]

does a 64bit raspbian buster image exist yet? using 64bit kernel in raspbian means its still using 32bit binaries of libraries and other binaries yes?

The current experiment is 32-bit Raspbian userland code with a 64-bit kernel.

Sakaki-san's Gentoo64 has more 64-bit stuff than Raspbian.

[ShiftPlusOne]

using 64bit kernel in raspbian means its still using 32bit binaries of libraries and other binaries yes?

Yes, but I was using a debian arm64 container within Raspbian.

Initially I was looking at what it would take to get multiarch working, but Raspbian and Debian don't mix well in that context.

Can't speak for docker specifically, but systemd-nspawn and plain chroot work fine.

[dom]

The instructions here provide a pretty simple way to install and run 64-bit debian packages on raspbian.

[DougieLawson]

The instructions here provide a pretty simple way to install and run 64-bit debian packages on raspbian.

Better option would be Sakaki-san's Raspbian 64 https://www.raspberrypi.org/forums/view ... 1#p1534981

[dom]

Better option would be Sakaki-san's Raspbian 64 https://www.raspberrypi.org/forums/view ... 1#p1534981

Yes, that will get you the same rpi-update kernel described in this thread and has a lot of the configuration for 64-bit debian handled for you.
Recommended if you want to start with a fresh image. If you have an existing image you want to experiment with then the linked thread gives a simple (copy/paste a few commands and wait ten minutes) way of testing some 64-bit packages.

[jdonald]

dom's succinct guidelines for how to get a 64-bit user environment are perfect.

@ShiftPlusOne thanks for taking a look and running more benchmark experiments.

With the 64-bit kernel active "uname -m" reports aarch64 even though one has a 32-but userland. I believe I have a case where this really muddles autoconf in libffi as well as my own configure.ac scripts that try to put built files somewhere related to the architecture in use. I am working to patch that for myself by making a /usr/local/bin/uname to take precedence over /bin/uname and return what I expected on a 32-bit platform

In the past I've also used a uname wrapper to deal with third-party software that runs into this problem.

If you're trying to make your own software robust against such configurations, here are two ways to query the userland architecture:

Code: Select all

dpkg --print-architecture

or

Code: Select all

cc -dumpmachine

dpkg has a shortcoming that it isn't portable to systems using yum or other package managers, while -dumpmachine can have other quirks like returning different answers from gcc vs clang.

For Ubuntu, I saw on Launchpad at one point they were building armhf packages on a system with an aarch64 kernel, so apparently they've solved this for most packages. Perhaps the debian/rules are often doing ./configure --host=$(gcc -dumpmachine)

also can I use docker 64bit version within raspbian now with the new kernel? or can a arm 64bit container work like https://hub.docker.com/layers/ownyourbi ... e55c0a761b

Docker alone doesn't work out of the box for jumping from 32-bit userland to 64-bit. I reported this issue at some point: https://forums.docker.com/t/unable-to-r ... cker/67187
I'm now seeing that was more than 3.5 years ago, so I suspect it's not trivial to fix. For some reason systemd-nspawn, LXC, and chroot just work while Docker doesn't.

The workaround I have for using aarch64 Docker images on Raspbian is to first set up an arm64 chroot with debootstrap then install and run docker-ce:arm64 in there.

I see the Docker Hub image you linked is using qemu-aarch64-static so it may not do what you expect. An example of a true aarch64 Docker image is arm64v8/debian.

[ShiftPlusOne]

Wrapping commands in 'setarch linux32' will work around the uname issue.

[sakaki]

Hello,

Slightly OT since the focus is on the Pi4, but: I have used this official 64-bit kernel (@ 4.19.69-v8+, with matching boot f/w, as pulled in by rpi-update) in my recently released raspbian-nspawn-64 image, so I've given it some testing on both an RPi4 and RPi3. Mostly, it seems to work well on both platforms; thanks for making it available!

One minor issue I encountered was that on the RPi3, while the path /dev/hwrng exists, there appears to be no functional driver for it, which causes the generator service rng-tools to fail. However, rng-tools starts up fine on the RPi4 (with the same kernel, same vanilla Raspbian 32-bit userland), and I've used the hwrng successfully on the RPi3 under Gentoo in 64-bit mode, so probably just a minor config or DT issue (CONFIG_HW_RANDOM_BCM2835 or such?)

Best, sakaki

[mitchejj]

The kernel headers are not wanted by most and are not trivial in size (around 8000 header files) so we are not going to include them by default with rpi-update. Stable kernels have kernel header files available from apt.

You might be interested in https://github.com/notro/rpi-source/wiki

rpi-source is exactly what I was hoping to find. As noted below it doesn’t support the 64-bit kernel

Now I’m heading down the rabbit hole to attempt to build a functional kernel in hopes have a better understanding of how the kernel builds and possible attempt to submit a patch to make rip-source work with the 64-bit kernel.

I think the last time I built my own kernel was circa Red Hat 6.0 w/ the 2.2 kernel.

[graphicw]

OpenGL performance decreases considerably with 64 bit kernel as is. Open Arena at 1080p was a slide show under the 64 bit whereas it performed at 20 fps and better with standard kernel. YouTube and such performed the same as standard kernel.