experimental support for multiarch with Raspbian and Debian arm64

[plugwash]

As some people here are probably aware Debian has a feature called "multiarch" that allows packages for multiple architectures to be installed on one system.

Unfortunately one of the rules of multiarch is that if a package is installed for multiple architectures (this is only possible for multi-arch:same packages, which mostly means libraries) the versions must match precisely. I understand why Debian made this design descision but it makes it tricky to multiarch between a parent distro and a derivative.

To solve this issue I have prepared a couple of repositories with arm64 builds of raspbian and raspberry pi foundation packages (and in one case a slightly modified version of a raspberry pi foundation package built for both armhf and arm64) that can be used to patch-over the version inconsistencies between Debian and Raspbian and hence allow a multiarch setup of raspbian with Debian arm64.

Before starting to install arm64 packages you should be running a 64-bit kernel. See https://www.raspberrypi.org/forums/view ... 9&t=250730 for more info on that.

setup instructions (not fully tested)

Code: Select all

sed -i 's/deb /deb [arch=armhf] /' /etc/apt/sources.list
sed -i 's/deb /deb [arch=armhf] /' /etc/apt/sources.list.d/raspi.list
echo deb [arch=arm64] http://ftp.uk.debian.org/debian buster main >> /etc/apt/sources.list
echo deb [arch=arm64] http://security.debian.org/ buster/updates main >> /etc/apt/sources.list
echo deb [arch=arm64] http://plugwash.raspbian.org/multiarch buster main >> /etc/apt/sources.list
echo deb http://plugwash.raspbian.org/multiarch-rpf buster main >> /etc/apt/sources.list.d/raspi.list
gpg --keyserver keys2.kfwebs.net --recv-keys 6D33866EDD8FFA41C0143AEDDCC9EFBF77E11517
gpg --export 6D33866EDD8FFA41C0143AEDDCC9EFBF77E11517 | apt-key add
gpg --keyserver keys2.kfwebs.net --recv-keys B7ED7C2CB9C95A8BDF94369CE77FC0EC34276B4B
gpg --export B7ED7C2CB9C95A8BDF94369CE77FC0EC34276B4B | apt-key add
apt-get update && apt-get install debian-archive-keyring

You should then be able to install arm64 packages, using for example "apt-get install iagno:armhf"

So far I have successfully installed arm64 versions of chromium , iagno, kfourinline and ffmpeg.

Notes:

* While libraries can usually be installed for multiple architectures, applications can normally only be installed for one architecture at a time.
* Sometimes you have to help apt along a bit with installing stuff by adding extra packages to the command line.
* kio can currently only be installed for one architecture at a time, which means for kde stuff you basically have to choose one architecture and will have to install kio:arm64 manually if you want to install arm64 kde stuff. This is not a raspbian-specific issue but a more general Debian one, I filed a bug at https://bugs.debian.org/cgi-bin/bugrepo ... bug=942129 .
* some packages simply don't support installing for a foreign architecture, most commonly because their "-data" package is arch all and not marked "multi-arch: foreign", again this is not a raspbian-specific issue.
* for now only buster is supported, I may or may not add support for bullseye once the infrastructure is more stable.

[fanoush]

wow, all that 64bit focus is great news, good stuff!

Before starting to install arm64 packages you should be running a 64-bit kernel. See https://www.raspberrypi.org/forums/view ... 9&t=250730 for more info on that.

The 4.19-75 64bit kernel is part of updated raspbian images, then the packages were removed from repo (due to /boot space issues) and now it is back(?). So aarch64/v8 kernel is now here to stay as part of stable regular updates? So in general there is now no need to do rpi-udate anymore to get 64bit kernel as that linked kernel thread suggests in first post? All installations with enough space in /boot are now getting 64bit kernel by default no matter which of 32/64bit one is running at the time of upgrade?

* While libraries can usually be installed for multiple architectures, applications can normally only be installed for one architecture at a time.

What is 'normal' for core parts of system in such setup? Is multiarch designed to be agnostic in such a way that you can e.g. have systemd aarch64 and the rest 32bit? I am asking because first thing for me would be to try to get aarch64 docker running and keep most of the system 32 bit. Not sure if multiarch is supposed to be used like this - replace random part of relatively low level services. Debian howto talks about 'foreign' architecture packages so aarch64 vs armv7l are not equal in multiarch system? Is it just that one is default if not specified or there is more to it?

[plugwash]

A multiarch system has a "native" architecture and one of more "foreign" architectures. I am aware of two differences between "native" and "foreign" but there may be others i'm not aware of.

1. The native architecture doesn't have to be specified in the UI of tools like apt.
2. Arch all packages are treated like packages from the native architecture. This means they can only satisfy foreign-architecture dependencies if they are marked "Multi-Arch: foreign". This is one of the biggest reasons why a lot of software can't be installed for foreign architectures.

I just tried typing "apt-get install systemd:arm64" and it proposed a solution (removing systemd for armhf, installing systemd for arm64 and installing some arm64 libraries), I didn't try actually saying yes to it though. Feel free to try it on a system you don't care about and report what happens.

[Kradenko]

Hi

Been watching the forums like a hawk regarding the 64Bit Rasbian OS which is on the way. Saw this post. I have been struggling to compile items due to some or other limit on memory allocation to processes. Here is my original thread: https://www.raspberrypi.org/forums/view ... p?t=253278

Would what you discuss above solve that issue? Or do I have a kernel or some library config issue?

Thanks!

[dickon]

Just come across this post. I'm having trouble resolving plugwash.debian.org:

Host plugwash.debian.org not found: 3(NXDOMAIN)

which is annoying.

Would what you discuss above solve that issue? Or do I have a kernel or some library config issue?

You said you'd had success on Gentoo; I can't imagine things would be that different on a debian/arm64 build.

[plugwash]

sorry that was a typo, while translating the sources.list files from my systems to a series of commands people can run on their system, it was meant to be plugwash.raspbian.org.

[Gavinmc42]

I finally got around to trying a chroot of Raspbian lite on Gentoo64 last night following Sakaki's instructions.
https://github.com/sakaki-/gentoo-on-rp ... too-System.
Turns out to be really easy to run arm32 on aarch64 if someone writes a wiki to follow.

Those instructions could probably be turned into an app that runs some scripts to load any Pi system.
Sure it is easy to swap OS by shutting down and swapping uSD and rebooting.
But wouldn't it be nice to be running an OS and just put another one in a USB port and run stuff from that one.

I currently use Gentoo64, Raspbian/ R Lite, PiCore, Buildroot and baremetal.
Being able to work on any of them from one OS would save some time.
Some space is saved by having the OS and apps on another USB drive that can be swapped in and out when needed.
Using 32bit OS/apps is needed sometimes as not everything is 64bit yet and some probably won't ever be.

Cross compiling is a bit of a pain at the moment, but native compiling just works most of the time.
Trying to have one OS with all the compilers and cross compilers on it is tricky.
Bit of an all the eggs in one basket situation.
Added to that is the Pi4 can run so many more apps now that very quickly 32/64 GB uSDs fill up.

This is all new to me but is there an app already that allows one button mounting of OS's from an USB stick?
I suppose network drive mounting would work too, but a bunch of USB sticks is easier to use and carry around.

[gishten]

I'm having trouble with adding your key and I'd appreciate your help.

Code: Select all

gpg --keyserver the.earth.li --recv-keys 6D33866EDD8FFA41C0143AEDDCC9EFBF77E11517
gpg: no valid OpenPGP data found.
gpg: Total number processed: 0

[dickon]

Looks like the is having a bad day. Try keys.gnupg.net instead; that's currently working for me.

[gishten]

Looks like the is having a bad day. Try keys.gnupg.net instead; that's currently working for me.

That worked for the first key but the second seems to not be on there.

Code: Select all

gpg --keyserver keys.gnupg.net --recv-keys B7ED7C2CB9C95A8BDF94369CE77FC0EC34276B4B
gpg: keyserver receive failed: No data

[plugwash]

I understand there has been some abuse of the keyserver network in recent times which has made key propagation slow and/or unreliable. Something seems to be up with "the.earth.li", so I just pushed the multiarch key to "keys2.kfwebs.net" and have edited the original post to reflect that.

(keys.gnupg.net is it's a round robin, so if there are propagation issues you aren't gauranteed to get a server that has the key you want)

[dickon]

Now *that's* what I'm talking about..[0]:

Code: Select all

root@pi4:~# file /bin/ls
/bin/ls: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 3.2.0, BuildID[sha1]=67a394390830ea3ab4e83b5811c66fea9784ee69, stripped
root@pi4:~# cat test.c
#include <stdio.h>

int main(void)
{
	printf("Hello, world! %d, %d, %d\n", sizeof(int), sizeof(long), sizeof(long long));
	return 0;
}
root@pi4:~# gcc test.c
root@pi4:~# file a.out 
a.out: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=431f2894208429a5dd36d1cd2c4bbac0f696060f, not stripped
root@pi4:~# ./a.out 
Hello, world! 4, 8, 8
root@pi4:~# which gcc
/usr/bin/gcc
root@pi4:~# dpkg -S /usr/bin/gcc
gcc:arm64: /usr/bin/gcc
root@pi4:~# 

*Very* cute.

[0] in https://www.raspberrypi.org/forums/view ... 5#p1541047

[Edit: to add the which gcc bits onwards.]

[dickon]

Mostly seems to work nicely.

I thought I'd try the opposite: install a debian/arm64 userland, and add this Raspbian repo as a foreign architecture. All went swimmingly, until I attempted to install pi-bluetooth. That package depends on raspberrypi-sys-mods, which is fine, and installs, but then I get this:

Code: Select all

root@rtr-dickonh:/# dpkg --print-foreign-architectures
armhf
root@rtr-dickonh:/# apt install pi-bluetooth
Reading package lists... Done
Building dependency tree       
Reading state information... Done
pi-bluetooth is already the newest version (0.1.11).
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
 pi-bluetooth : Depends: raspberrypi-sys-mods but it is not installable
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).
root@rtr-dickonh:/# apt install raspberrypi-sys-mods
Reading package lists... Done
Building dependency tree       
Reading state information... Done
raspberrypi-sys-mods:armhf is already the newest version (20190730).
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
 pi-bluetooth : Depends: raspberrypi-sys-mods but it is not installable
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).
root@rtr-dickonh:/# dpkg --print-architecture 
arm64
root@rtr-dickonh:/# 

I'm a bit stumped. Any ideas?

[plugwash]

You are running into one of the "design limitations" of Debian's multiarch.

Debian has (and has had for a very long time, long before multiarch existed) a concept of architecture independent (aka arch "all") binary packages. In multi-arch such packages are treated as packages of the system's native architecture. pi-bluetooth is arch all, rasperrypi-sys-mods is arch armhf, so assuming the packages have no multi-arch annotations (I haven't checked, but I strongly suspect they don't) the dependency is not satisfied on a system whose "native architecture" is arm64.

I think the way forward for this one is to provide an arm64 version of raspberrypi-sys-mods and possiblly make it "multi-arch: foreign"

[dickon]

That did it, ta.

Cloning https://github.com/RPi-Distro/raspberry ... ter/debian and issuing

Code: Select all

dpkg-buildpackage --target-arch arm64 --host-arch arm64

has done the trick.

[plugwash]

arm64 raspberrypi-sys-mods is now in the repo.

[dickon]

Outstanding, thanks.

I'm currently watching a video via a 32b Raspbian omxplayer on an aarch64 Debian userland. As I expected, this isn't difficult. Mostly, I've done it as a proof-of-concept, and, as far as I need it to work, it works.

Still mildly stunned at your (collective) responsiveness. Thanks very much.

[HawaiianPi]

Hmm... was trying to set this up and it seemed to be working until the last command. Anyone know why I got this error?

Code: Select all

W: GPG error: http://security.debian.org buster/updates InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY AA8E81B4331F7F50 NO_PUBKEY 112695A0E562B32A

Got it on the last step of setup instructions.

[rpdom]

Hmm... was trying to set this up and it seemed to be working until the last command. Anyone know why I got this error?

The error is because the debian-archive-keyring package isn't installed, so you can't do an update to enable to install debian-archive-keyring. A vicious circle.

You could try downloading the .deb file directly from Debian and installing it with dpkg

Code: Select all

wget http://ftp.debian.org/debian/pool/main/d/debian-archive-keyring/debian-archive-keyring_2019.1_all.deb
dpkg -i debian-archive-keyring_2019.1_all.deb

That should work. It's a bit messy and the file name will change at some point in the future.

[HawaiianPi]

The error is because the debian-archive-keyring package isn't installed, so you can't do an update to enable to install debian-archive-keyring. A vicious circle.

Ah, I see that raspbian-archive-keyring is installed by default.

EDIT:
Thanks for the info. That fixed it.

[dickon]

arm64 raspberrypi-sys-mods is now in the repo.

Looks like I need the same for libraspberrypi-bin and raspberrypi-bootloader, but I can't see any upstream packaging tools for them. The repo at https://github.com/raspberrypi/firmware is the one mentioned as source for libraspberrypi-bin, but doesn't contain any debian directories.

[joongul]

I am in the middle of converting to arm64 as much as possible without apt errors. Recently I get the following messages (this was from systemctl status). How can I fix it?

ERROR: ld.so: object '/usr/lib/arm-linux-gnueabihf/libarmmem-${PLATFORM}.so' from /etc/ld.so.preload cannot be preloaded (wrong ELF class: ELFCLASS32): ignored.

[ShiftPlusOne]

remove that line from /etc/ld.so.preload

[ulph0]

remove that line from /etc/ld.so.preload

thanks. error message is gone. i am trying to run minecraft with a 64 bit java. when i am trying now to run it with oracle-java, i get

Code: Select all

Exception in thread "main" java.awt.HeadlessException
	at java.awt.GraphicsEnvironment.checkHeadless(GraphicsEnvironment.java:204)
	at java.awt.Window.<init>(Window.java:536)
	at java.awt.Frame.<init>(Frame.java:420)
	at javax.swing.JFrame.<init>(JFrame.java:233)
	at net.minecraft.bootstrap.Bootstrap.<init>(Bootstrap.java:45)
	at net.minecraft.bootstrap.Bootstrap.main(Bootstrap.java:378)

if i try to install 64 bit openjdk, sooner or later i hit a depending package not in repository so that i can't fully install it.

[HawaiianPi]

Is there any way to get 64-bit Java working with multiarch, or is that a dead end?

I haved the 64-bit kernel and multiarch support installed, but trying to install openjdk-8-jre:armhf ends in error.

tim@RaspberryPi4B4:~ $ uname -r
4.19.75-v8+

Code: Select all

sudo apt install openjdk-8-jre:armhf
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  ca-certificates-java fonts-dejavu-extra java-common libatk-wrapper-java libatk-wrapper-java-jni
  openjdk-8-jre-headless
Suggested packages:
  default-jre icedtea-8-plugin fonts-ipafont-gothic fonts-ipafont-mincho fonts-wqy-microhei fonts-wqy-zenhei
  fonts-indic
The following NEW packages will be installed:
  ca-certificates-java fonts-dejavu-extra java-common libatk-wrapper-java libatk-wrapper-java-jni openjdk-8-jre
  openjdk-8-jre-headless
0 upgraded, 7 newly installed, 0 to remove and 0 not upgraded.
Need to get 27.6 MB of archives.
After this operation, 99.6 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://mirror.web-ster.com/raspbian/raspbian buster/main armhf java-common all 0.71 [14.4 kB]
Get:2 http://mirror.web-ster.com/raspbian/raspbian buster/main armhf openjdk-8-jre-headless armhf 8u212-b01-1+rpi1 [25.5 MB]
Get:3 http://mirror.web-ster.com/raspbian/raspbian buster/main armhf ca-certificates-java all 20190405 [15.7 kB]
Get:4 http://mirror.web-ster.com/raspbian/raspbian buster/main armhf fonts-dejavu-extra all 2.37-1 [1,982 kB]
Get:5 http://mirror.web-ster.com/raspbian/raspbian buster/main armhf libatk-wrapper-java all 0.33.3-22 [45.3 kB]
Get:6 http://mirror.web-ster.com/raspbian/raspbian buster/main armhf libatk-wrapper-java-jni armhf 0.33.3-22 [33.7 kB]
Get:7 http://mirror.web-ster.com/raspbian/raspbian buster/main armhf openjdk-8-jre armhf 8u212-b01-1+rpi1 [61.8 kB]
Fetched 27.6 MB in 3s (8,248 kB/s)
Selecting previously unselected package java-common.
(Reading database ... 97894 files and directories currently installed.)
Preparing to unpack .../0-java-common_0.71_all.deb ...
Unpacking java-common (0.71) ...
Selecting previously unselected package openjdk-8-jre-headless:armhf.
Preparing to unpack .../1-openjdk-8-jre-headless_8u212-b01-1+rpi1_armhf.deb ...
Unpacking openjdk-8-jre-headless:armhf (8u212-b01-1+rpi1) ...
Selecting previously unselected package ca-certificates-java.
Preparing to unpack .../2-ca-certificates-java_20190405_all.deb ...
Unpacking ca-certificates-java (20190405) ...
Selecting previously unselected package fonts-dejavu-extra.
Preparing to unpack .../3-fonts-dejavu-extra_2.37-1_all.deb ...
Unpacking fonts-dejavu-extra (2.37-1) ...
Selecting previously unselected package libatk-wrapper-java.
Preparing to unpack .../4-libatk-wrapper-java_0.33.3-22_all.deb ...
Unpacking libatk-wrapper-java (0.33.3-22) ...
Selecting previously unselected package libatk-wrapper-java-jni:armhf.
Preparing to unpack .../5-libatk-wrapper-java-jni_0.33.3-22_armhf.deb ...
Unpacking libatk-wrapper-java-jni:armhf (0.33.3-22) ...
Selecting previously unselected package openjdk-8-jre:armhf.
Preparing to unpack .../6-openjdk-8-jre_8u212-b01-1+rpi1_armhf.deb ...
Unpacking openjdk-8-jre:armhf (8u212-b01-1+rpi1) ...
Setting up java-common (0.71) ...
Setting up fonts-dejavu-extra (2.37-1) ...
Setting up libatk-wrapper-java (0.33.3-22) ...
Setting up libatk-wrapper-java-jni:armhf (0.33.3-22) ...
Setting up ca-certificates-java (20190405) ...
head: cannot open '/etc/ssl/certs/java/cacerts' for reading: No such file or directory
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGBUS (0x7) at pc=0xf5d33828, pid=4316, tid=0xf59de460
#
# JRE version:  (8.0_212-b01) (build )
# Java VM: OpenJDK Client VM (25.212-b01 mixed mode linux-aarch32 )
# Problematic frame:
# V  [libjvm.so+0x15e828]
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# //hs_err_pid4316.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#
/var/lib/dpkg/info/ca-certificates-java.postinst: line 88:  4315 Done                    echo -e "-diginotar_root_ca\n-diginotar_root_ca_pem"
      4316 Aborted                 | java -Xmx64m -jar $JAR -storepass "$storepass"
dpkg: error processing package ca-certificates-java (--configure):
 installed ca-certificates-java package post-installation script subprocess returned error exit status 134
dpkg: dependency problems prevent configuration of openjdk-8-jre-headless:armhf:
 openjdk-8-jre-headless:armhf depends on ca-certificates-java; however:
  Package ca-certificates-java is not configured yet.

dpkg: error processing package openjdk-8-jre-headless:armhf (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of openjdk-8-jre:armhf:
 openjdk-8-jre:armhf depends on openjdk-8-jre-headless (= 8u212-b01-1+rpi1); however:
  Package openjdk-8-jre-headless:armhf is not configured yet.

dpkg: error processing package openjdk-8-jre:armhf (--configure):
 dependency problems - leaving unconfigured
Processing triggers for ca-certificates (20190110) ...
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGBUS (0x7) at pc=0xf61d6828, pid=4997, tid=0xf5e81460
#
# JRE version:  (8.0_212-b01) (build )
# Java VM: OpenJDK Client VM (25.212-b01 mixed mode linux-aarch32 )
# Problematic frame:
# V  [libjvm.so+0x15e828]
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /etc/ssl/certs/hs_err_pid4997.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#
Aborted
E: /etc/ca-certificates/update.d/jks-keystore exited with code 1.
done.
Processing triggers for fontconfig (2.13.1-2) ...
Processing triggers for desktop-file-utils (0.23-4) ...
Processing triggers for mime-support (3.62) ...
Processing triggers for hicolor-icon-theme (0.17-2) ...
Processing triggers for gnome-menus (3.31.4-3) ...
Processing triggers for libc-bin (2.28-10+rpi1) ...
Processing triggers for man-db (2.8.5-2) ...
Errors were encountered while processing:
 ca-certificates-java
 openjdk-8-jre-headless:armhf
 openjdk-8-jre:armhf
E: Sub-process /usr/bin/dpkg returned an error code (1)
tim@RaspberryPi4B4:~ $ java -version
-bash: java: command not found

So I tried grabbing the Oracle Java tarballs, and the 32-bit version works, but the 64-bit version isn't recognized as executable.
jdk-8u231-linux-arm32-vfp-hflt.tar.gz
jdk-8u231-linux-arm64-vfp-hflt.tar.gz

Files were extracted to /opt/Java/ARM32 (and ARM64)

Testing the 32-bit version works as expected.
tim@RaspberryPi4B4:~ $ /opt/Java/ARM32/bin/java -version
java version "1.8.0_231"
Java(TM) SE Runtime Environment (build 1.8.0_231-b11)
Java HotSpot(TM) Server VM (build 25.231-b11, mixed mode)

But the 64-bit version fails.
tim@RaspberryPi4B4:~ $ /opt/Java/ARM64/bin/java -version
bash: /opt/Java/ARM64/bin/java: No such file or directory

And it's definitely there.

javaarmhf64.gif (30.19 KiB) Viewed 7818 times