experimental support for multiarch with Raspbian and Debian arm64

[plugwash]

As some people here are probably aware Debian has a feature called "multiarch" that allows packages for multiple architectures to be installed on one system.

Unfortunately one of the rules of multiarch is that if a package is installed for multiple architectures (this is only possible for multi-arch:same packages, which mostly means libraries) the versions must match precisely. I understand why Debian made this design descision but it makes it tricky to multiarch between a parent distro and a derivative.

To solve this issue I have prepared a couple of repositories with arm64 builds of raspbian and raspberry pi foundation packages (and in one case a slightly modified version of a raspberry pi foundation package built for both armhf and arm64) that can be used to patch-over the version inconsistencies between Debian and Raspbian and hence allow a multiarch setup of raspbian with Debian arm64.

Before starting to install arm64 packages you should be running a 64-bit kernel. See https://www.raspberrypi.org/forums/view ... 9&t=250730 for more info on that.

setup instructions (not fully tested)

Code: Select all

sed -i 's/deb /deb [arch=armhf] /' /etc/apt/sources.list
sed -i 's/deb /deb [arch=armhf] /' /etc/apt/sources.list.d/raspi.list
echo deb [arch=arm64] http://ftp.uk.debian.org/debian buster main >> /etc/apt/sources.list
echo deb [arch=arm64] http://security.debian.org/ buster/updates main >> /etc/apt/sources.list
echo deb [arch=arm64] http://plugwash.raspbian.org/multiarch buster main >> /etc/apt/sources.list
echo deb http://plugwash.raspbian.org/multiarch-rpf buster main >> /etc/apt/sources.list.d/raspi.list
gpg --keyserver keys2.kfwebs.net --recv-keys 6D33866EDD8FFA41C0143AEDDCC9EFBF77E11517
gpg --export 6D33866EDD8FFA41C0143AEDDCC9EFBF77E11517 | apt-key add
gpg --keyserver keys2.kfwebs.net --recv-keys B7ED7C2CB9C95A8BDF94369CE77FC0EC34276B4B
gpg --export B7ED7C2CB9C95A8BDF94369CE77FC0EC34276B4B | apt-key add
apt-get update && apt-get install debian-archive-keyring

You should then be able to install arm64 packages, using for example "apt-get install iagno:armhf"

So far I have successfully installed arm64 versions of chromium , iagno, kfourinline and ffmpeg.

Notes:

* While libraries can usually be installed for multiple architectures, applications can normally only be installed for one architecture at a time.
* Sometimes you have to help apt along a bit with installing stuff by adding extra packages to the command line.
* kio can currently only be installed for one architecture at a time, which means for kde stuff you basically have to choose one architecture and will have to install kio:arm64 manually if you want to install arm64 kde stuff. This is not a raspbian-specific issue but a more general Debian one, I filed a bug at https://bugs.debian.org/cgi-bin/bugrepo ... bug=942129 .
* some packages simply don't support installing for a foreign architecture, most commonly because their "-data" package is arch all and not marked "multi-arch: foreign", again this is not a raspbian-specific issue.
* for now only buster is supported, I may or may not add support for bullseye once the infrastructure is more stable.

[fanoush]

wow, all that 64bit focus is great news, good stuff!

Before starting to install arm64 packages you should be running a 64-bit kernel. See https://www.raspberrypi.org/forums/view ... 9&t=250730 for more info on that.

The 4.19-75 64bit kernel is part of updated raspbian images, then the packages were removed from repo (due to /boot space issues) and now it is back(?). So aarch64/v8 kernel is now here to stay as part of stable regular updates? So in general there is now no need to do rpi-udate anymore to get 64bit kernel as that linked kernel thread suggests in first post? All installations with enough space in /boot are now getting 64bit kernel by default no matter which of 32/64bit one is running at the time of upgrade?

* While libraries can usually be installed for multiple architectures, applications can normally only be installed for one architecture at a time.

What is 'normal' for core parts of system in such setup? Is multiarch designed to be agnostic in such a way that you can e.g. have systemd aarch64 and the rest 32bit? I am asking because first thing for me would be to try to get aarch64 docker running and keep most of the system 32 bit. Not sure if multiarch is supposed to be used like this - replace random part of relatively low level services. Debian howto talks about 'foreign' architecture packages so aarch64 vs armv7l are not equal in multiarch system? Is it just that one is default if not specified or there is more to it?

[plugwash]

A multiarch system has a "native" architecture and one of more "foreign" architectures. I am aware of two differences between "native" and "foreign" but there may be others i'm not aware of.

1. The native architecture doesn't have to be specified in the UI of tools like apt.
2. Arch all packages are treated like packages from the native architecture. This means they can only satisfy foreign-architecture dependencies if they are marked "Multi-Arch: foreign". This is one of the biggest reasons why a lot of software can't be installed for foreign architectures.

I just tried typing "apt-get install systemd:arm64" and it proposed a solution (removing systemd for armhf, installing systemd for arm64 and installing some arm64 libraries), I didn't try actually saying yes to it though. Feel free to try it on a system you don't care about and report what happens.

[Kradenko]

Hi

Been watching the forums like a hawk regarding the 64Bit Rasbian OS which is on the way. Saw this post. I have been struggling to compile items due to some or other limit on memory allocation to processes. Here is my original thread: https://www.raspberrypi.org/forums/view ... p?t=253278

Would what you discuss above solve that issue? Or do I have a kernel or some library config issue?

Thanks!

[dickon]

Just come across this post. I'm having trouble resolving plugwash.debian.org:

Host plugwash.debian.org not found: 3(NXDOMAIN)

which is annoying.

Would what you discuss above solve that issue? Or do I have a kernel or some library config issue?

You said you'd had success on Gentoo; I can't imagine things would be that different on a debian/arm64 build.

[plugwash]

sorry that was a typo, while translating the sources.list files from my systems to a series of commands people can run on their system, it was meant to be plugwash.raspbian.org.

[Gavinmc42]

I finally got around to trying a chroot of Raspbian lite on Gentoo64 last night following Sakaki's instructions.
https://github.com/sakaki-/gentoo-on-rp ... too-System.
Turns out to be really easy to run arm32 on aarch64 if someone writes a wiki to follow.

Those instructions could probably be turned into an app that runs some scripts to load any Pi system.
Sure it is easy to swap OS by shutting down and swapping uSD and rebooting.
But wouldn't it be nice to be running an OS and just put another one in a USB port and run stuff from that one.

I currently use Gentoo64, Raspbian/ R Lite, PiCore, Buildroot and baremetal.
Being able to work on any of them from one OS would save some time.
Some space is saved by having the OS and apps on another USB drive that can be swapped in and out when needed.
Using 32bit OS/apps is needed sometimes as not everything is 64bit yet and some probably won't ever be.

Cross compiling is a bit of a pain at the moment, but native compiling just works most of the time.
Trying to have one OS with all the compilers and cross compilers on it is tricky.
Bit of an all the eggs in one basket situation.
Added to that is the Pi4 can run so many more apps now that very quickly 32/64 GB uSDs fill up.

This is all new to me but is there an app already that allows one button mounting of OS's from an USB stick?
I suppose network drive mounting would work too, but a bunch of USB sticks is easier to use and carry around.

[gishten]

I'm having trouble with adding your key and I'd appreciate your help.

Code: Select all

gpg --keyserver the.earth.li --recv-keys 6D33866EDD8FFA41C0143AEDDCC9EFBF77E11517
gpg: no valid OpenPGP data found.
gpg: Total number processed: 0

[dickon]

Looks like the is having a bad day. Try keys.gnupg.net instead; that's currently working for me.

[gishten]

Looks like the is having a bad day. Try keys.gnupg.net instead; that's currently working for me.

That worked for the first key but the second seems to not be on there.

Code: Select all

gpg --keyserver keys.gnupg.net --recv-keys B7ED7C2CB9C95A8BDF94369CE77FC0EC34276B4B
gpg: keyserver receive failed: No data

[plugwash]

I understand there has been some abuse of the keyserver network in recent times which has made key propagation slow and/or unreliable. Something seems to be up with "the.earth.li", so I just pushed the multiarch key to "keys2.kfwebs.net" and have edited the original post to reflect that.

(keys.gnupg.net is it's a round robin, so if there are propagation issues you aren't gauranteed to get a server that has the key you want)

[dickon]

Now *that's* what I'm talking about..[0]:

Code: Select all

[email protected]:~# file /bin/ls
/bin/ls: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 3.2.0, BuildID[sha1]=67a394390830ea3ab4e83b5811c66fea9784ee69, stripped
[email protected]:~# cat test.c
#include <stdio.h>

int main(void)
{
	printf("Hello, world! %d, %d, %d\n", sizeof(int), sizeof(long), sizeof(long long));
	return 0;
}
[email protected]:~# gcc test.c
[email protected]:~# file a.out 
a.out: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=431f2894208429a5dd36d1cd2c4bbac0f696060f, not stripped
[email protected]:~# ./a.out 
Hello, world! 4, 8, 8
[email protected]:~# which gcc
/usr/bin/gcc
[email protected]:~# dpkg -S /usr/bin/gcc
gcc:arm64: /usr/bin/gcc
[email protected]:~# 

*Very* cute.

[0] in https://www.raspberrypi.org/forums/view ... 5#p1541047

[Edit: to add the which gcc bits onwards.]

[dickon]

Mostly seems to work nicely.

I thought I'd try the opposite: install a debian/arm64 userland, and add this Raspbian repo as a foreign architecture. All went swimmingly, until I attempted to install pi-bluetooth. That package depends on raspberrypi-sys-mods, which is fine, and installs, but then I get this:

Code: Select all

[email protected]:/# dpkg --print-foreign-architectures
armhf
[email protected]:/# apt install pi-bluetooth
Reading package lists... Done
Building dependency tree       
Reading state information... Done
pi-bluetooth is already the newest version (0.1.11).
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
 pi-bluetooth : Depends: raspberrypi-sys-mods but it is not installable
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).
[email protected]:/# apt install raspberrypi-sys-mods
Reading package lists... Done
Building dependency tree       
Reading state information... Done
raspberrypi-sys-mods:armhf is already the newest version (20190730).
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
 pi-bluetooth : Depends: raspberrypi-sys-mods but it is not installable
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).
[email protected]:/# dpkg --print-architecture 
arm64
[email protected]:/# 

I'm a bit stumped. Any ideas?

[plugwash]

You are running into one of the "design limitations" of Debian's multiarch.

Debian has (and has had for a very long time, long before multiarch existed) a concept of architecture independent (aka arch "all") binary packages. In multi-arch such packages are treated as packages of the system's native architecture. pi-bluetooth is arch all, rasperrypi-sys-mods is arch armhf, so assuming the packages have no multi-arch annotations (I haven't checked, but I strongly suspect they don't) the dependency is not satisfied on a system whose "native architecture" is arm64.

I think the way forward for this one is to provide an arm64 version of raspberrypi-sys-mods and possiblly make it "multi-arch: foreign"

[dickon]

That did it, ta.

Cloning https://github.com/RPi-Distro/raspberry ... ter/debian and issuing

Code: Select all

dpkg-buildpackage --target-arch arm64 --host-arch arm64

has done the trick.

[plugwash]

arm64 raspberrypi-sys-mods is now in the repo.

[dickon]

Outstanding, thanks.

I'm currently watching a video via a 32b Raspbian omxplayer on an aarch64 Debian userland. As I expected, this isn't difficult. Mostly, I've done it as a proof-of-concept, and, as far as I need it to work, it works.

Still mildly stunned at your (collective) responsiveness. Thanks very much.

[HawaiianPi]

Hmm... was trying to set this up and it seemed to be working until the last command. Anyone know why I got this error?

Code: Select all

W: GPG error: http://security.debian.org buster/updates InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY AA8E81B4331F7F50 NO_PUBKEY 112695A0E562B32A

Got it on the last step of setup instructions.

[rpdom]

Hmm... was trying to set this up and it seemed to be working until the last command. Anyone know why I got this error?

The error is because the debian-archive-keyring package isn't installed, so you can't do an update to enable to install debian-archive-keyring. A vicious circle.

You could try downloading the .deb file directly from Debian and installing it with dpkg

Code: Select all

wget http://ftp.debian.org/debian/pool/main/d/debian-archive-keyring/debian-archive-keyring_2019.1_all.deb
dpkg -i debian-archive-keyring_2019.1_all.deb

That should work. It's a bit messy and the file name will change at some point in the future.

[HawaiianPi]

The error is because the debian-archive-keyring package isn't installed, so you can't do an update to enable to install debian-archive-keyring. A vicious circle.

Ah, I see that raspbian-archive-keyring is installed by default.

EDIT:
Thanks for the info. That fixed it.

[dickon]

arm64 raspberrypi-sys-mods is now in the repo.

Looks like I need the same for libraspberrypi-bin and raspberrypi-bootloader, but I can't see any upstream packaging tools for them. The repo at https://github.com/raspberrypi/firmware is the one mentioned as source for libraspberrypi-bin, but doesn't contain any debian directories.

[joongul]

I am in the middle of converting to arm64 as much as possible without apt errors. Recently I get the following messages (this was from systemctl status). How can I fix it?

ERROR: ld.so: object '/usr/lib/arm-linux-gnueabihf/libarmmem-${PLATFORM}.so' from /etc/ld.so.preload cannot be preloaded (wrong ELF class: ELFCLASS32): ignored.

[ShiftPlusOne]

remove that line from /etc/ld.so.preload