Page 1 of 1

Tutorial: How (and why!) to set up a 64-bit kernel, 32-bit Raspbian host OS, 64-bit nspawn Debian guest OS RPi3 system

Posted: Tue Jan 29, 2019 1:07 am
by sakaki

As maintainer of the gentoo-on-rpi3-64bit image, I not infrequently receive email requests from users who:
  • don't want to give up on their familiar Raspbian desktop, repos, and tools;
  • don't want to have to build packages from source (during setup or maintenance);
  • would like to retain the ability to easily upgrade their Raspbian system going forward;
  • but still need to run a few (possibly GUI-based) 64-bit-only packages on their RPi3.
Well, in this tutorial, I'll show how to create a system that can straightforwardly address these requirements. Specifically, and taking off my Gentoo hat for a second, I'm going to run through how to build a hybrid RPi3 system with:
  • a 64-bit kernel (+ module set), but
  • a vanilla 32-bit Raspbian userland OS, with additionally
  • a 64-bit userland 'guest' OS (here, Debian Stretch), booted inside a systemd-nspawn container [1].
This is possible because ARM-v8a processors allow a system booted in aarch64 mode (with an aarch64 kernel) to run both aarch64 and aarch32 userland processes [2].

Here's a screenshot of the hybrid system in use, on an RPi3B+:


If you'd like to try this out without having to build it first, I have a released a bootable RPi3 image on GitHub, here.

Think of this as vanilla 32-bit Raspbian with a 'sidekick' 64-bit Debian Stretch guest OS always available, which you can enter at will from a Raspbian terminal using the ds64-shell command. You can also start 64-bit apps inside the guest from a Raspbian terminal using the ds64-run command. 64-bit apps started in this fashion can display on the host desktop, play sound and video, and access user pi's home directory, but are prevented from performing many harmful actions on the host. Package management in the guest is regular apt-get, with the full Debian aarch64 repository available, and day-to-day Raspbian operation (with the exception of anything needing e.g. MMAL or OpenMAX IL) is pretty much per the stock image. WiFi, Bluetooth, I2C etc. are all available.

Acknowledgement: the idea of this came from Crazyhead90's Raspbian-desktop-64bitkernel-pi3bplus image (64-bit kernel + 32-bit userland OS), and this post by jdonald (which suggests a chroot into a 64-bit guest from such a setup). I've just extended jdonald's idea to use the Linux containers support provided by systemd-nspawn [1].

Benefits and Drawbacks

The benefits of this hybrid approach are:
  • The 'normal' desktop, package set and functionality of 32-bit Raspbian is retained. Bluetooth, WiFi etc. all work and can be managed using the stock, familiar tools - in fact, there is surprisingly little impact to Raspbian when booted under a 64-bit kernel (we will discuss what doesn't work shortly). Bluetooth, WiFi, I2C, SPI etc. all work just fine.
  • No Raspbian-package-supplied files get 'clobbered' during the hybridization (and that even includes the original 32-bit kernel and modules [9]), so updates and new package installation can be done just as before (via apt-get, or the GUI application). So, for the most part, you should be able to follow along with other tutorials targeted at mainstream Pi users and so forth.
  • As we'll set it up, the containerized guest OS shares the default user's home directory, and can launch graphical apps on the main desktop. You can just use ds64-run to start a 64-bit app, and ds64-shell to enter a 64-bit shell, right from your 32-bit Raspbian console (you can see examples of this in the screenshot above). This makes it easy to e.g. run a 64-bit program alongside your regular 32-bit ones, a use case that comes up frequently in threads and support emails. To be clear, we won't be using KVM here, although, as the 'host' kernel we will be using supports it, you could if you wished [3].
  • Since the guest OS we're going to use - viz., Debian Stretch - is what Raspbian is currently based on, package management inside the (64-bit) guest should feel very familiar (apt-get etc.) And since all package management in the host and guest is binary-based, there's no need to compile anything (unless you want to!)
  • Setup is quick; the whole thing can be built starting from a 'vanilla' Raspbian image in around an hour or two, without cross-compilation toolchains etc.
  • Reverting back to use your old (32-bit) kernel again, should you wish to do so, is an even faster process, and can be done on a temporary basis too (I'll provide instructions for that).
  • Similar to chroot, only parts of the host filesystem that are explicitly mapped will be visible within the guest; however, unlike chroot, the filesystem hierarchy is fully virtualized.
  • Also unlike a (vanilla) chroot, systemd-nspawn (via underlying Linux kernel containerization technology) limits the guest OS' access to various kernel interfaces to read-only (/sys, /proc/sys, /sys/fs/selinux), prevents it changing the host's system clock, creating new device nodes, loading kernel modules, restarting the host, etc..
  • Similarly, although the host OS will be able to 'see' processes within the guest, unlike a chroot the converse will not be true.
  • As we'll arrange it here, the guest OS will actually be booted, so, again unlike a chroot, it will have its own (systemd, of necessity) init process, and this boot will happen automatically at system startup. This makes it easy to run e.g. services that require scheduling within the guest (e.g. a 64-bit cronjob of some sort), and also enables software with complex system or dbus interactions (such as a full-scale web browser) to be launched within the guest.
  • The process is very efficient: at the time of writing, booting Debian Stretch in a container took only a few seconds on an RPi3B+, and the resulting guest system consumed less than 10MiB of system memory (prior to any user apps being launched, of course). So there is really no penalty attached to bringing up the container automatically at boot (which is what we will arrange to do).
  • You can also (advanced point) have multiple instances of different guest operating systems in use simultaneously, should you wish, and even use the same OS filesystem tree for multiple instances - thanks to namespacing, their processes will not clash (although obviously the root filesystem will be shared).
  • The guest OS can easily be started with an ephemeral filesystem, in which all changes (to the guest) are lost once it is restarted (also an advanced point).
  • You can also easily more fully isolate your guest processes from the main OS, should you want or need to do that (also an advanced point). We're going to use a privileged container, and share the host's networking, in what follows, but you can relatively easily migrate to a unprivileged container with isolated networking, should you so wish (each of these changes makes administration and package management somewhat harder inside the container, which is why we're not going to pursue them for the proof-of-concept system here.)
The main drawbacks of this approach are:
  • As with all 64-bit (kernel) systems at the moment, the MMAL and OpenMAX IL layers do not work when booted in this way. So, for example, gpu-based video decoding is unavailable [4]. However, the so-called ARM-side (aka 'GL') video drivers are available (vc4-{f,}kms-v3d), and we will leverage this in what follows.
  • Although the userland Raspbian part of your system will be basically vanilla, booting under a 64-bit kernel is not currently officially supported, so you may find it somewhat harder to get support on these forums etc., if you go this route.
  • You will have to keep the repos up to date separately for host (Raspbian) and guest (Debian) systems. However, since both are binary distros, this isn't an onerous task.
  • The protection put in place by systemd-nspawn can sometimes be a double-edged sword: if, for example, you want to run a service that does something forbidden. But such a case is rare (and you can always chroot, or reduce the protections by parameter).
  • Only the 'pi' user is set up for easy cross-container filesystem access: if you add a new user, the same tricks won't work for them. Also, the pi users in the host and guest OSes are distinct; so setting the password for pi in Raspbian won't affect that in the Debian guest, and vice versa.
  • This isn't a true multi-lib approach (as you might get on a Linux PC say, in which the co-location of 32-bit and 64-bit applications and libraries within the same filesystem hierarchy is relatively common). Readers wanting to a taste of what that might look like on Raspbian should check out jdonald's raspbian-multiarch script.
Finally, three points of order before we begin:
  • Make sure you really do need this before proceeding! If the official, vanilla Raspbian OS (under a 32-bit kernel) is working well for you, and runs all the packages you need, stick with it (unless you just want to experiment, of course). Similarly, if you want to run a 'pure' 64-bit system, or (natively) run a different disto (such as Gentoo or Arch), you should use the appropriate dedicated 64-bit OS image, rather than this hybrid. You can browse here, and here, for discussion of some of the alternatives.
  • This tutorial assumes some basic knowledge of the command line. If you'd like to skip the actual setup and just try it out, I have a demonstrator image available for download on GitHub here.
  • Don't try hybridizing the Raspbian image you currently use for day-to-day work, particularly if you have anything of value on there! Instead, store that somewhere safe, and build up this system on a second, spare microSD card. That way, you'll easily be able to swap back to your original system at any time should you need to do so, or should anything go wrong. I've tried to make the tutorial comprehensive and easy to follow, but it's provided 'as is' and without warranty - proceed at your own risk! Always ensure you are in compliance with necessary laws if you intend distributing your modified image.


To carry out the steps in this tutorial, you'll need:
  • A Raspberry Pi with a 64-bit capable processor. The RPi3B and RPi3B+ should both be fine, so I'll use the generic term 'RPi3' in what follows [5].
  • A spare microSD card, on which to build up the image. You will need a card of capacity >= 8GB.
  • Some way of writing an image to the card prior to first boot on the RPi. Most people will use a PC with an appropriate adaptor for this.
  • Working internet connection that your Pi will be able to use (wired or WiFi).
The process should no more than between one to two hours to complete end-to-end (depending on the speed of your internet connection, mostly).

OK, if you have everything ready, then let's start!

Setting up a baseline Raspbian System

Begin by downloading an official Raspbian image from this page.

At the time of writing, three Raspbian variants are available for downloading. I recommend the "Raspbian Stretch with desktop" one (and use that in what follows), as this boots to a graphical desktop (which most users will want) but ships with only a core set of applications preinstalled, which reduces the image size. You can of course install whatever additional packages you want once the system is up and running [6].

Once you have downloaded the compressed image, write it to your microSD card, following the official installation guide. For simplicity, do not use the NOOBS option mentioned on that page, but rather write the downloaded image directly to the card (the recommended program Etcher makes this process straightforward to perform safely [7]).

Once you have the image written, boot your RPi3 with it, and perform the usual first-run wizard-driven setup, setting user pi's password, configuring WiFi etc. as required, and finally rebooting. When the system comes back up, ensure you have network connectivity established (you can ping out, browse the web etc.). NB: in what follows, I'm going to assume you have not deleted or renamed the default pi user during initial setup.

Next, if you didn't elect to "Update Software" when prompted by the first-start wizard, do so now. Open a terminal window on your RPi3, and issue:

Code: Select all

[email protected]:~ $ sudo apt-get update && sudo apt-get -y upgrade
Wait for this to complete - it may take 5 to 15 minutes or so, depending on how far back in time, with respect to the current date, your target image was released, and the speed of your internet connection. Once done, reboot if you were prompted to do so.
If you did update software when the first-start wizard prompted you to, then the above will (almost surely) be a no-op and complete very quickly, but there is no harm in running it.

Once complete (and after rebooting again, if necessary), you should next switch the graphics driver over to the 'fake kms' GL variant, as this will provide reasonable hardware acceleration even when booted under a 64-bit kernel. To do so, open a terminal and issue:

Code: Select all

[email protected]:~ $ sudo raspi-config
Select "7 Advanced Options" then "A7 GL Driver" then "G2 GL (Fake KMS) OpenGL desktop driver with fake KMS". Exit the application, and choose Yes when prompted "Would you like to reboot now?". Wait for your RPi3 to come back up, and ensure everything still works.

You're now ready to add a 64-bit kernel!

Installing a 64-bit Kernel

Because the RPi3 SoC's CPU is an ARMv8a device, it can be booted up in either 32-bit or 64-bit mode. The former is used by stock Raspbian, but only 32-bit userland is supported in this mode. However, if started in 64-bit mode (with a 64-bit kernel), the system supports both 64-bit and 32-bit userlands.

We'll exploit this fact here, to enable us to get (mostly) the 'best of both worlds' - regular (32-bit userland) Raspbian for day-to-day use, with the ability to drop into a 64-bit userland Debian 'guest' OS for those troublesome '64-bit only' applications.

OK, so to move forward we need a 64-bit kernel (and module set) with an appropriate configuration for the RPi3. For this project, we will use the 64-bit bcmrpi3-kernel-bis package, which is automatically built and released once a week by a bot, for the gentoo-on-rpi3-64bit project. The released, prebuilt kernels are based upon the upstream "bcmrpi3_defconfig", so have all necessary Pi-specific options enabled, plus a few additions to support KVM etc. Don't worry, there's nothing inherently 'Gentoo-specific' about this kernel, and for avoidance of doubt, your original 32-bit Raspbian-based kernel (and module set) will not be affected by installing it (and may be rolled-back to in case of problems). As it is a binary package, you don't have to compile anything either [8].

To get it, open a terminal on your RPi3, and issue (as the regular pi user):

Code: Select all

[email protected]: ~ $ wget -c
(You can find the most up-to-date kernel package wget string here (it is automatically updated each week by the buildbot); but the above version should work fine.)

Now, 'hold' the existing 32-bit raspberrypi-kernel package, to prevent its files being modified during future "apt-get upgrade" runs (which could overwrite the dtbs we're about to install):

Code: Select all

[email protected]:~ $ sudo apt-mark hold raspberrypi-kernel
Now store off the existing device-tree blobs:

Code: Select all

[email protected]:~ $ sudo mkdir -pv /boot/r32-dtbs
[email protected]:~ $ sudo mv -v /boot/*.dtb /boot/r32-dtbs/
There is no need to back up the 32-bit kernel itself, nor its modules, as the names of these will not clash with their 64-bit counterparts.

With that done, you can safely deploy the 64-bit kernel! This simply requires an untar:

Code: Select all

[email protected]: ~ $ sudo tar --exclude='COPYING.linux' \
  -xJf bcmrpi3-kernel-bis- -C /
[email protected]: ~ $ sync
Note that this won't 'clobber' any existing files (other than the DTBs, which we just backed up) - it writes:
  • a bootable 64-bit kernel to /boot/kernel8.img;
  • the kernel's config to /boot/config;
  • the kernel's to symbol table to /boot/;
  • a set of DTBs to /boot/<devname>.dtb; and
  • a matching module set into /lib/modules/<kernel release name>.
Other than the DTBs, none of these will be present in the baseline 32-bit Raspbian image you have just installed.

Lastly, make a backup of the new device tree blobs too, for ease of rollback later:

Code: Select all

[email protected]:~ $ sudo mkdir -pv /boot/r64-dtbs
[email protected]:~ $ sudo cp -v /boot/*.dtb /boot/r64-dtbs/
That's it! You now have a 64-bit kernel (and corresponding module set) installed, and your RPi3 will automatically try to use in preference to the (Raspbian-supplied) 32-bit kernel, where present.

So next, reboot your RPi3. With luck, you should find it now starts up into the 64-bit kernel, and then continues into (32-bit userland, vanilla) Raspbian, just as before!

You may see the message "dmi: Firmware registration failed." displayed during early boot. This may safely be ignored.

Open a terminal, and check all is well:

Code: Select all

[email protected]:~ $ uname -a
[email protected]:~ $ file $(which ls)
[email protected]:~ $ lsmod
Verify that your WiFi, Bluetooth etc. all work as before (they should do); try opening the bundled (32-bit) Chromium web browser and so forth.

Here's a screenshot of an RPi3 B+ at this stage in the process (your lsmod output etc. may vary slightly):


Notice how the kernel ("uname -a") is aarch64, the vc4 driver module is loaded, and the userland (as shown, for example, by the "file $(which ls)" command) is aarch32.

So far, so good: we're now ready to install our Debian guest!

Installing 64-bit Debian Guest OS (Userspace)

Let's begin by installing some components (in our 32-bit Raspbian userland) to enable us to create a Debian instance [10]. Issue:

Code: Select all

[email protected]:~ $ wget -c
[email protected]:~ $ sha256sum debian-archive-keyring_2017.5_all.deb

6a38407c47fefad2d8459dc271d109f1841ee857f993ed3ce2884e33f7f0f734  debian-archive-keyring_2017.5_all.deb
Ensure the keyring checksum matches that shown above (which is taken from here). Then, if all is well, install it, and some other packages we'll need also:

Code: Select all

[email protected]:~ $ sudo apt install ./debian-archive-keyring_2017.5_all.deb
[email protected]:~ $ sudo apt-get -y install debootstrap systemd-container pulseaudio zenity
We manually install the Debian signature keyring, as by default Raspbian has remapped it to raspbian-archive-keyring. The package debootstrap provides a tool to create a Debian base system from scratch, without requiring dpkg or apt, by downloading .deb files from a mirror and directly unpacking them into a target directory. The systemd-container package provides systemd's tools for nspawn and container/VM management. And we'll need to be running pulseaudio on the host (Raspbian) OS, to allow apps running in the 64-bit container to play sound. zenity is useful to allow graphical error messages etc. to be displayed on the X server.

Now we can create the initial Debian filesystem. We'll do this in the special location /var/lib/machines, with top level directory name 'debian-stretch-64'. Issue:

Code: Select all

[email protected]:~ $ sudo mkdir -pv /var/lib/machines
[email protected]:~ $ sudo debootstrap --arch=arm64 --include=systemd-container,pulseaudio,zenity stretch \
There is no need to use qemu-debootstrap, --foreign etc. here since our 'real' host OS, the booted Linux kernel, is already aarch64, even though our primary userspace OS (i.e., the one containing the init process first started by that kernel) is a 32-bit (aarch32 userspace) Raspbian system. Note also that to allow booting, systemd-nspawn requires that the userland OS in the container has systemd as PID 1, and has systemd-nspawn installed. For that reason, we include the systemd-container package (which supplies systemd-nspawn) in the baseline set for stretch, above, and we also include the pulseaudio package (to allow audio playback via the host's pulseaudio server), and zenity (to display dialogs on the host).

The debootstrap above will pull across and install a full baseline package set for an aarch64 (aka arm64) Debian Stretch system, with root directory /var/lib/machines/debian-stretch-64/. It will take some time to complete, perhaps up to 30 minutes, depending on the speed of your internet connection, so please be patient.

You can select a different release keyword in place of "stretch" should you wish - see man debootstrap for further details. However, we'll stick with "stretch" in this walkthrough, as it matches the current Raspbian variant.

Once it completes (you should see a message "I: Base system installed successfully."), you can remove the debian-archive-keyring package if you like (this step is optional):

Code: Select all

[email protected]:~ $ sudo apt purge debian-archive-keyring

Booting the 64-bit Debian Guest OS

Now you can try starting up your new 64-bit (userspace) Debian OS! What we're about to do is quite different from a chroot - the OS will actually be booted in a new process namespace, with its (fresh) systemd instance PID 1 in that namespace.

However, before first boot, begin by setting a root password within the container (as Debian doesn't allow password-free first login). Issue:

Code: Select all

[email protected]:~ $ sudo systemd-nspawn --settings=no --directory=/var/lib/machines/debian-stretch-64

Spawning container debian-stretch-64 on /var/lib/machines/debian-stretch-64.
Press ^] three times within 1s to kill container.
[email protected]:~#
This commands starts a shell as pid 1 in the container, without fully booting it. Only certain software can safely be run at pid1; for others, you need to add the --as-pid2 option. We've used the full path to the directory here, but you can just use the --machine option instead; see later commands. We use --settings=no to ensure no additional configuration files are read. Note also that the internal hostname for the machine, as reported in the bash shell prompt, is still 'raspberrypi', just like on the host; this is a little confusing, so we'll fix it shortly.

Now set the password (be sure to remember it!) - issue (working at the container root prompt you just activated):

Code: Select all

[email protected]:~# passwd root

Enter new UNIX password: <enter password>
Retype new UNIX password: <enter password again>
passwd: password updated successfully
Next, create a pi user inside the container, with UID and GID numerically matching those of the pi user outside (i.e., 1000 in both cases). This will enable straightforward bind-mounting of the pi user's home directory into the container, shortly. Issue:

Code: Select all

[email protected]:~# useradd --user-group --groups \
  adm,dialout,cdrom,sudo,audio,video,plugdev,games,users,input,netdev \
  --create-home --uid 1000 --shell /bin/bash pi
For more information please see the useradd manpage, here.

Then, set user pi's password in the container (again, be sure to remember it!), and then exit the container shell. Still working at the container root prompt, issue:

Code: Select all

[email protected]:~# passwd pi

Enter new UNIX password: <enter password>
Retype new UNIX password: <enter password again>
passwd: password updated successfully
And finally, ensure that the hostname 'debian-stretch-64' resolves in the container (necessary for sudo). Issue:

Code: Select all

[email protected]:~# echo -e "\tdebian-stretch-64" >> /etc/hosts
[email protected]:~# logout
Container debian-stretch-64 exited successfully.
With that done, you are ready to boot the container! Let's try doing so directly from the command line first (we'll set up a service shortly). Issue:

Code: Select all

[email protected]:~ $ sudo systemd-nspawn --settings=no --machine=debian-stretch-64 --boot \
  --bind=/home/pi --bind=/etc/resolv.conf

Spawning container debian-stretch-64 on /var/lib/machines/debian-stretch-64.
Welcome to Debian GNU/Linux 9 (stretch)!
[  OK  ] Reached target Remote File Systems.
[  OK  ] Started Update UTMP about System Run level Changes.

Debian GNU/Linux 9 raspberrypi console

raspberry pi login:
Note how we don't have to give the full path to the directory tree here, since we (deliberately) located its root directory in the special location /var/lib/machines/, so we can use the --machine syntax instead.

Notice in the above how we're bind-mounting user pi's home directory into the container, so that it becomes accessible to the pi user we just up inside the container too. We also bind-mount /etc/resolv.conf, so that DNS lookup works properly, even if one of the host's network interfaces is reconfigured.
If you have created different users on your RPi3, you should of course modify these instructions. It is allowed to have multiple --bind entries. For more details, see the systemd-nspawn manpage here.

With luck, this should bring you to a login prompt on the booted container 64-bit Debian OS, as shown above!

Prior to logging in at this console however, take the chance to change the machine's internal host name, for ease of reference. While we could just do this as root from within the container, you can exploit systemd's container integration, to issue the necessary command from a terminal in your Raspbian system. To try this, open another terminal window on your RPi3 (leaving the one with the Debian login prompt showing open as well), and in this fresh terminal window issue:

Code: Select all

[email protected]:~ $ sudo hostnamectl --machine=debian-stretch-64 set-hostname debian-stretch-64
A number of systemd commands take --machine= parameters in this way, allowing you to conveniently operate on them 'from outside'. See e.g. these notes.

You can directly open a shell inside the container at any time as well (provided it is running). Try that now; working in the same terminal window where you just entered the 'set-hostname', issue:

Code: Select all

[email protected]:~ $ sudo machinectl shell [email protected] /bin/bash

Connected to machine debian-stretch-64. Press ^] three times within 1s to exit session.
[email protected]:~#
Notice how the shell prompt now reflects the newly modified internal hostname for the container.

In this shell, take a look at all the processes running inside the container, issue: Notice how few there are, due to the process namespacing in use. Processes running on the host are invisible to those inside the namespace, unlike a vanilla chroot; those running on the guest are visible to the host, but with remapped PIDs. This is what allows e.g. the PID 1 /lib/systemd/systemd process to exist, as a distinct instance from that running at PID 1 on the host.

You can also view the systemd journal and service manager status of the container from the host system. To illustrate this, open a third terminal window on Raspbian now, and in that new terminal issue:

Code: Select all

[email protected]:~ $ sudo journalctl --machine=debian-stretch-64
You should now be able to browse through the container's log, using the normal journalctl navigation keys. Press q to exit when done. Then, in the same console, view the container's service manager state; issue:

Code: Select all

[email protected]:~ $ sudo systemctl --machine=debian-stretch-64
Again, use the normal navigation keys, and press q to quit when done. Then verify that everything all services did start up OK in the container; issue:

Code: Select all

[email protected]:~ $ sudo systemctl --machine=debian-stretch-64 --failed

0 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.
Right, that's enough of the detour ^-^ You can now login as either user pi or root, using the passwords you set a moment ago, in the original login terminal window (the one in which you issued the systemd-nspawn command)! Do that now, logging in as root this time:

Code: Select all

raspberrypi login: root
Password: <enter the password for the container root account>

Last login: <date>
Linux debian-stretch-64 4.14.93-v8-24b08c0b745d-bis+ #2 SMP PREEMPT <date> aarch64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.

[email protected]:~# 
Although the initial prompt still reads 'raspberrypi', it changes to 'debian-stretch-64' as soon as we log in, and future login prompts will have the correct container hostname.

Here's a screenshot after booting and logging-in to the container, as just described:


Tweaks to the 64-bit Debian Guest OS

Once logged into the container's console as root, update the (Debian stretch) package metadata, and install the package sudo (so that the pi user in the container, who we arranged to be in the sudo group earlier, can leverage this). Issue:

Code: Select all

[email protected]:~# apt-get update && apt-get -y upgrade
[email protected]:~# apt-get install -y sudo
Notice how networking 'just works' here: as we did not elect (for convenience) to put the container in a private network namespace, it is sharing the same network adaptors as Raspbian. The initial upgrade will most likely be a no-op, since debootstrap downloads the most up-to-date versions of package, but it is always safest to check.

Next, install the 'file' package, and use the eponymous command to check you really are running aarch64 userland software inside the container:

Code: Select all

[email protected]:~ $ sudo apt-get install -y file
[email protected]:~ $ file $(which ls)

/bin/ls: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/, for GNU/Linux 3.7.0, BuildID[sha1]=a242161239cb1f4ea9b8a7455013295e5473e3ec, stripped
The exact output on your system may differ slightly.

Then, log out of the root account, and log in as the (container's) pi account. Issue:

Code: Select all

[email protected]:-# logout

Debian GNU/Linux 9 debian-stretch-64 console

debian-stretch-64 login: pi
Password: <enter the password for the container pi account>
[email protected]:~ $ 
If you try e.g. the ls command here, you'll see the contents of the Raspbian pi user's home directory, thanks to the bind mount we carried out when launching the container. You can modify files in this directory if you wish.

You can now carry out any necessary admin work using this account, elevating privileges using sudo when necessary. Begin by setting up a locale (you may have noticed warnings printed about this when running the journalctl command earlier); issue:

Code: Select all

[email protected]:~ $ sudo apt-get install -y locales
[email protected]:~ $ sudo dpkg-reconfigure locales
Select any locales you want to be enabled and press OK, then select a system locale from the list. You can then check the locale has been generated with:

Code: Select all

[email protected]:~ $ locale -a
Next, set the timezone. Issue:

Code: Select all

[email protected]:~ $ sudo dpkg-reconfigure tzdata
and select an appropriate timezone.

Now reboot the container to take up the changes (notice how the following command does not reboot your RPi3 itself!):

Code: Select all

[email protected]:~ $ sudo reboot
Wait for the container to restart - it should only take a few seconds, then log back in as use pi at the console prompt, and check you now have no errors:

Code: Select all

[email protected]:~ $ journalctl --boot --priority=err

-- No entries --
You could of course run this command from a regular Raspbian prompt, using the --machine option, as illustrated earlier (requires sudo).

Congratulations, that's the basic container setup done! Now we can just install a few small graphical applications, to test that these can be launched on the main desktop. Issue:

Code: Select all

[email protected]:~ $ sudo apt-get install -y x11-apps
Now try it out:

Code: Select all

[email protected]:~ $ DISPLAY=:0.0 xeyes&
And you should find that a window with a pair of eyes opens on the main desktop!
This works because the pi user's .Xauthority file came across with the bind mount of pi's home directory. But what about /tmp/.X11-unix - why didn't we need to bind-mount that? The answer is because we haven't put this container in a separate network namespace, so the Unix abstract domain sockets for the desktop's X11 server are still visible, and that (together with a valid .Xauthority) is all an application needs to connect. See my notes on this point here. And in fact, that's just as well, since bind-mounts of entries in /tmp is broken in systemd-nspawn for systemd v232 (bug #4789), the version currently used by Debian Stretch (and Raspbian).

Now we can deal with sound. To do this, we'll connect to the host's pulseaudio server. First, make sure all apps will use pulseaudio, not ALSA. Still working inside the container, issue:

Code: Select all

[email protected]:~ $ sudo nano -w /etc/asound.conf
and add the following text to the end:

Code: Select all

# Use PulseAudio by default
pcm.!default {
  type pulse
  fallback "sysdefault"
  hint {
    show on
    description "Default ALSA Output (currently PulseAudio Sound Server)"

ctl.!default {
  type pulse
  fallback "sysdefault"
Save, and exit nano.

Right, that's all the preparation within the guest OS done, so we can shut it down for now. Still working at the pi prompt within the container, issue:

Code: Select all

[email protected]:~ $ sudo poweroff

[  OK  ] Removed slice system-getty.slice.
[  OK  ] Reached target Shutdown.
Container debian-stretch-64 has been shut down.

[email protected]:~ $ 
Notice that your RPi3 did not itself shut down!

Arranging for the 64-bit Debian Guest OS Container to Start Automatically on Boot

Now we have a working container (which you can backup when not running simply by running "sudo cp -ax /var/lib/machines/debian-stretch-64 <backup-location>" incidentally, should you wish to do so), we will next arrange for it to be started automatically at boot. That way, it'll always be available, and you'll be able to get a shell using ds64-shell or machinectl (and launch apps using ds64-run or systemd-run) at any time (instructions follow).

Fortunately, systemd has a built-in mechanism (the [email protected] template) for starting up containers stored in the /var/lib/machines/ directory, as ours is. To leverage it, we first need to create a config file for our container. To do so, working back within the Raspbian desktop again, issue:

Code: Select all

[email protected]:~ $ sudo mkdir -pv /etc/systemd/nspawn
[email protected]:~ $ sudo nano -w /etc/systemd/nspawn/debian-stretch-64.nspawn
and enter the following text in that file:

Code: Select all



Save, and exit nano.

When systemd automatically starts a container, the default behaviour is slightly different from when launched using systemd-nspawn from the command line (for further details, see this manpage), hence the entries in the configuration file above:
  • By default, containers started by [email protected] are booted, so we don't need to specify this here.
  • However, also by default such containers will be launched in a private namespace. We turn this off using the PrivateUsers=no directive, as user mapping makes e.g. sharing the pi user's home directory problematic.
  • Not obvious, but automatically started containers drop the CAP_NET_ADMIN capability (unless they have a private network namespace), so we re-instate it.
  • Just as before, we have to specify the bind-mount for user pi's home directory into the container.
  • We also bind mount the special directory /run/user to /run/host-user in the container. This will allow connection to the pulseaudio server socket [12].
  • We also bind mount the DNS servers file, /etc/resolv.conf, so that changes to this are propagated into the container.
  • By default, automatically started containers run in their own network namespace, and don't share network interfaces or configuration with host. We don't want that, so we restore the default command-line-launched behaviour with Private=no.
  • Similarly, by default automatically started containers get a veth tunnel created back to the host (a sort of inter-network-namespace wormhole); this implies Private=yes, so we explicitly turn it off, with VirtualEthernet=no.
There are many other options available here, and of course you are free to tweak the above configuration to suit your own requirements. Stick with it as written for now however, until you get your system fully up and running.

Now we can create the startup service instance. To do so, simply issue:

Code: Select all

[email protected]:~ $ sudo systemctl enable
[email protected]:~ $ sudo systemctl enable [email protected]
That's it - all done! Now reboot your RPi3, and then open a terminal window and check that your 64-bit Debian container has booted and is sitting quietly in the background waiting for you:

Code: Select all

[email protected]:~ $ sudo systemctl list-machines
NAME               STATE   FAILED JOBS
raspberrypi (host) running      0    0
debian-stretch-64  running      0    0

2 machines listed.
Assuming you see something like the above, all is good! You can now drop into a shell within the container if you like; to do so, issue:

Code: Select all

[email protected]:~ $ sudo machinectl --setenv=DISPLAY="$DISPLAY" shell \
  [email protected] /bin/bash

Connected to machine debian-stretch-64. Press ^] three times within 1s to exit session.
[email protected]:~ $ 
We export the host's value of the $DISPLAY variable into the guest OS, to make app-launching straightforward. We'll replace this with the ds64-shell shorthand command later in the tutorial.

Final Tweaks to the 32-bit Raspbian Host OS

You can now do any operations you like within the guest, it is a fully booted instance. For example, launch the "xeyes" program on the host desktop, and log out of the shell:

Code: Select all

[email protected]:~ $ xeyes&
[email protected]:~ $ logout

Connection to machine debian-stretch-64 terminated.

[email protected]:~ $ 
Well, that didn't last long: notice how the application quit (and disappeared from the host desktop) when you logged out of the guest? You can get around that by launching such apps using systemd-run from a Raspbian (i.e., host) console. Try that now; issue:

Code: Select all

[email protected]:~ $ systemd-run \
  --setenv=DISPLAY="$DISPLAY" \
  --setenv=PULSE_SERVER="unix:/run/host-user/1000/pulse/native" \
  --uid=1000 --gid=1000 --machine=debian-stretch-64 \
This rubric is also handy if you want to e.g. create a menu or desktop entry to launch a 64-bit application within your Raspbian desktop. Note that giving absolute paths (/usr/bin/xeyes, not just xeyes) is mandatory with systemd-run. Notice that we've passed a few more environment variables here, to fix up a QT display issue that sometimes occurs, and to allow the host's pulseaudio server to be used.
Note how we had to specify that the process run as the pi user (uid 1000 gid 1000), since root (inside the container) has no .Xauthority for the display, as we have things set up currently.

If that worked, next set up a convenience script ds64-run (on the host), so that you can launch graphical 64-bit applications easily from within 32-bit Raspbian. Issue:

Code: Select all

[email protected]:~ $ sudo nano -w /usr/local/bin/ds64-run
and then enter the following text in that file:

Code: Select all

# Run the specified application, with any arguments, in the
# 64-bit Debian container, using the host OS' X-server and pulseaudio
# server. Also include a QT fixup for apps like vlc that need it.
sudo systemd-run \
	--setenv=DISPLAY="${DISPLAY}" \
	--setenv=PULSE_SERVER="unix:/run/host-user/1000/pulse/native" \
	--uid=1000 --gid=1000 \
	--machine=debian-stretch-64 \
Save, and exit nano. Then run:

Code: Select all

[email protected]:~ $ sudo chmod +x /usr/local/bin/ds64-run
Now, provided the 64-bit container is running, you can simply issue:

Code: Select all

[email protected]:~ $ ds64-run /usr/bin/xeyes
to start this program (or any other; remember to give the appropriate fully qualified path; you can also pass arguments). You can even add a menu item to Raspbian to launch a 64-bit application using this approach, using Preferences -> Main Menu Editor, and putting "ds64-run <full-app-path>" in the Command: field.

You can also setup a simple shorthand script to open a shell. Issue:

Code: Select all

[email protected]:~ $ sudo nano -w /usr/local/bin/ds64-shell
and put the following text in that file:

Code: Select all

# Open a shell as the pi user into the 64-bit Debian container.
exec sudo machinectl shell \
	--setenv=DISPLAY="${DISPLAY}" \
	--setenv=PULSE_SERVER="unix:/run/host-user/1000/pulse/native" \
        --uid=1000 \
	debian-stretch-64 \
Save and exit nano. Then run:

Code: Select all

[email protected]:~ $ sudo chmod +x /usr/local/bin/ds64-shell
Now you can just enter ds64-shell to get a pi container shell (and can sudo from there, if you need root privileges). You can launch apps directly from this command line, they should show up on the main host display. Issue the following, for example:

Code: Select all

[email protected]:~ $ ds64-shell

Connected to machine debian-stretch-64. Press ^] three times within 1s to exit session.
[email protected]:~ $ xeyes
and the xeyes app should open.
Remember, it'll still be closed when you close the shell, so the preferred way to launch desktop apps from the container is to use the ds64-run command we just defined, issued from a host (Raspbian 32-bit) terminal.

Congratulations, the final tweaks to your setup are now complete!

Installing Larger Apps in the 64-bit Debian Guest

Remember, you can do whatever you like in your 64-bit container. So to close, let's try installing and running a large 64-bit application, firefox. Close Xeyes if still open on your host desktop, and then, in the container shell that should still be open, issue:

Code: Select all

[email protected]:~ $ sudo apt-get install -y firefox-esr
This will take some time to download and install. Once done, you could launch firefox directly from the command line, but, as we've seen, the problem is that it will then quit when the container shell is closed. Instead, click into the other open (Raspbian) console, and issue:

Code: Select all

[email protected]:~ $ ds64-run /usr/bin/firefox
Hopefully everything should work, including audio in sites like YouTube! You can even create a simple launcher for firefox on your RPi3 desktop if you like, following the instructions just given for Xeyes.
If you do, use "ds64-run /usr/bin/firefox %u" in the Command: field; the %u allows a URL to be passed.

That's it, have fun!

Useful Command Cheatsheet

Here are some useful commands for working with your new raspbian-stretch-64bit container; most of these are in the main text above, but are gathered here for convenience. There's a lot more you can do of course, think of these just as a necessary minimum set. All the below commands are issued from a terminal in the host (32-bit Raspbian) OS.

Check the status of all containers:

Code: Select all

[email protected]:~ $ sudo systemctl list-machines
See the journal, since last boot, of your 64-bit container (assuming it is running):

Code: Select all

[email protected]:~ $ sudo journalctl --machine=debian-stretch-64 --boot
Check service status on your 64-bit container:

Code: Select all

[email protected]:~ $ sudo systemctl --machine=debian-stretch-64
Add --failed to see only units that haven't started correctly.

Of course, when logged into the container, you can issue commands like journalctl, systemctl etc. directly.

Get a (regular user) shell inside your 64-bit container (see earlier for the definition of this shorthand command):

Code: Select all

[email protected]:~ $ ds64-shell

Connected to machine debian-stretch-64. Press ^] three times within 1s to exit session.
[email protected]:~ $ 
You can then run "sudo su -" to get a root shell, if you need, or just elevate privileges as required for individual commands, using sudo.
You can open as many concurrent shells as you like. To exit the shell, just type logout, press ctrl-D, or press ctrl-] three times within a second. Operations like poweroff issued inside the container only affect the container - not your Rpi3 itself. Note that issuing reboot will not work properly when the container is managed by a service as here; better to use machinectl (as shown next) to stop and then start the container again. Networking identical to the host system is available (provided your host Raspbian OS has it configured), so you can ping, wget, apt-get, run web browsers etc. all from withing the container. By default, the pi user's home directory (only) is mapped inside the container for access.

Stop (poweroff) the 64-bit container (will stop any apps running from within it):

Code: Select all

[email protected]:~ $ sudo machinectl stop debian-stretch-64
You can also issue "sudo poweroff" when logged into the container.

And start (boot) it again:

Code: Select all

[email protected]:~ $ sudo machinectl start debian-stretch-64
Remember, the way we have things set up, the debian-stretch-64 container will be auto-started at boot, so you will only need this if the container has crashed, or you have manually taken it down for some reason (to take a backup, for example).

Run a 64-bit application in the container, displaying on the host's desktop, and using the host's pulseaudio server (see earlier for the definition of this shorthand command):

Code: Select all

[email protected]:~ $ ds64-run /usr/bin/firefox ""
Remember to use absolute paths for the application.

Prevent the container from starting at boot:

Code: Select all

[email protected]:~ $ sudo machinectl disable debian-stretch-64
Enable auto-start on boot again:

Code: Select all

[email protected]:~ $ sudo machinectl enable debian-stretch-64

To access files in the container from the host, remember that user pi's home directory is already mapped. The container's root directory prefix is /var/lib/machines/debian-stretch-64.

There are many other options available. Please take the time to read the machinectl, systemctl and systemd-run manpages.

Also please remember this is 'proof-of-concept', so many bugs and issues may remain!

Pre-Built, Bootable System for RPi3 B and B+

If these instructions have intrigued you, and you'd like to try out a fully-operational RPi3 B/B+ bootable image that incorporates them, then please see my raspbian-nspawn-64 project on GitHub, here. Screenshot:


Full download and usage instructions are on the GitHub page linked above.

Have fun, sakaki


[1] Actually, systemd-nspawnd leverages (init-system-agnostic) technologies in the Linux kernel (for example, various forms of namespacing) to do most of the heavy lifting. On non-systemd platforms, such as my OpenRC based gentoo-on-rpi3-64bit image, the same effect can be achieved via the firejail app: see for example my notes on this here.

[2] However, if booted in aarch32 mode, only aarch32 userland processes are allowed (unless you resort to emulation of some sort).

[3] For further details on using KVM, please see my notes here and here.

[4] However, at the time of writing, H/W acceleration for 64bit mode is on the verge of becoming available via V4L2, see e.g. this post.

[5] It should also work with the RPi3A, RPi2 Bv2, CM3 and CM3L, since they also have a 64-bit SoC, but I don't have examples of any of those to check (and you need appropriate DTB files too).

[6] The actual version tested was the 2018-11-13 release of "Raspbian Stretch with desktop".

[7] Advanced users can just use "unzip -p raspbian_latest > /dev/sd<x> && sync && partprobe /dev/sd<x>" or similar to write the image.

[8] It is relatively straightforward to compile your own kernel if you want to. See e.g. these notes. Gentoo users may also be interested in my own cross-compilation notes, here.

[9] The only exception here are the dtb files, but we'll back these up prior to switching to the 64-bit kernel, and put the Raspbian package "raspberrypi-kernel" (which provides these files) on hold, so they won't get overridden. These changes can easily be reversed and indeed I will show how to do that in a follow-on post.

[10] The Arch wiki has some nice detail on this process (here). Some further references that might be useful are Containerizing Graphical Applications On Linux With systemd-nspawn by J. Ramsden and
Systemd and Containers: An Introduction to systemd-nspawn
by A. Yemelianov. For further background on the underlying Linux technologies (viz.: namespacing, seccomp-bpf and capabilities) leveraged by "concessionaire" apps like systemd-nspawn, firejail and docker, see e.g. my notes here, here and here (of course, other facilities such as cgroups are also used). For more an introduction to the interaction of systemd-nspawn with other systemd tools, see e.g., systemd for Administrators, part XXI "Container Integration".

[11] Both because piggybacking on the host's networking is easier (you can ping, wget etc. out of the box), but also because turning it on would restrict access to the host's Unix abstract domain sockets, and we need such access, to be able to use the host's X11 server (since in systemd-232, due to bug #4789, bind mounts in /tmp don't work properly).

[12] See e.g "Running Steam in a systemd-nspawn Container". Note that we don't take the approach of bind-mounting /run/user/1000/pulse directly, as that will fail (source path not yet created) when systemd tries to bring the container up at boot time. We also don't bind /dev/snd, nor any of the /dev/{dri,shm}, as Debian's mesa doesn't seem to be able to make use of them even if they are there. If running an enhanced mesa, you could consider binding these; remember to write-enable them (via entries in "systemctl edit [email protected]", as suggested here) if you do.

Appendices follow in next post (char limit ><)
Edit: fix minor typos, mkdir omission.

Re: Tutorial: How (and why!) to set up a 64-bit kernel, 32-bit Raspbian host OS, 64-bit nspawn Debian guest OS RPi3 syst

Posted: Tue Jan 29, 2019 1:10 am
by sakaki
This is a follow on to my previous post, required by the board's per-message character limit ><

Postscript 1: Reverting to a 32-bit Kernel Again

If you'd like to revert back to using the original 32-bit kernel again, just perform the following steps. Bear in mind that you cannot boot the 64-bit container in this mode (due to an ARMv8a architectural restriction), so we'll disable that too.

So, to revert back, open a terminal and issue:

Code: Select all

[email protected]:~ $ sudo mv /boot/kernel8.img{,.bak}
[email protected]:~ $ sudo rm -v /boot/*.dtb
[email protected]:~ $ sudo cp -v /boot/r32-dtbs/*.dtb /boot/
[email protected]:~ $ sudo machinectl disable debian-stretch-64
[email protected]:~ $ sudo apt-mark unhold raspberrypi-kernel
This will restore the old dtbs, rename the 64-bit kernel so the bootloader won't see it, disable the 64-bit container, and allow the 32-bit Raspbian kernel package to update again. Now reboot, and you should be cleanly back to a 'standard' 32-bit kernel / 32-bit userland Raspbian system.

If you want, at some stage, to undo this, and go back to the 64-bit hybrid system then, provided you haven't uninstalled anything important, you can do so easily. Just open a terminal window and issue:

Code: Select all

[email protected]:~ $ sudo apt-mark hold raspberrypi-kernel
[email protected]:~ $ sudo mv /boot/kernel8.img{.bak,}
[email protected]:~ $ sudo rm -v /boot/r32-dtbs/*.dtb
[email protected]:~ $ sudo mv /boot/*.dtb /boot/r32-dtbs/
[email protected]:~ $ sudo cp -v /boot/r64-dtbs/*.dtb /boot/
[email protected]:~ $ sudo machinectl enable debian-stretch-64
Just reboot, and you should have your full hybrid system back again, debian-stretch-64 container and all!

Postscript 2: Updating the 64-bit Kernel

If you are running the hybrid system, and would like to update your 64-bit kernel, you can do so very easily. Find the wget string of the latest bcmrpi3-kernel-bis package here, and then issue (the following is just an example, you must use your own string):

Code: Select all

[email protected]: ~ $ wget -c
Once it is downloaded, simply untar it to install (again, use the filename you just downloaded in the below, which is just an illustration):

Code: Select all

[email protected]: ~ $ sudo tar --exclude='COPYING.linux' \
  -xJf bcmrpi3-kernel-bis- -C /
[email protected]: ~ $ sync
Then reboot, and you should be using your new kernel!
Once happy, you can safely delete the old module set in /lib/modules/<old-kernel-release-name>, should you wish (this step is optional).

Note that although the kernel is autobuilt weekly, there's generally no need to update on anything like this frequency!

4 March 2020: thanks to ShiftPlusOne, raspbian-nspawn-64 can now be installed from the official Raspbian repo, just like any other regular package, using apt! For more details, please see this post.



Re: Tutorial: How (and why!) to set up a 64-bit kernel, 32-bit Raspbian host OS, 64-bit nspawn Debian guest OS RPi3 syst

Posted: Mon Jul 29, 2019 2:11 pm
by leshik
Thanks, this thread is gold!

I wasn't particularly interested in running GUI applications but needed a way to run Docker for both armhf and aarch64 to build images for both architectures. This thread perfectly solved my problem. Here is the only additional step that I had to perform to make Docker work under systemd-nspawn: ... emd-nspawn

BTW, I used Debian Buster instead of Stretch for both host and guest. The only thing that should be addressed regarding Docker is to make it use legacy iptables as Debian has switched to nft, though it's not RPi or ARM specific. For those who are interested, here is how:

Code: Select all

update-alternatives --set iptables /usr/sbin/iptables-legacy
update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy

Re: Tutorial: How (and why!) to set up a 64-bit kernel, 32-bit Raspbian host OS, 64-bit nspawn Debian guest OS RPi3 syst

Posted: Tue Aug 13, 2019 1:47 pm
by nightops
Any chance this will get recompiled for the Pi 4?

Re: Tutorial: How (and why!) to set up a 64-bit kernel, 32-bit Raspbian host OS, 64-bit nspawn Debian guest OS RPi3 syst

Posted: Fri Aug 23, 2019 10:46 pm
by sakaki
nightops wrote:
Tue Aug 13, 2019 1:47 pm
Any chance this will get recompiled for the Pi 4?
Yes, I intend to do so in the relatively near term.

Best, sakaki

Re: Tutorial: How (and why!) to set up a 64-bit kernel, 32-bit Raspbian host OS, 64-bit nspawn Debian guest OS RPi3 syst

Posted: Wed Mar 18, 2020 10:42 pm
by blee0518
sakaki wrote:
Fri Aug 23, 2019 10:46 pm
nightops wrote:
Tue Aug 13, 2019 1:47 pm
Any chance this will get recompiled for the Pi 4?
Yes, I intend to do so in the relatively near term.

Best, sakaki
Is this update for RPi 4 headless bootable or VNC enabled?

Re: Tutorial: How (and why!) to set up a 64-bit kernel, 32-bit Raspbian host OS, 64-bit nspawn Debian guest OS RPi3 syst

Posted: Thu Mar 19, 2020 7:59 pm
by sakaki
blee0518 wrote:
Wed Mar 18, 2020 10:42 pm
Is this update for RPi 4 headless bootable or VNC enabled?
As noted in my post here, you can now just install the raspbian-nspawn-64 package on a vanilla Raspbian system from the official repo (thanks to ShiftPlusOne!); there's no need for a custom image anymore ^-^

As to running it headless, you can (see instructions in link just given) install on a RaspbianLite image too, and the Debian Buster 64-bit container runs its own systemd init and is automatically booted, so yes, you could easily use it in a typical headless context.

As to adding VNC since you have a vanilla Raspbian host you can just follow the normal processes to install and configure a VNC server there.

hth, sakaki