code_exec
Posts: 271
Joined: Sun Sep 30, 2018 12:25 pm

TUTORIAL: Enabling KVM virtualisation on the Raspberry Pi 3

Sun Feb 03, 2019 8:27 am

KVM is an accelerator which can be used with QEMU to run guests of the same architecture as the host at accelerated speed. On x64 PCs, this can be enabled in the BIOS if the CPU supports it (I am aware that some Intel Atom CPUs do not). However, the Pi does not have a BIOS, and there is no config.txt entry that enables KVM. So, to get KVM acceleration on the Raspberry Pi 3, you need to compile a 64-bit (ARMv8) kernel with KVM support (as far as I know, KVM does not support ARMv7 yet).

Install build dependencies:

Code: Select all

sudo apt install bc build-essential gcc-aarch64-linux-gnu git unzip libncurses5-dev bison flex libssl-dev
Download kernel sources:

Code: Select all

git clone --depth=1 https://github.com/raspberrypi/linux
Setup kernel:

Code: Select all

cd linux
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- bcmrpi3_defconfig
Enable KVM:

Code: Select all

make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- menuconfig
Use the arrow keys to navigate menuconfig. Find 'Virtualisation' and enable it using the space key. Press enter to access a menu of Virtualisation options. Enable all of them using the space key. Use the Tab key to select 'Save' and press enter, and then save the configuration as .config.

Now, find out how many CPU cores you have. Enter 'nproc' into the terminal and you will be given a number. Multiply that number by 1.5. Then run the following, replacing 6 with the number you got after multiplying nproc by 1.5:

Code: Select all

make -j 6 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu-
This will take a while, no matter how powerful your PC is. Once done, mount the ext4 partition (partition 2) of your Pi's microSD card or USB storage to /mnt and the FAT32 partition (partition 1) of your Pi's microSD card or USB storage to /mnt/boot.

Then run:

Code: Select all

sudo cp arch/arm64/boot/Image /mnt/boot/kernel8.img
sudo make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu INSTALL_MOD_PATH=/mnt modules_install
Remove your microSD card or USB storage from your PC and insert it into your Pi, and turn your Pi on. KVM should be available. To verify this, run:

Code: Select all

dir /dev
If /dev/kvm is listed then KVM acceleration is available.

Enjoy!
Ubuntu 18.04 LTS desktop images for the Raspberry Pi 3.

https://github.com/CodeExecution/Ubuntu-ARM64-RPi

code_exec
Posts: 271
Joined: Sun Sep 30, 2018 12:25 pm

Re: TUTORIAL: Enabling KVM virtualisation on the Raspberry Pi 3

Sun Feb 03, 2019 10:55 am

For those who aren't so keen on compiling the kernel from source, here is a precompiled kernel based on kernel 4.14.95:

https://1drv.ms/u/s!Asov5XVfbr4hm1ap71lWIwkD4nW1

Download the archive and extract it. Then run:

Code: Select all

cd linux-4.14.95-kvm-rpi3
sudo cp kernel8.img /boot/
sudo cp -r modules/ /lib/modules/
sudo cp dtbs/* /boot/ 
Enjoy!
Ubuntu 18.04 LTS desktop images for the Raspberry Pi 3.

https://github.com/CodeExecution/Ubuntu-ARM64-RPi

User avatar
sakaki
Posts: 214
Joined: Sun Jul 16, 2017 1:11 pm

Re: TUTORIAL: Enabling KVM virtualisation on the Raspberry Pi 3

Sun Feb 03, 2019 12:13 pm

Nice tutorial code_exec!

As another option for those wanting to use a pre-built kernel, I maintain a weekly autobuild of an augmented bcmrpi3_defconfig kernel (which includes the necessary settings to enable KVM) here (full instructions for downloading and installation are on the linked page also).

Incidentally, as this is the default kernel used by the gentoo-on-rpi3-64bit image [1], you can try running KVM 'out of the box' on an RPi3 booted using this. I have instructions for doing so here and here (which ought to work with minor adjustments on any distro). Here's a screenshot of running an Ubuntu aarch64 18.04 Server LTS cloud image guest:

Image
Note that two different kernels are in use here: this isn't just a chroot.

Best, sakaki

[1] Although there's nothing 'Gentoo-specific' about the kernel per se.

code_exec
Posts: 271
Joined: Sun Sep 30, 2018 12:25 pm

Re: TUTORIAL: Enabling KVM virtualisation on the Raspberry Pi 3

Sun Feb 03, 2019 2:41 pm

sakaki's tutorial will also work on other Linux distros for the Pi 3 such as Debian with this KVM-enabled kernel. It will also work on x64 hosts without KVM enabled but will be slower than on actual ARM64 hardware.
Ubuntu 18.04 LTS desktop images for the Raspberry Pi 3.

https://github.com/CodeExecution/Ubuntu-ARM64-RPi

gilius
Posts: 96
Joined: Sun Apr 08, 2018 1:12 pm

Re: TUTORIAL: Enabling KVM virtualisation on the Raspberry Pi 3

Sun Mar 03, 2019 12:23 pm

code_exec, your post is intriguing to say the least, but completely lacks any context. For example, are you upgrading an existing raspi2 kernel or replacing it with different kernel? And presumably this tutorial is for Ubuntu ports that are missing KVM and will not work with other distros?

gilius
Posts: 96
Joined: Sun Apr 08, 2018 1:12 pm

Re: TUTORIAL: Enabling KVM virtualisation on the Raspberry Pi 3

Mon Mar 04, 2019 10:26 pm

There's a few typos in the commands:
sudo make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu INSTALL_MOD_PATH=/mnt modules_install
=
sudo make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- INSTALL_MOD_PATH=/mnt modules_install

sudo cp -r modules/ /lib/modules/
=
sudo cp -r modules/ /lib/

But I couldn't get it to work either way. Compiling the kernel didn't produce any .ko modules with a "kvm" prefix, so I think some steps have been left out. Also, I'm not a dev, so it's not clear what concept is being used:
-upgrading the raspi2 kernel
-changing the raspi2 kernel
-adding a kernel alongside the raspi2 kernel

gilius
Posts: 96
Joined: Sun Apr 08, 2018 1:12 pm

Re: TUTORIAL: Enabling KVM virtualisation on the Raspberry Pi 3

Sat Mar 09, 2019 11:45 pm

I tested again by mounting partition 2 to /mnt before mounting partition 1 to /mnt/boot

kernel8.img is then copied to the system-boot partition 1 under /mnt/boot
The config.txt points to kernel8.bin

I tried editing config.txt to kernel8.img and kernel8.bin, but I just get a rainbow screen and nothing boots up on my B+.

gilius
Posts: 96
Joined: Sun Apr 08, 2018 1:12 pm

Re: TUTORIAL: Enabling KVM virtualisation on the Raspberry Pi 3

Sun Mar 10, 2019 12:16 am

I've cross-referenced with this guide too:
https://github.com/sakaki-/gentoo-on-rp ... rossdev-PC

Unsure why there's just a rainbow screen?

EDIT: replacing with the old kernel8.img then boots up, so there must be something wrong with that 20 MB file. The older kernel8 is only 0.5 MB.

gilius
Posts: 96
Joined: Sun Apr 08, 2018 1:12 pm

Re: TUTORIAL: Enabling KVM virtualisation on the Raspberry Pi 3

Tue Mar 12, 2019 10:28 pm

I've been cross-referencing many guides on this and it seems both code_exec and sakaki have missed out the stages that compile the DTBs, including Overlays, as well as the modules. I'm hoping that might solve the issue...

One guy also did another additional step where he replaced /lib/firmware/brcm with brcmfmac43430-sdio.bin and .txt

User avatar
sakaki
Posts: 214
Joined: Sun Jul 16, 2017 1:11 pm

Re: TUTORIAL: Enabling KVM virtualisation on the Raspberry Pi 3

Tue Mar 12, 2019 11:40 pm

gilius wrote:
Tue Mar 12, 2019 10:28 pm
I've been cross-referencing many guides on this and it seems both code_exec and sakaki have missed out the stages that compile the DTBs, including Overlays, as well as the modules. I'm hoping that might solve the issue...

One guy also did another additional step where he replaced /lib/firmware/brcm with brcmfmac43430-sdio.bin and .txt
You need to build a kernel, module set, dtbs and dtbos from an appropriate kernel source tree, appropriately configured, to use KVM. I didn't run through the steps to do that explicitly in my KVM tutorials linked earlier in the thread (i.e., this one and this one) because they are based upon my gentoo-on-rpi3-64bit image as a starting point, and that already has an appropriate kernel.

However, in the same wiki as those tutorials, I have also posted a guide on how to cross-compile a bcmrpi3_defconfig 64-bit kernel (here) [1]. As noted earlier in this thread though (emphasis added):
sakaki wrote:
Sun Feb 03, 2019 12:13 pm
As another option for those wanting to use a pre-built kernel, I maintain a weekly autobuild of an augmented bcmrpi3_defconfig kernel (which includes the necessary settings to enable KVM) here (full instructions for downloading and installation are on the linked page also).
That is, you need to modify the base config created by "make bcmrpi3_defconfig" step in the just-linked kernel-build tutorial, in order to enable KVM (you do this prior to the main "make" step). A script that will do this may be viewed here; the relevant lines are these (in case you just want to e.g. "make menuconfig" and enable the items yourself).

hth,

sakaki

[1] The project's readme also contains instructions for natively building such a kernel on the RPi3, here. This takes a long time, however (unless you have distcc support).

gilius
Posts: 96
Joined: Sun Apr 08, 2018 1:12 pm

Re: TUTORIAL: Enabling KVM virtualisation on the Raspberry Pi 3

Wed Mar 13, 2019 12:32 am

I FINALLY FOUND THE FIX FOR Code_Exec's TUTORIAL!!!!!!!!! :twisted:

Replace this:

Code: Select all

make -j 6 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu-
With this:

Code: Select all

make -j 6 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- Image modules dtbs
That is all that was required! Still haven't found a way to dual-boot Kernels since there is no EFI menu...

Return to “General discussion”