Barben360
Posts: 7
Joined: Wed Nov 21, 2018 11:21 am

Why can't I run 32-bit kernels on QEMU's RPi3?

Tue Apr 09, 2019 2:58 pm

Hello,

On a real Raspberry Pi 3, you can run a 32-bit kernel thanks to aarch64 32-bit retrocompatibility.

However when I run a 32-bit kernel using raspi3 machine on QEMU, it just doesn't run, I have to use the raspi2 machine. Any idea on why? Is there a problem with the retrocompatibility of QEMU's aarch64 TCG?

When I get into QEMU's code, the only differences I see between raspi2 and raspi3 machines are:
- boot process,
- board revision value,
- the CPU (4x Cortex-A7 vs 4x Cortex-A53)

On real hardware, is the boot process really different for 32-bit and 64-bit kernels? If it is, how does the hardware make the difference? This seems quite impossible to me.

I don't think board revision value is important, that might only be for informational purpose.

Perhaps their are some missing or hard-coded parameters in the CPU that break retrocompatibility?

Thanks for your help with that!

bzt
Posts: 344
Joined: Sat Oct 14, 2017 9:57 pm

Re: Why can't I run 32-bit kernels on QEMU's RPi3?

Tue Apr 09, 2019 3:44 pm

Hi,
Barben360 wrote:
Tue Apr 09, 2019 2:58 pm
Hello,

On a real Raspberry Pi 3, you can run a 32-bit kernel thanks to aarch64 32-bit retrocompatibility.

However when I run a 32-bit kernel using raspi3 machine on QEMU, it just doesn't run, I have to use the raspi2 machine. Any idea on why? Is there a problem with the retrocompatibility of QEMU's aarch64 TCG?
If you take a closer look, you can see that raspi3 is configured for AArch64 CPU only https://github.com/qemu/qemu/blob/maste ... spi.c#L112 because there's no other command line option to specify if we're loading kernel7.img or kernel8.img (for example the load address and the smpboot 32/64 code among others are selected by the -M raspi2/raspi3 argument). There could be an "raspi3-32" machine type, but considering that both raspi2 and raspi3 otherwise uses the same peripheral driver (bcm2836.c), it would be just an alias to raspi2 anyway.
One could also use the TARGET define, and use qemu-system-aarch32 and qemu-system-aarch64 to decide modes, but that way you'd have to recompile, you couldn't support both 32 and 64 bits in the same binary. The current solution allows 32 bit too with qemu-system-aarch64 -M raspi2.
Barben360 wrote:
Tue Apr 09, 2019 2:58 pm
When I get into QEMU's code, the only differences I see between raspi2 and raspi3 machines are:
- boot process,
- board revision value,
- the CPU (4x Cortex-A7 vs 4x Cortex-A53)
And also the CPU mode as well as the smp boot code and the name of the peripheral class :-)
Barben360 wrote:
Tue Apr 09, 2019 2:58 pm
On real hardware, is the boot process really different for 32-bit and 64-bit kernels? If it is, how does the hardware make the difference? This seems quite impossible to me.
On real hardware the board boots from GPU (start.elf). That code looks for kernel7.img, loaded to 0x8000 and initializes AArch32 mode. On the other hand, if it founds kernel8.img then it loads that to 0x80000 and initializes AArch64 mode. So it is the GPU that switches boot time ARM modes, but qemu does not emulate the GPU and it does not emulate start.elf at all. The kernel.img file does not have a header either, so qemu cannot determine for which ARM mode it was compiled for. Therefore you must select the proper kernel/machine combination on command line, and for simplicity there's only one option to do that (-M).
Barben360 wrote:
Tue Apr 09, 2019 2:58 pm
I don't think board revision value is important, that might only be for informational purpose.
Nope, it's not. Just informational.
Barben360 wrote:
Tue Apr 09, 2019 2:58 pm
Perhaps their are some missing or hard-coded parameters in the CPU that break retrocompatibility?
Yes, exactly. All parameters are hardcoded for the reasons I wrote above.
Barben360 wrote:
Tue Apr 09, 2019 2:58 pm
Thanks for your help with that!
I hope I've answered your question.

Cheers,
bzt

Barben360
Posts: 7
Joined: Wed Nov 21, 2018 11:21 am

Re: Why can't I run 32-bit kernels on QEMU's RPi3?

Tue Apr 09, 2019 4:41 pm

Thank you very much for your answer, everything is much clearer now.

Do you think it would be possible to have the 32-bit boot process while still using cortex-A53 instead of downgrading to cortex-A7?

When modifying QEMU's code to force cortex-A53 (also modifying clusterid for mp_affinity setting) while keeping raspi2 machine boot process, I can't boot a 32-bit kernel.

bzt
Posts: 344
Joined: Sat Oct 14, 2017 9:57 pm

Re: Why can't I run 32-bit kernels on QEMU's RPi3?

Tue Apr 09, 2019 10:36 pm

Hi,
Barben360 wrote:
Tue Apr 09, 2019 4:41 pm
Thank you very much for your answer, everything is much clearer now.

Do you think it would be possible to have the 32-bit boot process while still using cortex-A53 instead of downgrading to cortex-A7?
I don't think it will be possible soon. When I first submited my patch, I've created a separate bcm2837 class and configurable SoC (the qemu interface was different back then). The maintainers didn't like it, and I had to merge it with bcm2836 class and rewrite my patch three times. Finally Pekka managed to push the patch through in its current form.
When modifying QEMU's code to force cortex-A53 (also modifying clusterid for mp_affinity setting) while keeping raspi2 machine boot process, I can't boot a 32-bit kernel.
For the current implementation this is not possible as far as I know. There's a cpu_type which is either cortex-a7 or cortex-a53 (without the mode, that's a change in qemu interface since my first patch). The latter is defined in aarch64_a53_initfn, and there's no 32 bit counterpart in cpu.c, only for cortex_a7_initfn and this comment in line 182:

Code: Select all

if (arm_feature(env, ARM_FEATURE_AARCH64)) {
    /* 64 bit CPUs always start in 64 bit mode */
    env->aarch64 = 1;
Maybe I miss something, but with the current interface I can't see how to tell the cortex-a53 class from the bcm driver to start in AArch32.

But do you really care about A-53? I mean A-7 is also a Quad ARM CPU with exactly the same ARM Control MMIO, and the peripheral driver is the same too. Is there anything A-53 specific in AArch32 mode that you miss from A-7?

Cheers,
bzt

Barben360
Posts: 7
Joined: Wed Nov 21, 2018 11:21 am

Re: Why can't I run 32-bit kernels on QEMU's RPi3?

Wed Apr 10, 2019 7:27 am

Thanks again for your very clear answer.

No, there is nothing I miss from Cortex-A7, but I was wondering why such a tweak was necessary since Aarch64 is supposed to be Aarch32-retrocompatible.

I am currently working on a hardware simulator (see www.hiventive.com), and Raspberry Pi 3 is our proof of concept, and you can communicate with devices on the GPIO (terminal through UART, buttons, leds). We use QEMU to simulate the CPU, most other IPs are coded in SystemC. Today, our Raspberry Pi 3 only runs in 64-bit mode, and now thanks to you I know why.

We are currently in beta-testing phase, let me know if you are interested in getting a beta key, it would be very interesting for us to have a feedback from someone who contributed to QEMU.

Barben360
Posts: 7
Joined: Wed Nov 21, 2018 11:21 am

Re: Why can't I run 32-bit kernels on QEMU's RPi3?

Wed Apr 10, 2019 8:43 am

I finally managed to run a 32-bit kernel in QEMU using Cortex-A53!

I just deactivated aarch64 feature in bcm2836.c:

Code: Select all

for (n = 0; n < BCM283X_NCPUS; n++) {
        object_initialize_child(obj, "cpu[*]", &s->cpus[n], sizeof(s->cpus[n]),
                                info->cpu_type, &error_abort, NULL);
        // legal version of unset_feature(ARM_CPU(&s->cpus[n])->env, ARM_FEATURE_AARCH64)
        ARM_CPU(&s->cpus[n])->env.features &= ~(1ULL << 32);
 }
Of course this forces 32-bit and therefore it must be associated with some control to select 32-bit or not. But running your raspi3 machine with a 32-bit kernel becomes possible with this. Of course this is not very clean and there are few chances to upstream this to QEMU. But for my simulator this will be OK since I'm the only judge of my code's quality :lol:

Thank you very much for your help!

Barben360
Posts: 7
Joined: Wed Nov 21, 2018 11:21 am

Re: Why can't I run 32-bit kernels on QEMU's RPi3?

Thu Apr 11, 2019 8:24 am

Hi,

I have noticed something with raspi2 machine when running a Raspbian kernel (I have tried nothing else): even if the kernel boots perfectly, I can see in the boot log that SMP fails to bring the 3 secondary CPUs up. Using Cortex-A53 I have the same problem. Is this a known issue?

Cheers,
Barben360

Return to “Bare metal, Assembly language”