Entering aarch64 execution state

[achingupta]

What is the mechanism for releasing the CPUs from reset in aarch64 instead of the aarch32 execution state on the RPI3? Is there a configuration input or does one have to rely on the RMR_EL3 to switch from aarch32 to aarch64? This would be the first step to run any aarch64 code. Alternatively, is it that we start in aarch64 and transition to aarch32?

[dom]

There is a config.txt entry to make the ARM start in 64-mode.

Code: Select all

arm_control=0x200

(This will obviously fail to boot unless you provide a kernel.img compiled for 64-bit mode).

[achingupta]

Thanks! that helps.

[dwelch67]

Is there documentation or is this like the RPI2 where the answer is "it is the same except the old core is replaced by a new core"?

[swarren]

So "arm_control=0x200" selects AArch64 mode. What value selects AArch32 mode, or does that only happen if you completely remove that entry from config.txt?

[dom]

So "arm_control=0x200" selects AArch64 mode. What value selects AArch32 mode, or does that only happen if you completely remove that entry from config.txt?

Bit 9 of arm_control determines if arm is booted in 32 or 64-bit mode. You could set arm_control=0x0 or remove it completely to boot in 32-bit mode.

[Burngate]

This'll probably appear somewhat thick, but ...
Do all four cores have to be in the same state?

I would have expected to be able to, eg., run RiscOS 32-bit on one core, and have 64-bit bare-metal stuff on a second (I've almost managed this with my Pi 2, RiscOS on one and bare-metal on a second)
but that arm_control=0x200 seems to indicate only bit 9 does anything

[dom]

When the arm comes out of reset it samples a single bit signal to say whether boot (for all cores) should be in 64-bit or 32-bit mode.

64-bit mode does allow a way of switching back to 32-bit mode (but if you boot in 32-bit mode you can't switch to 64-bit).

So from a 64-bit boot it is probably possible to switch a core into 32-bit mode to run RISC OS (but the A53 documentation would be the place to determine what is possible).

[achingupta]

You can go from aarch64 execution state to aarch32 without resetting the core by using the correct ERET instruction. In order to go from aarch32 to aarch64, the reset management register can be used. However, it does have caveats.

[MrTomasz]

Hi dom,

I have few questions :

1. When ARM boots in AArch64 mode, in which exception level (EL) we are while trying to load & run kernel.img ? Is it EL3 or we are already switched to EL2/EL1 by VC arm loader ?
2. Are there any other special magic things we have to do ? (arm_control=0x200 was undocumented until somebody asked)
3. In AArch64 mode, is kernel image still loaded @ 0x8000 or at different address?
4. Before [64-bit] kernel.img is entered, do we have all ARM cores configured, up and running or just the first one?

I believe I did minimum amount of required work to just boot kernel (*) and see log on UART, however still no luck...

(*) - not a "make ARCH=arm64 defconfig Image" simple shot

[dom]

1. When ARM boots in AArch64 mode, in which exception level (EL) we are while trying to load & run kernel.img ? Is it EL3 or we are already switched to EL2/EL1 by VC arm loader ?
2. Are there any other special magic things we have to do ? (arm_control=0x200 was undocumented until somebody asked)
3. In AArch64 mode, is kernel image still loaded @ 0x8000 or at different address?
4. Before [64-bit] kernel.img is entered, do we have all ARM cores configured, up and running or just the first one?

There is no 64-bit arm loader code. You should be running with kernel_old=1 and your 64-bit arm code should start from zero.
The arms will just be in the default reset state. i.e. all four cores will be running, starting from address zero.
It is up to your arm code to check the arm core number and ensure one is the "master" and make the others wait until it has set things up suitably.

[MrTomasz]

Thanks for the reply dom.

I was thinking about that if I should use kernel_old=1 to boot from 0x0 or not... I'll check that and let's see.

[MrTomasz]

Hello again,

Still nothing... I think the quicker thing will be to test some baremetal program or some things like U-Boot or Coreboot - the reason for that is that Linux kernel doesn't like to be entered in EL3 mode, where the other things I mentioned can and are properly switching to EL2/EL1.

I've also tried boot-wrapper-arch64, but this one is producing ELF file with fancy addressing for kernel and DTB, which after objcopy (as kernel.img is not ELF file) makes me 256MB file... I need to relocate kernel image and DTB to make it smaller, and try it again...

I am also unsure about other plenty things, but would like to see at least simple kernel booting log (or at least few lines of it until it fails on something)...

Is there anyone who succeed at least with running baremetal AArch64 code which does ultra-simple thing like priting a char on UART or blinking a LED ? Just to see if it physically works and runs some code...

@dom
As I understand all posts regarding RPi3 SoC, the things like ARM core management, Interrupt controller, etc. - everything is same as on RPi2 ? -- I mean done via custom mailboxes and no GIC inside, no PSCI calls needed or whatever?

[dom]

@dom
As I understand all posts regarding RPi3 SoC, the things like ARM core management, Interrupt controller, etc. - everything is same as on RPi2 ? -- I mean done via custom mailboxes and no GIC inside, no PSCI calls needed or whatever?

We are using an identical kernel on Pi2 and Pi3 so I believe everything must be the same. The only difference I'm aware of between the arm code on Pi2 and Pi3 is in the bootcode we inject from firmware.
The mechanism to enable SMP is slightly different:

Code: Select all

#ifdef BCM2710
        mrrc p15, 1, r0, r1, c15  @ CPU Extended Control Register
        orr r0, r0, #(1<<6)       @ SMP
        mcrr p15, 1, r0, r1, c15  @ CPU Extended Control Register
#else
        mrc p15, 0, r0, c1, c0, 1 @ Read Auxiliary Control Register
        orr r0, r0, #(1<<6)       @ SMP
        mcr p15, 0, r0, c1, c0, 1 @ Write Auxiliary Control Register
#endif

(BCM2710 is the Pi3 version). However if you are only using one core, that shouldn't be essential.

[MrTomasz]

Thanks for the reply.

Another question - machine specific code in arm/mach-bcm2709 - is it required for just proving that kernel code is executed and something is sent to UART (in this case bootlog) ? I don't care if kernel or soc will crash just right after first bits of init. I didn't finish this part yet, so that's why I am asking, as I don't want to do too much for the first shot to see if it actually does anything in aarch64.

I just do not have any proof that it actually runs aarch64 code properly...plus include EL3 to EL2/1 switch before kernel and we already complicated things a bit...


BTW. How did you validate AArch64 mode? Do you have any simple baremetal sample code ?

[swarren]

Here's something that demos AArch64 mode:
https://github.com/swarren/rpi-3-aarch64-demo

The compiler-optimization dependency is odd. I wonder if that's related to your issue?

[MrTomasz]

Thanks swarren!

I hope I can validate at least some basic things and start finally my kernel.

What do you mean by "compiler-optimization dependency is odd" ?

[rst]

The compiler-optimization dependency is odd. I wonder if that's related to your issue?

Thanks for the demo! Please try to add

Code: Select all

disable_commandline_tags=1

to your config.txt file. The boot loader will otherwise overwrite your code (without -O2) at 0x100 with ATAGS.

[dom]

Here's something that demos AArch64 mode:
https://github.com/swarren/rpi-3-aarch64-demo

Nice!

[swarren]

Please try to add

Code: Select all

disable_commandline_tags=1

to your config.txt file. The boot loader will otherwise overwrite your code (without -O2) at 0x100 with ATAGS.

Thanks, that fixed it. I actually modified the code so that this option isn't needed though, by leaving a hole in the binary.

[marcus_c]

Hi.

I got U-Boot working in 64-bit mode.

To build it, do the following steps:

Code: Select all

git clone https://github.com/zeldin/u-boot-rpi3.git
cd u-boot-rpi3
make rpi_3_defconfig
make CROSS_COMPILE=aarch64-unknown-linux-gnu-

Copy the resulting u-boot.bin to your boot partition, and make sure that config.txt contains

Code: Select all

dtoverlay=pi3-miniuart-bt-overlay
kernel=u-boot.bin
kernel_old=1
arm_control=0x200
disable_commandline_tags=1

And presto, 64-bit U-Boot.

This should make it easier to get a kernel running.

[dom]

I got U-Boot working in 64-bit mode.

Excellent!

[Electron752]

Hi,

I've been doing some reading of web pages and it appears that 64 bit ARM may not be that easy since there are some pointer dependencies between the firmware, bootloader, and linux kernel.

I was wondering if anybody has any thoughts on getting a 64 bit compile of the linux kernel running with the video and audio disabled. That should eliminate some of the dependency on the firmware. The console would simply be one of the COM ports of the SOC.

I'm thinking this would still be acceptable for server and embedded applications since the video isn't really required. People could use those cheap FTDI based USB to TTL COM cables that are floating all over the Internet.

Thoughts/Comments....

Thoughts.

[ghans]

Thats what the BSD guys are doing too ...


ghans

[MrTomasz]

Hi,

I've been doing some reading of web pages and it appears that 64 bit ARM may not be that easy since there are some pointer dependencies between the firmware, bootloader, and linux kernel.

I was wondering if anybody has any thoughts on getting a 64 bit compile of the linux kernel running with the video and audio disabled. That should eliminate some of the dependency on the firmware. The console would simply be one of the COM ports of the SOC.

I'm thinking this would still be acceptable for server and embedded applications since the video isn't really required. People could use those cheap FTDI based USB to TTL COM cables that are floating all over the Internet.

Thoughts/Comments....

Thoughts.

That's what I am exactly working on as a first step of bringing up RPi3 in AArch64 world, however was bit busy in last days...

With U-boot running in AArch64 (thanks to @marcus_c for the news), I think it should be much easier and faster to test it. AFAIK U-Boot exits EL3 state, so kernel should boot smoothly now.