Entering aarch64 execution state


171 posts   Page 1 of 7   1, 2, 3, 4, 5 ... 7
by achingupta » Mon Feb 29, 2016 10:36 am
What is the mechanism for releasing the CPUs from reset in aarch64 instead of the aarch32 execution state on the RPI3? Is there a configuration input or does one have to rely on the RMR_EL3 to switch from aarch32 to aarch64? This would be the first step to run any aarch64 code. Alternatively, is it that we start in aarch64 and transition to aarch32?
Posts: 4
Joined: Mon Feb 29, 2016 9:52 am
by dom » Mon Feb 29, 2016 11:57 am
There is a config.txt entry to make the ARM start in 64-mode.
Code: Select all
arm_control=0x200


(This will obviously fail to boot unless you provide a kernel.img compiled for 64-bit mode).
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5069
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge
by achingupta » Mon Feb 29, 2016 12:58 pm
Thanks! that helps.
Posts: 4
Joined: Mon Feb 29, 2016 9:52 am
by dwelch67 » Mon Feb 29, 2016 2:34 pm
Is there documentation or is this like the RPI2 where the answer is "it is the same except the old core is replaced by a new core"?
Posts: 690
Joined: Sat May 26, 2012 5:32 pm
by swarren » Tue Mar 01, 2016 6:08 am
So "arm_control=0x200" selects AArch64 mode. What value selects AArch32 mode, or does that only happen if you completely remove that entry from config.txt?
Posts: 41
Joined: Tue Mar 01, 2016 5:56 am
by dom » Tue Mar 01, 2016 11:34 am
swarren wrote:So "arm_control=0x200" selects AArch64 mode. What value selects AArch32 mode, or does that only happen if you completely remove that entry from config.txt?

Bit 9 of arm_control determines if arm is booted in 32 or 64-bit mode. You could set arm_control=0x0 or remove it completely to boot in 32-bit mode.
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5069
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge
by Burngate » Tue Mar 01, 2016 5:27 pm
This'll probably appear somewhat thick, but ...
Do all four cores have to be in the same state?

I would have expected to be able to, eg., run RiscOS 32-bit on one core, and have 64-bit bare-metal stuff on a second (I've almost managed this with my Pi 2, RiscOS on one and bare-metal on a second)
but that arm_control=0x200 seems to indicate only bit 9 does anything
User avatar
Posts: 4738
Joined: Thu Sep 29, 2011 4:34 pm
Location: Berkshire UK
by dom » Tue Mar 01, 2016 5:57 pm
When the arm comes out of reset it samples a single bit signal to say whether boot (for all cores) should be in 64-bit or 32-bit mode.

64-bit mode does allow a way of switching back to 32-bit mode (but if you boot in 32-bit mode you can't switch to 64-bit).

So from a 64-bit boot it is probably possible to switch a core into 32-bit mode to run RISC OS (but the A53 documentation would be the place to determine what is possible).
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5069
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge
by achingupta » Wed Mar 02, 2016 5:45 pm
You can go from aarch64 execution state to aarch32 without resetting the core by using the correct ERET instruction. In order to go from aarch32 to aarch64, the reset management register can be used. However, it does have caveats.
Posts: 4
Joined: Mon Feb 29, 2016 9:52 am
by MrTomasz » Tue Mar 08, 2016 11:18 pm
Hi dom,

I have few questions :

1. When ARM boots in AArch64 mode, in which exception level (EL) we are while trying to load & run kernel.img ? Is it EL3 or we are already switched to EL2/EL1 by VC arm loader ?
2. Are there any other special magic things we have to do ? (arm_control=0x200 was undocumented until somebody asked)
3. In AArch64 mode, is kernel image still loaded @ 0x8000 or at different address?
4. Before [64-bit] kernel.img is entered, do we have all ARM cores configured, up and running or just the first one?

I believe I did minimum amount of required work to just boot kernel (*) and see log on UART, however still no luck...

(*) - not a "make ARCH=arm64 defconfig Image" simple shot
Posts: 10
Joined: Wed Jan 06, 2016 11:37 am
Location: London, UK
by dom » Wed Mar 09, 2016 11:43 am
MrTomasz wrote:1. When ARM boots in AArch64 mode, in which exception level (EL) we are while trying to load & run kernel.img ? Is it EL3 or we are already switched to EL2/EL1 by VC arm loader ?
2. Are there any other special magic things we have to do ? (arm_control=0x200 was undocumented until somebody asked)
3. In AArch64 mode, is kernel image still loaded @ 0x8000 or at different address?
4. Before [64-bit] kernel.img is entered, do we have all ARM cores configured, up and running or just the first one?

There is no 64-bit arm loader code. You should be running with kernel_old=1 and your 64-bit arm code should start from zero.
The arms will just be in the default reset state. i.e. all four cores will be running, starting from address zero.
It is up to your arm code to check the arm core number and ensure one is the "master" and make the others wait until it has set things up suitably.
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5069
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge
by MrTomasz » Wed Mar 09, 2016 12:20 pm
Thanks for the reply dom.

I was thinking about that if I should use kernel_old=1 to boot from 0x0 or not... I'll check that and let's see.
Posts: 10
Joined: Wed Jan 06, 2016 11:37 am
Location: London, UK
by MrTomasz » Thu Mar 10, 2016 4:38 pm
Hello again,

Still nothing... I think the quicker thing will be to test some baremetal program or some things like U-Boot or Coreboot - the reason for that is that Linux kernel doesn't like to be entered in EL3 mode, where the other things I mentioned can and are properly switching to EL2/EL1.

I've also tried boot-wrapper-arch64, but this one is producing ELF file with fancy addressing for kernel and DTB, which after objcopy (as kernel.img is not ELF file) makes me 256MB file... I need to relocate kernel image and DTB to make it smaller, and try it again...

I am also unsure about other plenty things, but would like to see at least simple kernel booting log (or at least few lines of it until it fails on something)...

Is there anyone who succeed at least with running baremetal AArch64 code which does ultra-simple thing like priting a char on UART or blinking a LED ? Just to see if it physically works and runs some code...

@dom
As I understand all posts regarding RPi3 SoC, the things like ARM core management, Interrupt controller, etc. - everything is same as on RPi2 ? -- I mean done via custom mailboxes and no GIC inside, no PSCI calls needed or whatever?
Posts: 10
Joined: Wed Jan 06, 2016 11:37 am
Location: London, UK
by dom » Thu Mar 10, 2016 6:18 pm
MrTomasz wrote:@dom
As I understand all posts regarding RPi3 SoC, the things like ARM core management, Interrupt controller, etc. - everything is same as on RPi2 ? -- I mean done via custom mailboxes and no GIC inside, no PSCI calls needed or whatever?


We are using an identical kernel on Pi2 and Pi3 so I believe everything must be the same. The only difference I'm aware of between the arm code on Pi2 and Pi3 is in the bootcode we inject from firmware.
The mechanism to enable SMP is slightly different:
Code: Select all
#ifdef BCM2710
        mrrc p15, 1, r0, r1, c15  @ CPU Extended Control Register
        orr r0, r0, #(1<<6)       @ SMP
        mcrr p15, 1, r0, r1, c15  @ CPU Extended Control Register
#else
        mrc p15, 0, r0, c1, c0, 1 @ Read Auxiliary Control Register
        orr r0, r0, #(1<<6)       @ SMP
        mcr p15, 0, r0, c1, c0, 1 @ Write Auxiliary Control Register
#endif

(BCM2710 is the Pi3 version). However if you are only using one core, that shouldn't be essential.
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5069
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge
by MrTomasz » Thu Mar 10, 2016 6:43 pm
Thanks for the reply.

Another question - machine specific code in arm/mach-bcm2709 - is it required for just proving that kernel code is executed and something is sent to UART (in this case bootlog) ? I don't care if kernel or soc will crash just right after first bits of init. I didn't finish this part yet, so that's why I am asking, as I don't want to do too much for the first shot to see if it actually does anything in aarch64.

I just do not have any proof that it actually runs aarch64 code properly...plus include EL3 to EL2/1 switch before kernel and we already complicated things a bit...


BTW. How did you validate AArch64 mode? Do you have any simple baremetal sample code ?
Posts: 10
Joined: Wed Jan 06, 2016 11:37 am
Location: London, UK
by swarren » Fri Mar 11, 2016 6:31 am
Here's something that demos AArch64 mode:
https://github.com/swarren/rpi-3-aarch64-demo

The compiler-optimization dependency is odd. I wonder if that's related to your issue?
Posts: 41
Joined: Tue Mar 01, 2016 5:56 am
by MrTomasz » Fri Mar 11, 2016 9:51 am
Thanks swarren!

I hope I can validate at least some basic things and start finally my kernel.

What do you mean by "compiler-optimization dependency is odd" ?
Posts: 10
Joined: Wed Jan 06, 2016 11:37 am
Location: London, UK
by rst » Fri Mar 11, 2016 11:08 am
swarren wrote:The compiler-optimization dependency is odd. I wonder if that's related to your issue?
Thanks for the demo! Please try to add
Code: Select all
disable_commandline_tags=1
to your config.txt file. The boot loader will otherwise overwrite your code (without -O2) at 0x100 with ATAGS.
Posts: 261
Joined: Sat Apr 20, 2013 6:42 pm
Location: Germany
by dom » Fri Mar 11, 2016 12:10 pm
swarren wrote:Here's something that demos AArch64 mode:
https://github.com/swarren/rpi-3-aarch64-demo

Nice!
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5069
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge
by swarren » Fri Mar 11, 2016 5:15 pm
rst wrote:Please try to add
Code: Select all
disable_commandline_tags=1
to your config.txt file. The boot loader will otherwise overwrite your code (without -O2) at 0x100 with ATAGS.


Thanks, that fixed it. I actually modified the code so that this option isn't needed though, by leaving a hole in the binary.
Posts: 41
Joined: Tue Mar 01, 2016 5:56 am
by marcus_c » Sun Mar 13, 2016 12:20 am
Hi.

I got U-Boot working in 64-bit mode.

To build it, do the following steps:
Code: Select all
git clone https://github.com/zeldin/u-boot-rpi3.git
cd u-boot-rpi3
make rpi_3_defconfig
make CROSS_COMPILE=aarch64-unknown-linux-gnu-

Copy the resulting u-boot.bin to your boot partition, and make sure that config.txt contains
Code: Select all
dtoverlay=pi3-miniuart-bt-overlay
kernel=u-boot.bin
kernel_old=1
arm_control=0x200
disable_commandline_tags=1

And presto, 64-bit U-Boot.

This should make it easier to get a kernel running.
Posts: 11
Joined: Sun Mar 13, 2016 12:15 am
by dom » Sun Mar 13, 2016 12:08 pm
marcus_c wrote:I got U-Boot working in 64-bit mode.

Excellent!
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5069
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge
by Electron752 » Tue Mar 15, 2016 12:27 pm
Hi,

I've been doing some reading of web pages and it appears that 64 bit ARM may not be that easy since there are some pointer dependencies between the firmware, bootloader, and linux kernel.

I was wondering if anybody has any thoughts on getting a 64 bit compile of the linux kernel running with the video and audio disabled. That should eliminate some of the dependency on the firmware. The console would simply be one of the COM ports of the SOC.

I'm thinking this would still be acceptable for server and embedded applications since the video isn't really required. People could use those cheap FTDI based USB to TTL COM cables that are floating all over the Internet.

Thoughts/Comments....

Thoughts.
Posts: 142
Joined: Mon Mar 02, 2015 7:09 pm
by ghans » Tue Mar 15, 2016 12:46 pm
Thats what the BSD guys are doing too ...


ghans
• Don't like the board ? Missing features ? Change to the prosilver theme ! You can find it in your settings.
• Don't like to search the forum BEFORE posting 'cos it's useless ? Try googling : yoursearchtermshere site:raspberrypi.org
Posts: 7203
Joined: Mon Dec 12, 2011 8:30 pm
Location: Germany
by MrTomasz » Tue Mar 15, 2016 2:58 pm
Electron752 wrote:Hi,

I've been doing some reading of web pages and it appears that 64 bit ARM may not be that easy since there are some pointer dependencies between the firmware, bootloader, and linux kernel.

I was wondering if anybody has any thoughts on getting a 64 bit compile of the linux kernel running with the video and audio disabled. That should eliminate some of the dependency on the firmware. The console would simply be one of the COM ports of the SOC.

I'm thinking this would still be acceptable for server and embedded applications since the video isn't really required. People could use those cheap FTDI based USB to TTL COM cables that are floating all over the Internet.

Thoughts/Comments....

Thoughts.


That's what I am exactly working on as a first step of bringing up RPi3 in AArch64 world, however was bit busy in last days...

With U-boot running in AArch64 (thanks to @marcus_c for the news), I think it should be much easier and faster to test it. AFAIK U-Boot exits EL3 state, so kernel should boot smoothly now.
Posts: 10
Joined: Wed Jan 06, 2016 11:37 am
Location: London, UK