Electron752
Posts: 142
Joined: Mon Mar 02, 2015 7:09 pm

Re: Entering aarch64 execution state

Wed Apr 06, 2016 12:45 am

OK, I've collected my changes on github for everyone's benefit.

The kernel source is here:
https://github.com/Electron752/linux

A complete compiled boot partition with the 64bit kernel and u-boot is here:
https://github.com/Electron752/boot64-rpi3

I would be curious if anybody wants to try it out. It has working 4 cpu's, video, USB, networking, and DMA. The MMC support is still the stock kernel driver so it's a bit flaky and has some performance issues. Porting the foundation's MMC/SD driver to 64bit would be a good thing to do next.

I''ve been testing the 64bit kernel with 64bit debian, but I suspect even 32bit raspbian will work. Although I don't know what the point of doing that would be.

IMPORTANT: To get all 4 cpus to initialize, it's necessary to use swarren's version of u-boot. Otherwise only 1 cpu will initialize. It's also necessary to build swarren's arm64 stub and cat it with the u-boot binary. I've done this already in my precompiled 64bit boot partition that I shared out.

swarren
Posts: 45
Joined: Tue Mar 01, 2016 5:56 am

Re: Entering aarch64 execution state

Wed Apr 06, 2016 2:03 am

schorsch76 wrote:
swarren wrote:I've only tested the spin table with the test app in my rpi-3-aarch64-demo repo. However, Eric Anholt got it working with the mainline kernel. It looks like that code is here:

https://github.com/anholt/linux/commit/ ... 317b0ecR30
(that's the latest commit in the bcm2837-64 branch in case it gets rebased)
Have you any link to see what he did? Mailling list?
Thanks
I don't think he posted any instructions. I imagine he just built.installed U-Boot, built/installed the kernel, and ran the relevant load/booti commands in U-Boot. Perhaps this link will help you with U-Boot: https://github.com/swarren/u-boot/commi ... b6f8c066d5. I don't know of any similar explicit instructions for the kernel, although the instructions written earlier in this thread look like they should work.

Electron752
Posts: 142
Joined: Mon Mar 02, 2015 7:09 pm

Re: Entering aarch64 execution state

Wed Apr 06, 2016 3:04 am

Just want to add that with my changes that I posted, I'm able to get X Windows to load and work just fine. So I think 64bit userland is all just going to work.

I'm running the xfce4 desktop + lightdm. Of course, this is unaccelerated X windows without sound, but it seems to run reasonably fast.

I might start a new topic soon since this topic has kind of lost it's original intention which was how to enter 64 bit mode. It's become more of a port the current linux kernel to RPI 3 running 64 bit arm.

Like I said, I think the next major thing that probably should be gotten to work is the port of the foundation's MMC driver. That should improve file I/O performance by a bunch. I also don't think we are ever going to see a "offical" linux kernel source release that supports the RPI 3 completely. So some reason, the foundation's kernel source has never been completely added to the upstream source.

Rascas
Posts: 448
Joined: Tue Mar 11, 2014 6:18 pm
Location: Porto, Portugal
Contact: Website

Re: Entering aarch64 execution state

Wed Apr 06, 2016 11:52 am

Maybe a stupid question, but does this kernel boot with the VC4 blob or the Eric Anholt open-source driver ?

I have not tested it, but great job on getting the 64 bit kernel working.

Electron752
Posts: 142
Joined: Mon Mar 02, 2015 7:09 pm

Re: Entering aarch64 execution state

Wed Apr 06, 2016 12:03 pm

Rascas wrote:Maybe a stupid question, but does this kernel boot with the VC4 blob or the Eric Anholt open-source driver ?

I have not tested it, but great job on getting the 64 bit kernel working.
I'm not sure what you are referring to. It boots with the official foundation firmware(I think this is the VC4 blob) bounced off of swarren's latest version of u-boot to setup the arm64 init environment.

The video is a port of the foundation video driver which for 2D graphics is essentially a dumb frame buffer. I don't have any accelerated 3D video working(And I might never make it work) .

As for MMC, I'm using the stock kernel driver written by Stephen Warren. It appears to be very old and it's very, very slow and flaky. I was planing to port the foundation MMC driver to 64bit for performance reasons but I don't have it working at this time.

User avatar
schorsch76
Posts: 22
Joined: Sun Apr 28, 2013 8:01 am

Re: Entering aarch64 execution state

Wed Apr 06, 2016 1:26 pm

There is an intersting Topic regarding 3D:

Code: Select all

Kernel driver status

The V3D kernel driver is now upstream in kernel 4.5. The devicetree has not been updated for the driver, so you can find a patch for that at: https://github.com/anholt/linux/tree/vc4-kms-v3d-squash-2-boot.

The Raspberry Pi Foundation's kernel tree now includes the driver in the current rpi-update 4.1 branch, and also the 4.5 branch.
https://dri.freedesktop.org/wiki/VC4/

Rascas
Posts: 448
Joined: Tue Mar 11, 2014 6:18 pm
Location: Porto, Portugal
Contact: Website

Re: Entering aarch64 execution state

Wed Apr 06, 2016 1:37 pm

Thanks for the anwsers.
What I was more interested to know is if this 64 bit kernel, does support hardware video decode (H264, VC1 MPEG2, etc).

For what I know, this is currently only supported whith the "normal" Broadcom video driver and not Eric Anhot's open source driver, hence my previous question.

Electron752
Posts: 142
Joined: Mon Mar 02, 2015 7:09 pm

Re: Entering aarch64 execution state

Wed Apr 06, 2016 1:42 pm

I might eventually look into it, but 3d video running on the RPI isn't that important to me right now. I don't play video games much anymore, mesa certainly support software 3d, and I think if you ssh into the rpi and use X window forwarding the application usually uses the 3d capability on the machine it is displayed on as opposed to the machine it is run on. I mostly got video to work just so that I didn't need to rely on the serial cable as much...

If 3D in 64bit is important to someone else, they can certainly use what I have put on github as a guide or a starting point and add 3d themself. Personally, I think better MMC support is much more important. In fact, MMC support right now is so slow and flaky that I moved the root partition to a USB flash drive and just use the MMC card for booting.

Electron752
Posts: 142
Joined: Mon Mar 02, 2015 7:09 pm

Re: Entering aarch64 execution state

Wed Apr 06, 2016 1:54 pm

Rascas:

Sorry I missed your question. The answer is no. It's software only. I only have a dumb video frame buffer working. Everything is software only. I was able to play a DVD rip on the RPI 3 in 64 bit mode though with ffplay without any issues. I didn't try HD video yet.

Electron752
Posts: 142
Joined: Mon Mar 02, 2015 7:09 pm

Re: Entering aarch64 execution state

Wed Apr 06, 2016 1:56 pm

BTW, I think the RPI 3 is fast enough to do software decoding. So hardware decoding is just icing on the cake.

Rascas
Posts: 448
Joined: Tue Mar 11, 2014 6:18 pm
Location: Porto, Portugal
Contact: Website

Re: Entering aarch64 execution state

Wed Apr 06, 2016 2:12 pm

Yeah.
It is still too early to reach conclusions, but from your ffmpeg benchmarks, it might be no advantages of running a 64 bit kernel + 64 bit userland on the RPi 3, for video (2D/3D) related stuff. It might be even worse, like someone said, because of the bigger memory and memory bandwidth needs of 64 bits binaries.

EDIT: Not thought right :S FFmpeg enconding only uses CPU and not GPU.
Last edited by Rascas on Wed Apr 06, 2016 2:21 pm, edited 1 time in total.

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5171
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: Entering aarch64 execution state

Wed Apr 06, 2016 2:16 pm

Electron752 wrote:BTW, I think the RPI 3 is fast enough to do software decoding. So hardware decoding is just icing on the cake.
Well, not for 1080p video, which needs hardware support.

For general access to gpu, then vchiq needs to work. Once that works (test with "vcgencmd version") then alsa audio, 3D, HW video decode, camera etc should start to work.

But, similarly to DMA, the GPU will expect pointers to be 32-bit bus addresses, so whatever is needed for DMA/sdcard will be needed for vchiq.

There may also be an issue with some GPU APIs which accept a "cookie". This is typically a 32-bit opaque value that is passed to GPU and is returned asynchronously with the output.
User code can use these for a variety of purposes, but an obvious one is a pointer to a class or structure, which in a 64-build may be a problem.

Solutions would be to increase the size of these cookies to 64-bits (which would break backwards compatibility with older 32-bit builds if not addressed somehow),
or compressing the userdata somehow as it is passed to the GPU and uncompressed on the way back (e.g. use the 32-bit cookie as an index into a 64-bit array).

I'm not sure how much code uses this. I don't believe omxplayer does, but I do believe MMAL (in kodi) does.

Electron752
Posts: 142
Joined: Mon Mar 02, 2015 7:09 pm

Re: Entering aarch64 execution state

Wed Apr 06, 2016 2:25 pm

Rascas wrote:Yeah.
It is still too early to reach conclusions, but from your ffmpeg benchmarks, it might be no advantages of running a 64 bit kernel + 64 bit userland on the RPi 3, for video (2D/3D) related stuff. It might be even worse, like someone said, because of the bigger memory and memory bandwidth needs of 64 bits binaries.
I agree. At this point, except for 64 bit swapfiles and memory mapped files I see very little benefit to 64 bit other then the geek factor :)

BTW, I'm completely testing with an unmodified version of 64bit Debian sid. So that kernel knows absolutely nothing about video decoding. That's all done in software by ffmpeg.

If anybody is interested, I can do a brief writeup on bootstraping a 64bit version of Debian. It requires using debootstrap and qmeu-static from a PC running Debian, both of which arn't the mostly friendly tools to use but it isn't that hard.

Electron752
Posts: 142
Joined: Mon Mar 02, 2015 7:09 pm

Re: Entering aarch64 execution state

Wed Apr 06, 2016 2:30 pm

dom wrote:
Electron752 wrote:BTW, I think the RPI 3 is fast enough to do software decoding. So hardware decoding is just icing on the cake.
Well, not for 1080p video, which needs hardware support.

For general access to gpu, then vchiq needs to work. Once that works (test with "vcgencmd version") then alsa audio, 3D, HW video decode, camera etc should start to work.

But, similarly to DMA, the GPU will expect pointers to be 32-bit bus addresses, so whatever is needed for DMA/sdcard will be needed for vchiq.
I think the key is that the BCM architecture is still mostly 32bit. So like with USB DMA, you end up with a very simple mapping between kernel virtual addresses(64 bit) and physical addresses(32 bit). The linux kernel source already has the setup builtin to deal with these kinds of issues. It's just nobody bothered to update the architecture specific parts of the code to arm64.

As long as the "cookies" don't get passed to userland, I don't see any problems.

But them, I'm starting to doubt the purpose of porting the linux kernel to 64 bit at this point...

Rascas
Posts: 448
Joined: Tue Mar 11, 2014 6:18 pm
Location: Porto, Portugal
Contact: Website

Re: Entering aarch64 execution state

Wed Apr 06, 2016 2:33 pm

Electron752 wrote: I agree. At this point, except for 64 bit swapfiles and memory mapped files I see very little benefit to 64 bit other then the geek factor :)

BTW, I'm completely testing with an unmodified version of 64bit Debian sid. So that kernel knows absolutely nothing about video decoding. That's all done in software by ffmpeg.

If anybody is interested, I can do a brief writeup on bootstraping a 64bit version of Debian. It requires using debootstrap and qmeu-static from a PC running Debian, both of which arn't the mostly friendly tools to use but it isn't that hard.
Yeah, wasn't thinking clearly has I was writing :P But then I made an edit.

xylnao
Posts: 3
Joined: Wed Apr 06, 2016 9:58 am

Re: Entering aarch64 execution state

Wed Apr 06, 2016 3:13 pm

Electron752,

Thank you for your impressive job. My rip3 enjoys the 64-bit world now!

Already there is a web page from Debian project which has almost complete description about the bootstrapping process.

https://wiki.debian.org/Arm64Port
Electron752 wrote:
Rascas wrote:Yeah.
If anybody is interested, I can do a brief writeup on bootstraping a 64bit version of Debian. It requires using debootstrap and qmeu-static from a PC running Debian, both of which arn't the mostly friendly tools to use but it isn't that hard.

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5171
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: Entering aarch64 execution state

Wed Apr 06, 2016 4:16 pm

Electron752 wrote: I''ve been testing the 64bit kernel with 64bit debian, but I suspect even 32bit raspbian will work. Although I don't know what the point of doing that would be.
Well, I'd be interested to know. Moving to a 64-bit kernel as standard is much more likely if you can just rpi-update a 32-bit raspbian image and still have it work.

Electron752
Posts: 142
Joined: Mon Mar 02, 2015 7:09 pm

Re: Entering aarch64 execution state

Wed Apr 06, 2016 4:57 pm

Sorry it took so long. I had to download raspbian again.

I'm happy to report that 32bit raspbian boots just fine with the 64bit boot environment and 64bit kernel. It took a very, very long time to boot up because the MMC driver needs alot of work but otherwise it works just fine including the x windows desktop.

I actually couldn't believe it at first, so I had to open a terminal in the raspbian desktop and run "uname -a". And guess what, it prints aarch64 as the kernel architecture!

BTW, since aarch64 seems to have little performance benefit ATM perhaps it would be good to use a 64 bit kernel on the RPI 3 but still keep 32 bit userland. That way it would be possible to get 64 bit swapfiles but not create the confusion or compatibility issues of a new release.

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5171
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: Entering aarch64 execution state

Wed Apr 06, 2016 5:15 pm

Electron752 wrote: I'm happy to report that 32bit raspbian boots just fine with the 64bit boot environment and 64bit kernel. It took a very, very long time to boot up because the MMC driver needs alot of work but otherwise it works just fine including the x windows desktop.
Thanks - it's good to confirm that works.
BTW, since aarch64 seems to have little performance benefit ATM perhaps it would be good to use a 64 bit kernel on the RPI 3 but still keep 32 bit userland. That way it would be possible to get 64 bit swapfiles but not create the confusion or compatibility issues of a new release.
It's still early days. There may be important applications that will show bigger gains.
The other point is gcc for 32-bit arm is quite mature. gcc for 64-bit arm is much less mature. The performance of 64-bit code compared to 32-bit code may increase with future compiler updates.

Electron752
Posts: 142
Joined: Mon Mar 02, 2015 7:09 pm

Re: Entering aarch64 execution state

Wed Apr 06, 2016 5:24 pm

It may also be good for supporting relatively closed source applications such as wolfgram/mathematica. They probably have no reason to recompile their application or want to deal with supporting 2 different architectures... Or supporting applications with hand optimized asm code.

Since the RPI 3 has only been out for less then a month and a few people on the web have taken it this far, it probably wouldn't take much work to get the full 64 bit kernel working.

pepedog
Posts: 1043
Joined: Fri Oct 07, 2011 9:55 am

Re: Entering aarch64 execution state

Wed Apr 06, 2016 7:40 pm

Got archlinuxarm booting to 64 bit userland, thanks Electron752
Can you add CONFIG_DEVPTS_MULTIPLE_INSTANCES=y sometime please.
Also what is your mkimage command as mine fail to boot

Electron752
Posts: 142
Joined: Mon Mar 02, 2015 7:09 pm

Re: Entering aarch64 execution state

Wed Apr 06, 2016 8:13 pm

pepedog wrote:Got archlinuxarm booting to 64 bit userland, thanks Electron752
Can you add CONFIG_DEVPTS_MULTIPLE_INSTANCES=y sometime please.
Also what is your mkimage command as mine fail to boot
Cool, I have some more changes pending as well. I might post a new version in a few days. I also found that a bunch of components in the current config don't make alot of sense. Such as the intel USB controllers, PCI, and a few other things. Yes, PCI would be cool on a RPI, but I don't think it's quite ready for that atm.

Here is the mkimage command I used for building boot.scr.uimg. I think you need to have built u-boot for 64 bit.

Code: Select all

tools/mkimage -A arm64 -O linux -T script -C none -n boot.scr -d boot.scr boot.scr.uimg
I noticed that u-boot doesn't have much of a menu system, but it supports PXE boot. I wonder how hard it would be to get syslinux to work so that people can have a menu system for choosing the kernel to boot. Say 32 bit for actual use for now, and 64 bit kernel for testing and messing around.

Perhaps even get u-boot itself to parse the menu scripts that syslinux uses without resorting to actually using syslinux.

Electron752
Posts: 142
Joined: Mon Mar 02, 2015 7:09 pm

Re: Entering aarch64 execution state

Thu Apr 07, 2016 1:04 am

I started attempting to get sound to work, and boy did I hit a brick wall really really fast. I see now that we have two paths that are mutually exclusive and I think I'm only seeing the tip of it.

The foundation has one view on the way things should work with the binary blob doing a bunch of things and the upstream source people want things done completely differently. And it appears that it is difficult to do things piecemeal. I started porting the foundation sound source to the upstream tree and the first thing that happened is that the MMC/SD card reader stopped working. Some of it has to do with configuration conflicts in the address space.

If I had to takes sides, I would probably side with the foundation. Not because I'm posting here, but because this is very standard practice in PC peripheral drivers. I'm finding that most PC network and video card drivers upload large binary blobs(aka firmware) to the subsystem every time it is powered on. Why pick on the little RPI when Nvidia, Amd, and Intel are already doing worse.

I mostly have all the pieces of the foundation's sound system moved over and compiling. I just have to cleanup a few more architecture specific pieces and for the most part it should just work.

swarren
Posts: 45
Joined: Tue Mar 01, 2016 5:56 am

Re: Entering aarch64 execution state

Thu Apr 07, 2016 2:03 am

What does "Some of it has to do with configuration conflicts in the address space." mean, in detail? I don't see how there would be any difference between the 32-bit and 64-bit ports here as far as a standard kernel driver is concerned. Equally, FW-vs-not shouldn't influence anything here either.

swarren
Posts: 45
Joined: Tue Mar 01, 2016 5:56 am

Re: Entering aarch64 execution state

Thu Apr 07, 2016 2:07 am

Electron752 wrote:I noticed that u-boot doesn't have much of a menu system, but it supports PXE boot. I wonder how hard it would be to get syslinux to work so that people can have a menu system for choosing the kernel to boot. Say 32 bit for actual use for now, and 64 bit kernel for testing and messing around.

Perhaps even get u-boot itself to parse the menu scripts that syslinux uses without resorting to actually using syslinux.
If you create file /extlinux/extlinux.conf, containing roughly syslinux syntax, you should find that U-Boot's default boot scripts will automatically locate and execute it. You can put multiple entries in the file, and get prompted which to boot.

However, I don't /think/ 64-bit U-Boot has support for booting a 32-bit kernel. I imagine it could be added though; the code is basically already there for the 32-bit port. The only thing you'd need to add is some assembly stub to switch bit size.

Return to “Bare metal, Assembly language”