Entering aarch64 execution state

[Electron752]

What does "Some of it has to do with configuration conflicts in the address space." mean, in detail? I don't see how there would be any difference between the 32-bit and 64-bit ports here as far as a standard kernel driver is concerned. Equally, FW-vs-not shouldn't influence anything here either.

The upstream source uses the driver "brcm,bcm2835-cprman" to do a bunch of configuration, where as the foundation source uses the VC4 subsystem to do some it. The upstream source has a much simpler sound driver which doesn't appear to support audio through HDMI, but the foundation's audio driver's step on the I/O address space of the "brcm,bcm2838-cprman". So if their is only going to be one 64 bit port, then either one or the other has to be chosen.

Of course, one option would be to port both the foundation source and the upstream source to 64bit. But as it appears that after zeldin's checkin, I've been doing most of the kernel changes and I only have a limited amount of time on my hand. So I can't probably do both.

BTW, without getting too elaborate on this some of this cookie passing scares me as bad design. It might be OK for a low income person that just wants a cheap video game system or someone that want's to tinker with UARTs, but it's not good for embedded or network attached systems. Personally, I would take the runtime performance hit and have mapping tables(either hash or tree based) on both sides of the cookie passes, but that's just my opinion.

[swarren]

Ah right - multiple pieces of SW using the same registers. Perhaps this is nit-icking over terminology, but I guess I wouldn't call that just an address space conflict; the conflict is regarding control of logic in HW module itself, not just the address space used to access it.

I would assume that, just like for the 32-bit port, and assuming the Pi Fondation does want to switch their kernel to 64-bit ASAP, we will at least temporarily see separate downstream/upstream kernel versions. Things often just work out that way.

In 32-bit land, I assume (although I haven't heard anyone confirm this explicitly), that the upstream driver work will gradually be incorporated into new revisions of the downstream SW. I've heard talk of some (future?) Pi Foundation kernel already having pulled in his upstream VC4 driver. I imagine that since Eric (who does most of the core upstream stuff) works for Broadcom, the SoC vendor, he'll drive or push for that to happen, and the Pi Foundation will probably want it too since it'll mean having to maintain a smaller set of patches relative to mainline. Hopefully other driver conversions will follow over time.

[Electron752]

But I thought that the whole reason we have this issue is that companies like Broadcom want to keep parts of the chip hidden to protect IP. So I don't think the whole thing is every going to be completely open sourced without some kind of binary blob. Or at least not the whole things without some limited functionality.

Just look what happened with the orignal IBM PC. IBM chose to use all standard components so it got cloned really, really fast. Look were IBM is today.

So I think if Broadcom has any sense of protecting their pocketbooks, they won't go too far down that path...

[swarren]

There are almost certainly HW modules that specs won't be published for and/or physically can't be accessed from the ARM CPU due to the way the SoC is constructed internally. The VC FW will always have to manage those. I'd expect the rest to converge on control via the ARM CPU though. Eric is making that happen upstream IIUC.

[Electron752]

Assuming HDMI audio never makes it into the upstream source because of business reasons, the question becomes how important is HDMI audio.

For embedded systems, servers, and IoT it probably isn't important at all. For Desktop users, it's probably critical.

Since I already have an Intel PC and am more personally interested in embedded, servers, and IoT I can personally live without it. And accelerated 3D for that matter.

[Electron752]

Just thinking off the top of my head, perhaps a compromise would be to have both sets of drivers in the kernel image. I can certainly rename the driver ids to be difference. Then the end user or whoever can apply some kind of switch that activates one set for a desktop scenario or activates the other set for a embedded scenario.

I'm looking at the drivers, and the drivers themselves are really not that big. So perhaps it is possible to port both of them.

I need to ponder this whole thing for awhile...

[Electron752]

Hi all.

After ponding for awhile, I've come to the conclusion that I'm doing this backward. Rather then try to copy the foundation files back into the upstream source, I think the better method is to start with the foundation source and attempt to merge in the small number of changes that were needed to the upstream to get things to boot/work. This seems to make more sense at this point.

Basically, with some very small exceptions the 32 bit version and the 64 bit version are more then same then they are different. For example, only a few lines need to change in the device tree for 64 bit. So why not just have a few #ifdefs in the device tree source.

I've learned alot, but it's clear now this isn't the correct path. I'm going to retrace what it took to get the upstream source to boot and do it over again with the foundation tree. Perhaps I can comment out or disable stuff until I get a bootable system, rather then the other way around.

Anyway, I'm going to go offline for a week or so and work on this on my own. I'll check back in if I get stuck, or I make some major headway.

Thanks.

[xylnao]

Adding some kernel configs and packages to Electron752's system, I successfully got the Docker Engine working!

[quote="Electron752"]Just want to add that with my changes that I posted, I'm able to get X Windows to load and work just fine. So I think 64bit userland is all just going to work.

I'm running the xfce4 desktop + lightdm. Of course, this is unaccelerated X windows without sound, but it seems to run reasonably fast.

[dom]

After ponding for awhile, I've come to the conclusion that I'm doing this backward. Rather then try to copy the foundation files back into the upstream source, I think the better method is to start with the foundation source and attempt to merge in the small number of changes that were needed to the upstream to get things to boot/work. This seems to make more sense at this point.

Yes, I'd agree that is best. Note that later foundation kernels use increasingly more upstream drivers. Start with at least the rpi-4.4.y tree (which moves to device tree only, and drops the platform devices and moves to upstream interrupt controller, as well as most of the supported peripheral drivers).
rpi-4.5.y is cleaner still, and believed to be fully working but doesn't have the testing of rpi-4.4.y (which is used by OpenELEC, OSMC, BRANCH=next rpi-update and will be moved to very soon).
rpi-4.6.y is still buggy for me (sometimes stalls on boot for some minutes, but seems functional after that).

[TehWoomiestRPI]

Hey everyone, sorry to just hop in.

Using Electron's prebuilt image I was able to load a Arch Linux 64bit user land.

I was able to get the mmc to be more reliable by adding dtoverlay=sdhost to config.txt. The performance is still less than desired, but I noticed a LOT less errors, and I've haven't had a Kernel Panic sense using this overlay.

Hope this tidbit of information helps!

[Electron752]

HI,

I tried getting the foundation kernel source to work in 64 bit, but I completely failed to get something to boot. I think too much is different in the way things are initialized between the two trees especially the IRQ handler code. I don't really know enough about the low layers of the architecture and don't really have the correct tools such as a JTAG debugger to really be able to debug that part of the source.

On the bright side:
1. I got the hardware random number generator to work in the upstream tree which allows a bunch of networking and crypto modules to be enabled. This appears to just have been a typo in the device tree.

2. I got both the foundation SD and MMC drivers to build in the upstream tree without any modifications. The MMC driver works just fine with DMA enabled, the SD driver doesn't so I'm leaving it disabled in the device tree. It appears to have something to do with clocks. It's possible I'm just doing something wrong in the device tree.

With #2, I was able to remove the upstream SD driver from the build configuration and the device tree. MMC card performance seems decent now and the weird hangs on card i/o are gone..

I will probably post a new version this weekend. I want to get more of the standard modules re-enabled before I post. Since the foundation is slowly moving to the upstream version, I wonder what it would take for someone to get these changes merged back in the upstream tree?

I might stop at this point for awhile since the things I personally care about are working and 64 bit on the RPI 3 is still in very early stages.

[dom]

Since the foundation is slowly moving to the upstream version, I wonder what it would take for someone to get these changes merged back in the upstream tree?

If you are interested in the upstreaming effort then check the linux-rpi-kernel mailing list.
Work is going on continually to improve upstream support, but it is a slow, slow process.

It is unlikely we'll get everything we need upstream - e.g. vchiq and dwc_otg are controversial. But the aim is to use as few downstream patches as possible.
Currently upstream is significantly lacking in features compared to downstream.

[n0w4y]

I've only tested the spin table with the test app in my rpi-3-aarch64-demo repo. However, Eric Anholt got it working with the mainline kernel. It looks like that code is here:

https://github.com/anholt/linux/commit/ ... 317b0ecR30
(that's the latest commit in the bcm2837-64 branch in case it gets rebased)

Have you any link to see what he did? Mailling list?
Thanks

I don't think he posted any instructions. I imagine he just built.installed U-Boot, built/installed the kernel, and ran the relevant load/booti commands in U-Boot. Perhaps this link will help you with U-Boot: https://github.com/swarren/u-boot/commi ... b6f8c066d5. I don't know of any similar explicit instructions for the kernel, although the instructions written earlier in this thread look like they should work.

I have managed to reproduce the boot/scheme from source repositories.

1. https://github.com/raspberrypi/tools
-> build armstubs/armstub8.bin
The Makefile is somewhat dysfunctional but easy to read. Build manually.
2. pad armstub8.bin to 32768 bytes
3. build u-boot.bin from https://github.com/swarren/u-boot <remotes/origin/rpi_dev branch>
-> armstub8.bin is linked to load at 0x0
-> swarren u-boot is linked to load/run at 0x8000 ; the padding results in a binary that jumps to correct offset at 0x8000 in RAM (post load).
4. cat armstub8.bin_padded u-boot.bin > u-boot.padded.bin

Among other things, armstub8 preamble drops the primary core to EL2 and puts the secondaries into
spin loops (in EL2). The primary branches to u-boot (at 0x8000).

The secondary cores are brought out of spin by spin-table/method entries in DTS.

With mini-uart as console:
setenv bootargs "earlyprintk console=ttyS0,115200 root=/dev/mmcblk0p2 rootfs=ext4 rootwait"
setenv load_dtb "fatload mmc 0:1 ${fdt_addr_r} bcm2837-rpi-3-b.dtb"
setenv load_kernel "fatload mmc 0:1 ${kernel_addr_r} image"
setenv boot_it "booti ${kernel_addr_r} - ${fdt_addr_r}"
setenv bootcmd "run load_dtb ; run load_kernel ; run boot_it"
saveenv

Populate /dev/mmcblk0p2 (2nd partition of uSD) with desired runtime.

Repo/commits:
tools: b65595ffb74e5853ba61916f49bdbccfc54f1300 <Fri Apr 8>
u-boot: c805f24f06121afbd27dce0c741ec635493d9495 <Wed Apr 6>

Given that git repositories are moving targets the above commits produce a
working setup.

Hope this helps.

[TehWoomiestRPI]

^^^^^^^^^

OMG this is awesome!

Thank you!

[fsck]

I'm not entirely sure how to make a suitable rootfs for this. Could someone be so kind as to provide a complete SD card image? Or/and write up a little guide?

[marcus_c]

fsck:
You can use a Gentoo stage3 as a starting point:
http://mirror.bytemark.co.uk/gentoo/experimental/arm64/

[xylnao]

I'm not entirely sure how to make a suitable rootfs for this. Could someone be so kind as to provide a complete SD card image? Or/and write up a little guide?

Hi, fsck!

I put things together and made a sample image with which you can
do not very much, but not bad for the first try.

Download this image, uncompress it and write it into your microsd
card (bigger than 2G) then your rpi3 immediately boots up.

http://www.tom-yam.or.jp/rpi3/rpi3-arm6 ... 414.img.xz

This image consists of the first vfat boot partition and the second
ext4 root partition (debootstrapped from the Debian distribution).
The password of the root account is 'raspberry'. If you prefer a serial
console, rename the file 'boot.scr.uimg.serialconsole' on the boot
partition to 'boot.scr.uimg'.

Enjoy!

[dmc1954]

Thank you Xylnao!

I was able to boot my rpi3 with your image. The lack of a GUI Desktop was not a problem, since I prefer using a command line.

Some of the things that I was able to do with this image was:
1) Configure eth0 with ifconfig/route
2) Run "adduser pi"
3) Run "systemctl restart sshd"
4) Resize /dev/mmcblk0p2 to use the remaing 8GB SDcard and reboot
5) Run apt-get update and apt-get upgrade using the Debian arm64/aarch64 repository
6) Run uname -a
7) Run gcc -v
8 ) Install vim with apt-get
9) Backuped the 8GB SDcard

Thanks again,
David

[Franknberry]

I'm not entirely sure how to make a suitable rootfs for this. Could someone be so kind as to provide a complete SD card image? Or/and write up a little guide?

Hi, fsck!

I put things together and made a sample image with which you can
do not very much, but not bad for the first try.

Download this image, uncompress it and write it into your microsd
card (bigger than 2G) then your rpi3 immediately boots up.

http://www.tom-yam.or.jp/rpi3/rpi3-arm6 ... 414.img.xz

This image consists of the first vfat boot partition and the second
ext4 root partition (debootstrapped from the Debian distribution).
The password of the root account is 'raspberry'. If you prefer a serial
console, rename the file 'boot.scr.uimg.serialconsole' on the boot
partition to 'boot.scr.uimg'.

Enjoy!

Thanks for your Image
its working very well
got Lxde on it
i am writing this with epiphany-browser from aarch64

[Hallo32]

Hi,

will there be a "full" image for the aarch64 in the future?

Thanks @ll for the work!

[mstorsjo]

FWIW, I tried the prebuilt kernel/boot partition, and got it working fine with a normal 32 bit raspbian userland. (This allows me to run statically linked 64 bit binaries at least.) I ran into some issues with the networking though.

One gotcha that I didn't see mentioned earlier in this thread, was that eth0 got a random MAC address each time. Normally, the bootloader (/boot/start*) figures out the serial number and generates the device's MAC address based on that, passing it to the kernel via a command line parameter (as smsc95xx.macaddr=B8:27:EB:AA:BB:CC, see /proc/cmdline on a normal 32 bit kernel setup). With u-boot, this isn't set up, so the driver picks a random address instead.

One way of working around this that I figured out was to add a file such as /etc/modprobe.d/smsc95xx.conf, with the following content:

Code: Select all

options smsc95xx macaddr=B8:27:EB:AA:BB:CC

Additionally, before I got this fixed, I mostly didn't get "link up" on eth0 at all; I got it once with a random MAC address, but then for two whole days I never got this interface up unless I ran with the normal 32 bit kernel. IIRC, after fixing the MAC address, it still didn't come up, but the following day it seems to come up just fine. Not sure if this is related to the random MAC address or if it is some unrelated issue I was seeing.

[dom]

Additionally, before I got this fixed, I mostly didn't get "link up" on eth0 at all; I got it once with a random MAC address, but then for two whole days I never got this interface up unless I ran with the normal 32 bit kernel. IIRC, after fixing the MAC address, it still didn't come up, but the following day it seems to come up just fine. Not sure if this is related to the random MAC address or if it is some unrelated issue I was seeing.

Probably related to /etc/udev/rules.d/70-persistent-net.rules
That attempts to give the same "eth0"/"eth1"/etc names to the same ethernet adapters (identified by MAC address).
I suspect that random MAC addresses are causing increasing "eth7"/"eth8" etc names to be used.

I believe deleting /etc/udev/rules.d/70-persistent-net.rules should be safe and remove these unwanted entries.

[dukla2000]

Download this image, uncompress it and write it into your microsd card (bigger than 2G) then your rpi3 immediately boots up.

Many thanks to all in this thread progressing arm64. Have been stabbing around with xylnao image for the last 24 hours, got a long way to a fairly usable destop system (update, upgrade, added user, added lxde etc) including networking via USB tether to my phone. Added sound with no ill effects to config.txt but didn't test it.

Ground to a halt on BT: could install bluetooth package OK (pulls bluez) and when I plugged my BT dongle in got promising noises in dmesg. But bluetoothctl was unresponsive (except to CTL-C), the desktop would hang and any boot with dongle plugged in would hang, if I removed dongle could get back to where I was.

This isn't intended as a bug report. It is intended as huge kudos/strokes to all the folk who have been getting aarch64 this far on a RPi3 - thanks, congrats and (hopefully) keep up the amazing work!

[mstorsjo]

Probably related to /etc/udev/rules.d/70-persistent-net.rules
That attempts to give the same "eth0"/"eth1"/etc names to the same ethernet adapters (identified by MAC address).
I suspect that random MAC addresses are causing increasing "eth7"/"eth8" etc names to be used.

I believe deleting /etc/udev/rules.d/70-persistent-net.rules should be safe and remove these unwanted entries.

I don't seem to have such a file (very recent raspbian), and the interface was named eth0 in all the cases where I observed it not to get the link up.

[Guillaume75]

Hello list,

Thank you xylnao for your share... it works like a charm.
I installed LXDE and lightdm.
BTW when running inxi -F is shows;
Kernel: 4.5.0+ aarch64 (32 bits)
I was wondering why 32 bits?

Also did you succeed to make work wireless?
Thanks again for your work!
Guillaume