jdonald
Posts: 415
Joined: Fri Nov 03, 2017 4:36 pm

Re: 64-bit weekly kernel autobuilds for RPi4 released (bcm2711_defconfig)

Wed Aug 07, 2019 4:56 pm

gilius2k15 wrote:
Mon Aug 05, 2019 1:34 pm
Raspex failed.

Ubuntu 18.04 upgraded to 19.04 - but the KVM would no longer initialize display.
I think you picked up on this in the other thread, but this sounds like it's due to missing the needed fixes in Mesa 19.1. From what I've read Ubuntu 19.04 uses Mesa 19.0 (Mesa 19.1 released in June, while the 04 means April), which is bound to fail with a black screen.

Instructions for cross-compiling Mesa: Raspberry Pi cross compile wiki. Usually works well for me, but if anyone has a ppa with a 19.1+ armhf build, would be great to use that instead.

Or did you get RaspEX running on the metal prior? If so, then I'd be more inclined to believe KVM is at fault here.
sakaki wrote:
Tue Jul 30, 2019 10:17 pm
minimal (!) testing I did showed (under fkms):
  • total_mem > 3072 breaks USB (but system still usable-ish if rootfs on microSD) with latest kernels; bounce-buffers apparently needed but not yet implemented in aarch64 (see e.g. this comment, and this one);
  • I'm having trouble getting HDMI audio to work with the most recent 64-bit kernels; but that may just be something in my local setup;
  • vlc --vout mmal_vout and omxplayer don't work, ton of mmal errors from the former; didn't try kodi;
  • dispmanx works (at least things like hello_dispmanx do);
  • no sign of the camera or v4l2 interfaces, even under the -bis kernel that supposedly turns them on (and even when booted with the appropriate firmware), but I may just have missed a setting there; this is on my todo list;
  • FIQ - don't know the correct answer here, @6by9? Also see below quote from rst.
Thanks sakaki. Good to hear anyone can use 3 GB now, dispmanx works, and it's probably just a configuration issue for camera interfaces.

Regarding accelerated video decode, I thought that was supposed to work according to 6by9's comment back in January?

gilius2k15
Posts: 55
Joined: Thu Jul 18, 2019 8:42 am

Re: 64-bit weekly kernel autobuilds for RPi4 released (bcm2711_defconfig)

Thu Aug 08, 2019 7:16 am

Raspex, Ubuntu 19.04 and 19.10 all have rainbow screen > black screen using sakaki's kernel regardless of VC4.

Ubuntu 18.04 (including a less than perfect upgrade to 19.04) both work but have a black screen when installing a desktop GUI - most likely due to the mesa version.

The failure to initialize kvm guest display turned out to be a corrupt UEFI bios file.

I would love to be able to compile the mesa so we could get ubuntu working properly with KVM, otherwise without VC4 it's too slow and unreliable. Gentoo is already at 19.1 and doesn't seem to have these problems.

The 19.10 docker is now working, but doesn't come with KVM enabled. VC4 is turned off by default - but 19.10 apparently has mesa 19.1 already. I need check that.
https://github.com/satmandu/docker-rpi4-imagebuilder/

jdonald
Posts: 415
Joined: Fri Nov 03, 2017 4:36 pm

Re: 64-bit weekly kernel autobuilds for RPi4 released (bcm2711_defconfig)

Thu Aug 08, 2019 9:13 pm

Got it, I misinterpreted your prior post. As RaspEX relies Ubuntu packages, at first I thought you were saying you upgraded to a 19.04 flavor inside RaspEX.

Rainbow screen: I think even if one deletes the entire root partition or configures the wrong partition UUID, having any kernel will normally take you into at least the start of a boot phase. Might this suggest that one of the boot files got corrupted on transfer? Just like earlier in this thread when System-p4.map and kernel8-p4.img were copied incompletely and ended up with the wrong size/md5sum.

gilius2k15
Posts: 55
Joined: Thu Jul 18, 2019 8:42 am

Re: 64-bit weekly kernel autobuilds for RPi4 released (bcm2711_defconfig)

Thu Aug 08, 2019 9:32 pm

I was thinking a similar thing - that the kernel should at least take us into the initial boot up regardless of what userland we apply it to - but I tested thoroughly ensuring all files are copied across properly; however, sakaki's kernel always results in a black screen after the rainbow screen for those particular distros.

I did start a post once to try to understand the connection between the bootloader, kernel, drivers and filesystem - but the true inner workings linux remains a mystery to me - particularly in the context of Raspberry Pis and Android phones not working the same way as desktop linux, etc.

jdonald
Posts: 415
Joined: Fri Nov 03, 2017 4:36 pm

Re: 64-bit weekly kernel autobuilds for RPi4 released (bcm2711_defconfig)

Sat Aug 10, 2019 8:58 pm

Got my Pi 4 and have started testing. With raspbian_buster_64bit_kernel.img in fkms, I've been unable to make use of accelerated graphics. Whether running from 32-bit userland or a 64-bit container, games always see the renderer reported as llvmpipe (CPU rendering) rather than Broadcom V3D.

glxgears takes about 170% CPU whereas on Raspbian Buster 32-bit it's closer to 7%. I rebooted and tested each configuration at least three times to make sure it's not a fluke.

This is like the problem cbratschi and I encountered at the end of that other thread, except with a 64-bit kernel it doesn't get fixed by rebooting.

@sakaki, @gilius2k15, et al.: is GPU rendering working for you with the 64-bit kernel on Raspbian or any of the other distros?

jerrm
Posts: 197
Joined: Wed May 02, 2018 7:35 pm

Re: 64-bit weekly kernel autobuilds for RPi4 released (bcm2711_defconfig)

Sun Aug 11, 2019 7:49 am

I did some testing using Sakaki's 64 bit Buster image and was surprised by how much of a performance gain there was for my test case. See https://www.raspberrypi.org/forums/view ... 0#p1517081.

Could there be any config/compile differences other than 32 bit vs 64 bit impacting the results?

jdonald
Posts: 415
Joined: Fri Nov 03, 2017 4:36 pm

Re: 64-bit weekly kernel autobuilds for RPi4 released (bcm2711_defconfig)

Mon Aug 12, 2019 5:12 am

jerrm wrote:
Sun Aug 11, 2019 7:49 am
Could there be any config/compile differences other than 32 bit vs 64 bit impacting the results?
Config differences including but not limited to:
* sakaki's image defaults to total_mem=1024. Did you change that before running your test? Having less memory available probably can't speed things up, but good to make everything as consistent as possible when you can.
* FIQ is broken for dwc_otg in the 64-bit kernel, and your experiment was transferring data on USB3 the whole time? Normally the 32-bit kernel should be faster / more functional / not broken when it comes to USB, but who knows maybe there's some issue of excess interrupts slowing things down.

webbsmurfen
Posts: 59
Joined: Fri Jun 27, 2014 12:22 pm
Location: Sweden

Re: 64-bit weekly kernel autobuilds for RPi4 released (bcm2711_defconfig)

Mon Aug 12, 2019 7:23 am

jdonald wrote:
Mon Aug 12, 2019 5:12 am
jerrm wrote:
Sun Aug 11, 2019 7:49 am
Could there be any config/compile differences other than 32 bit vs 64 bit impacting the results?
Config differences including but not limited to:
* sakaki's image defaults to total_mem=1024. Did you change that before running your test? Having less memory available probably can't speed things up, but good to make everything as consistent as possible when you can.
* FIQ is broken for dwc_otg in the 64-bit kernel, and your experiment was transferring data on USB3 the whole time? Normally the 32-bit kernel should be faster / more functional / not broken when it comes to USB, but who knows maybe there's some issue of excess interrupts slowing things down.
Just a notice..
Dont use more than 3Gb ( total_mem=3072 ) else the USB/Ethernet will start to behave erratic and malfunction

jerrm
Posts: 197
Joined: Wed May 02, 2018 7:35 pm

Re: 64-bit weekly kernel autobuilds for RPi4 released (bcm2711_defconfig)

Mon Aug 12, 2019 1:41 pm

jdonald wrote:
Mon Aug 12, 2019 5:12 am
jerrm wrote:
Sun Aug 11, 2019 7:49 am
Could there be any config/compile differences other than 32 bit vs 64 bit impacting the results?
Config differences including but not limited to:
* sakaki's image defaults to total_mem=1024. Did you change that before running your test? Having less memory available probably can't speed things up, but good to make everything as consistent as possible when you can.
* FIQ is broken for dwc_otg in the 64-bit kernel, and your experiment was transferring data on USB3 the whole time? Normally the 32-bit kernel should be faster / more functional / not broken when it comes to USB, but who knows maybe there's some issue of excess interrupts slowing things down.
I had actually set the total_mem to 2048 to match the pi. It was a 2GB pi so the tests were on even memory footing.

Yes usb3 the whole time. Ran a compare of a couple of the smaller 8GB image files and they restored correctly. Dmesg/syslog/etc were all clean during the run.

User avatar
sakaki
Posts: 376
Joined: Sun Jul 16, 2017 1:11 pm

Re: 64-bit weekly kernel autobuilds for RPi4 released (bcm2711_defconfig)

Mon Aug 12, 2019 3:25 pm

jdonald wrote:
Sat Aug 10, 2019 8:58 pm
Got my Pi 4 and have started testing. With raspbian_buster_64bit_kernel.img in fkms, I've been unable to make use of accelerated graphics. Whether running from 32-bit userland or a 64-bit container, games always see the renderer reported as llvmpipe (CPU rendering) rather than Broadcom V3D.

glxgears takes about 170% CPU whereas on Raspbian Buster 32-bit it's closer to 7%. I rebooted and tested each configuration at least three times to make sure it's not a fluke.

This is like the problem cbratschi and I encountered at the end of that other thread, except with a 64-bit kernel it doesn't get fixed by rebooting.

@sakaki, @gilius2k15, et al.: is GPU rendering working for you with the 64-bit kernel on Raspbian or any of the other distros?
Nope, still getting llvmpipe here under bcm2711_defconfig, Gentoo 64-bit userland also. The same kernel tree (but built under bcmrpi3_defconfig), with the same (64-bit) Gentoo userland (X, mesa etc), does give accelerated access booted on an RPi3, so not sure what the issue is there.

In some better news, with the tweaked config in bcm2711-kernel-bis, as of release 4.19.65.20190810 (at kernel commit f1c1b67b26ed), and a tweaked ffmpeg (patchlines, patches), v4l2 m2m hw codecs appear to be working on the RPi4 (on my Gentoo 64-bit (userland+kernel) alpha test image anyhow). The v2 camera module also works now on the RPi4 (under the same setup).

@RP{F,T} engineers: are there any in-house mesa patches etc. I need to be applying to get v3d acceleration from a 64-bit userland with the current bcm2711_defconfig 64-bit kernel (as can be done with the RPi3 when booted under a bcmrpi3_defconfig kernel and upstream mesa)?

Best, sakaki

User avatar
bomblord
Posts: 157
Joined: Sun Jul 14, 2019 2:54 am

Re: 64-bit weekly kernel autobuilds for RPi4 released (bcm2711_defconfig)

Mon Aug 12, 2019 3:28 pm

sakaki wrote:
Mon Aug 12, 2019 3:25 pm
jdonald wrote:
Sat Aug 10, 2019 8:58 pm
Got my Pi 4 and have started testing. With raspbian_buster_64bit_kernel.img in fkms, I've been unable to make use of accelerated graphics. Whether running from 32-bit userland or a 64-bit container, games always see the renderer reported as llvmpipe (CPU rendering) rather than Broadcom V3D.

glxgears takes about 170% CPU whereas on Raspbian Buster 32-bit it's closer to 7%. I rebooted and tested each configuration at least three times to make sure it's not a fluke.

This is like the problem cbratschi and I encountered at the end of that other thread, except with a 64-bit kernel it doesn't get fixed by rebooting.

@sakaki, @gilius2k15, et al.: is GPU rendering working for you with the 64-bit kernel on Raspbian or any of the other distros?
Nope, still getting llvmpipe here under bcm2711_defconfig, Gentoo 64-bit userland also. The same kernel tree (but built under bcmrpi3_defconfig), with the same (64-bit) Gentoo userland (X, mesa etc), does give accelerated access booted on an RPi3, so not sure what the issue is there.

In some better news, with the tweaked config in bcm2711-kernel-bis, as of release 4.19.65.20190810 (at kernel commit f1c1b67b26ed), and a tweaked ffmpeg (patchlines, patches), v4l2 m2m hw codecs appear to be working on the RPi4 (on my Gentoo 64-bit (userland+kernel) alpha test image anyhow). The v2 camera module also works now on the RPi4 (under the same setup).

@RP{F,T} engineers: are there any in-house mesa patches etc. I need to be applying to get v3d acceleration from a 64-bit userland with the current bcm2711_defconfig 64-bit kernel (as can be done with the RPi3 when booted under a bcmrpi3_defconfig kernel and upstream mesa)?

Best, sakaki
Any chance this means we're getting close to a full stand-alone 64-bit OS for the pi 4?


User avatar
sakaki
Posts: 376
Joined: Sun Jul 16, 2017 1:11 pm

Re: 64-bit weekly kernel autobuilds for RPi4 released (bcm2711_defconfig)

Mon Aug 12, 2019 3:48 pm

bomblord wrote:
Mon Aug 12, 2019 3:28 pm
Any chance this means we're getting close to a full stand-alone 64-bit OS for the pi 4?
Not sure about the exact status for other distros, but as to Gentoo, I aim to release a Pi3/4 64-bit Gentoo bootable image (v1.5.0) within 2 weeks, real-world NMIs permitting.
lingon wrote:
Mon Aug 12, 2019 3:36 pm
Have you seen this fresh patch to enable 4 GB RAM?
https://github.com/raspberrypi/linux/is ... -520269280
Yay! I'd missed that - thanks for the heads up. Hopefully this will go into upstream via this PR before too long; if not, I'll add it to my bcm2711-kernel-bis build later this week and test; if it looks OK, I'll ship it with the forthcoming 1.5.0 image ^-^

Best, sakaki

jdonald
Posts: 415
Joined: Fri Nov 03, 2017 4:36 pm

Re: 64-bit weekly kernel autobuilds for RPi4 released (bcm2711_defconfig)

Mon Aug 12, 2019 3:53 pm

Thanks for the heads-up lingon.

Looking at open pull requests, looks like #3063 is still outstanding from a month ago and will fix V3D in 64-bit.

User avatar
sakaki
Posts: 376
Joined: Sun Jul 16, 2017 1:11 pm

Re: 64-bit weekly kernel autobuilds for RPi4 released (bcm2711_defconfig)

Mon Aug 12, 2019 3:58 pm

jdonald wrote:
Mon Aug 12, 2019 3:53 pm
Thanks for the heads-up lingon.

Looking at open pull requests, looks like #3063 is still outstanding from a month ago and will fix V3D in 64-bit.
I'll drop that into -bis too, if it doesn't make it upstream. Thanks both for the pointers!

Best, sakaki

User avatar
bomblord
Posts: 157
Joined: Sun Jul 14, 2019 2:54 am

Re: 64-bit weekly kernel autobuilds for RPi4 released (bcm2711_defconfig)

Mon Aug 12, 2019 4:02 pm

sakaki wrote:
Mon Aug 12, 2019 3:48 pm
bomblord wrote:
Mon Aug 12, 2019 3:28 pm
Any chance this means we're getting close to a full stand-alone 64-bit OS for the pi 4?
Not sure about the exact status for other distros, but as to Gentoo, I aim to release a Pi3/4 64-bit Gentoo bootable image (v1.5.0) within 2 weeks, real-world NMIs permitting.

Best, sakaki
Awesome! I appreciate the hard work you're putting into this. Lord knows I don't have the knowledge for it xP.

User avatar
Gavinmc42
Posts: 3894
Joined: Wed Aug 28, 2013 3:31 am

Re: 64-bit weekly kernel autobuilds for RPi4 released (bcm2711_defconfig)

Tue Aug 13, 2019 12:54 am

Awesome! I appreciate the hard work you're putting into this. Lord knows I don't have the knowledge for it xP.
Me neither, lucky there are some experts around.
I've created a trivial patch to add bounce buffers on ARM64 for pcie-bcrmstb.
What is trivial for these Linux wizards is magic to us mere mortals.

Just ordered my second Pi4, a 2GB one just to avoid this issue.
But it might be fixed before my supplier has stock of the 4GB :D
Looking forward to having 4GB on a Gentoo64 box, that should make my big compiles less painful.
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges


User avatar
sakaki
Posts: 376
Joined: Sun Jul 16, 2017 1:11 pm

Re: 64-bit weekly kernel autobuilds for RPi4 released (bcm2711_defconfig)

Tue Aug 13, 2019 12:53 pm

jdonald wrote:
Mon Aug 12, 2019 3:53 pm
Thanks for the heads-up lingon.

Looking at open pull requests, looks like #3063 is still outstanding from a month ago and will fix V3D in 64-bit.
Just a quick heads-up; I applied #3063 to my -bis kernel and booted a 32-bit Raspbian Buster 'with desktop' userland on it. glxgears -info is now showing GL_RENDERER = V3D 4.2, and X is reporting "glamor X acceleration enabled on V3D 4.2", so target in sight now.

My 64-bit Gentoo userland (same kernel) still falls back to llvmpipe, but that's just because I don't have the v3d gallium option set in mesa, yet. Needs a tweak to the ebuild for that. Trying this now, will report back.
cjan wrote:
Tue Aug 13, 2019 3:19 am
does use last profiles?
https://www.gentoo.org/news/2019/08/11/arm64stable.html
Yes, the forthcoming image uses (as the 1.4.2 did) a custom profile, but this inherits from a stock (now stable) 17.0 profile.

Best, sakaki

User avatar
sakaki
Posts: 376
Joined: Sun Jul 16, 2017 1:11 pm

Re: 64-bit weekly kernel autobuilds for RPi4 released (bcm2711_defconfig)

Tue Aug 13, 2019 2:08 pm

With the tweaked mesa and patched kernel, V3D acceleration now working in 64-bit Gentoo userland on an RPi4 ^-^

Screenshot:
Image
(It's a dual framebuffer, although the above only shows the left screen)

Onto that bounce buffer patch now!

Best,
sakaki

User avatar
sakaki
Posts: 376
Joined: Sun Jul 16, 2017 1:11 pm

Re: 64-bit weekly kernel autobuilds for RPi4 released (bcm2711_defconfig)

Tue Aug 13, 2019 5:00 pm

Quick status report #2...

arm64 pcie bounce buffer patch now also applied (PR#3144); seems to work well. Now have the full 4G memory unlocked, and V3D acceleration, on a 64-bit kernel / 64-bit userland Pi4 system ^-^

Screenshot:
Image

Just in time for the soft-freeze on my forthcoming 1.5.0 Gentoo image ^-^

Thanks again to yaroslavros for PR#3144 and phire for PR#3063, and to jdonald and lingon for the pointers!

Will push the kernel mods to bcm2711-kernel-bis later today (or early tomorrow), and force a build / release cycle, so others can try them. Will post again once done.

Best, sakaki

sparkyhall
Posts: 145
Joined: Mon Aug 27, 2012 9:14 am

Re: 64-bit weekly kernel autobuilds for RPi4 released (bcm2711_defconfig)

Tue Aug 13, 2019 5:11 pm

This is all brilliant news, never tried Gentoo before, you've just tempted me ;)

Nice work Sakaki.

User avatar
bomblord
Posts: 157
Joined: Sun Jul 14, 2019 2:54 am

Re: 64-bit weekly kernel autobuilds for RPi4 released (bcm2711_defconfig)

Tue Aug 13, 2019 5:49 pm

sakaki wrote:
Tue Aug 13, 2019 5:00 pm
Quick status report #2...

arm64 pcie bounce buffer patch now also applied (PR#3144); seems to work well. Now have the full 4G memory unlocked, and V3D acceleration, on a 64-bit kernel / 64-bit userland Pi4 system ^-^

Screenshot:
Image

Just in time for the soft-freeze on my forthcoming 1.5.0 Gentoo image ^-^

Thanks again to yaroslavros for PR#3144 and phire for PR#3063, and to jdonald and lingon for the pointers!

Will push the kernel mods to bcm2711-kernel-bis later today (or early tomorrow), and force a build / release cycle, so others can try them. Will post again once done.

Best, sakaki
Well that's got me incredibly excited to say the least!

User avatar
Gavinmc42
Posts: 3894
Joined: Wed Aug 28, 2013 3:31 am

Re: 64-bit weekly kernel autobuilds for RPi4 released (bcm2711_defconfig)

Wed Aug 14, 2019 12:48 am

I'm sitting here waiting for the post to bring me my 2GB Pi4 and this news pops up.
Well done Sakaki.
This muddle salutes you.

Updated 20 seconds ago :o
Er, that's after midnight UK time.
This is all brilliant news, never tried Gentoo before, you've just tempted me ;)
It is the best supported 64bit OS for Pi's and probably the fastest.
1.50 will be very useful
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

cjan
Posts: 737
Joined: Sun May 06, 2012 12:00 am

Re: 64-bit weekly kernel autobuilds for RPi4 released (bcm2711_defconfig)

Wed Aug 14, 2019 12:56 am

'With the tweaked mesa' what does it mean? recompile or patch?

Return to “General discussion”