Easily run 64-bit Debian Buster packages on a Raspbian system: new RPi4 / RPi3 image released (systemd-nspawn, LXDE)

[sakaki]

Ok so I got it up and running, and after installing my favorite apps (took fairly long time), below is is the screen capture of the desktop.

Great! Kodi is a bit weird given the way it takes over the desktop; I have it running on the gentoo-on-rpi3-64bit image, but haven't tried it in this setup.

[LTolledo]

Just further update:

with all the apps running response began to become sluggish, checking via htop found the swap maxed out.
So I increased the swap file size from default 100 to 1000

response improved, but still moving program windows around leaves "after image" (sort of the way the old windows solitaire cards do after completing the game).
maybe graphics system not coping up as fast as possible.

No complaints, just pointing observations....

if there are other things that you would like to be tested, feel free to enumerate. Will happily test it !

[ShiftPlusOne]

First go at packaging up the kernel:
https://www.dropbox.com/s/ngjj4uo82wojl ... l.tar?dl=1

Pretty much identical to raspberrypi-kernel.

[Rascas]

@LTolledo what is the output of:
apt-cache policy kodi
In both 32 and the 64 bit chroot?
I didn't tested this image yet but if it runs the vc4-fkms graphics driver, Kodi won't run good yet. It still needs some patching to have hardware video decoding, unfortunately I don't have the skills to do that by myself.

[LTolledo]

@Rascas (and those interested as well)

Here's the 32bit version

Code: Select all

[email protected]:~# apt-cache policy kodi
kodi:
  Installed: 2:18.0-1~stretch
  Candidate: 2:18.0-1~stretch
  Version table:
 *** 2:18.0-1~stretch 500
        500 http://archive.raspberrypi.org/debian stretch/main armhf Packages
        100 /var/lib/dpkg/status
     2:17.1+dfsg1-3 500
        500 http://raspbian.raspberrypi.org/raspbian stretch/main armhf Packages
[email protected]:~# 

the 64bit version:

Code: Select all

[email protected]:~# apt-cache policy kodi
kodi:
  Installed: 2:17.1+dfsg1-3
  Candidate: 2:17.1+dfsg1-3
  Version table:
 *** 2:17.1+dfsg1-3 500
        500 http://deb.debian.org/debian stretch/main arm64 Packages
        100 /var/lib/dpkg/status
[email protected]:~# 

anything that I should be "afraid" of?

Update:
actually there is something that was afraid of....
both 32bit and 64bit Kodi dont work. today I tried running 32bit Kodi. the hourglass appeared then disappeared after few seconds
same with 64bit Kodi....
maybe still needs work.... I'll be patient.... keep up the good work guys/gals!

anything else you want tested? be happy to oblige...

Update2:
System hanged after doing the following
While chromium running, 2 tabs (Raspberry Pi Forums, SD Association)
Opened 32bit terminal: ran update and upgrade.... completed no problem... terminal window remained active
Opened 64bit terminal: ran update..... system hanged...
no choice but to do hard reset (AC side power off/ AC side power ON)

is update and upgrade different on 32bit and 64bit sides?

[sakaki]

Just further update:

with all the apps running response began to become sluggish, checking via htop found the swap maxed out.
So I increased the swap file size from default 100 to 1000

response improved, but still moving program windows around leaves "after image" (sort of the way the old windows solitaire cards do after completing the game).
maybe graphics system not coping up as fast as possible.

No complaints, just pointing observations....

if there are other things that you would like to be tested, feel free to enumerate. Will happily test it !

So, if you have a chance, one thing that would be useful would be to install, using a regular 32-bit terminal on the same raspbian-nspawn-64 image you are currently using, each of the core apps you use (libreoffice etc. or whatever). You can of course have the 32-bit and 64-bit versions installed simultaneously (as e.g. Chromium does on your system right now). Then reboot, and open each your core apps in 32-bit form, and measure the memory used for each. Reboot, and do the same measurements, but use the 64-bit variants of each package this time. Then please tabulate a 32 vs 64-bit memory usage table and report it in this thread (this won't be quite a fair A vs B of course, since there are different versions of core packages in the aarch32 and aarch64 repos (e.g. Chromium), but it'll still be interesting).

Also, when you run a "fully-loaded" desktop using the 32-bit apps (from a clean boot), on this image using the fkms driver as usual, how does the performance (swap, window ghosting etc.) compare to the same loadout using the 64-bit variants of each package?

It may be worth using zswap as gentoo-on-rpi3-64bit does, this can materially improve performance when say alt-tabbing between Chromium and libreoffice, quite a normal use case.

No obligation to do any of this of course, but if you have time it'd be very useful.

Many thanks for taking out the time to contribute on this!

Best, sakaki

[LTolledo]

Hi there Sasaki-san

I've already got the 32bit and 64bit of my fave programs installed (thats why it took a long time to "install")

So you want me to test:

Clean boot32
run htop32
start 32bit core program (libreoffice calc)
note memory /swap
start another 32bit core program (gimp)
note memory/swap
and another 32bit program.....and another (maybe try to load as much as 10 programs running)
tabulate results
shutdown

Clean boot64
run htop64 (or htop32 preferred?)
start 64bit core program (libreoffice calc64)
note memory/swap
start another 64bit core program (gimp64)
note memory/swap
and another 64bit program.....and another (maybe try to load as much as 10 programs running)
tabulate results
shutdown

report results to this forum

if the above sequence is OK then I'll make time.

Also noted while writing halfway to this post I installed 64bit htop. System froze during the "processing triggers"
regained control after about a minute (give or take 10 secs).

[Rascas]

@Rascas (and those interested as well)

Here's the 32bit version

Code: Select all

[email protected]:~# apt-cache policy kodi
kodi:
  Installed: 2:18.0-1~stretch
  Candidate: 2:18.0-1~stretch
  Version table:
 *** 2:18.0-1~stretch 500
        500 http://archive.raspberrypi.org/debian stretch/main armhf Packages
        100 /var/lib/dpkg/status
     2:17.1+dfsg1-3 500
        500 http://raspbian.raspberrypi.org/raspbian stretch/main armhf Packages
[email protected]:~# 

the 64bit version:

Code: Select all

[email protected]:~# apt-cache policy kodi
kodi:
  Installed: 2:17.1+dfsg1-3
  Candidate: 2:17.1+dfsg1-3
  Version table:
 *** 2:17.1+dfsg1-3 500
        500 http://deb.debian.org/debian stretch/main arm64 Packages
        100 /var/lib/dpkg/status
[email protected]:~# 

anything that I should be "afraid" of?

Update:
actually there is something that was afraid of....
both 32bit and 64bit Kodi dont work. today I tried running 32bit Kodi. the hourglass appeared then disappeared after few seconds
same with 64bit Kodi....
maybe still needs work.... I'll be patient.... keep up the good work guys/gals!

anything else you want tested? be happy to oblige...

32-bit:
Installed: 2:18.0-1~stretch - This version won't work with this image I believe, it is the optimized version for the Broadcom drivers. This version should work: 2:17.1+dfsg1-3

64-bit:
2:17.1+dfsg1-3 - This version should work, like the 32bit one. Only the logs will tell.

[LTolledo]

32-bit:
Installed: 2:18.0-1~stretch - This version won't work with this image I believe, it is the optimized version for the Broadcom drivers. This version should work: 2:17.1+dfsg1-3

64-bit:
2:17.1+dfsg1-3 - This version should work, like the 32bit one. Only the logs will tell.

As mentioned in previous post both Kodi32 and Kodi64 failed to run
attaching the top part of the kodi64 log (located at /home/pi/.kodi/temp)

Code: Select all

19:40:50.858 T:547567643216  NOTICE: special://profile/ is mapped to: special://masterprofile/
19:40:50.858 T:547567643216  NOTICE: -----------------------------------------------------------------------
19:40:50.858 T:547567643216  NOTICE: Starting Kodi from Debian (17.1 Debian package version: 2:17.1+dfsg1-3). Platform: Linux ARM 64-bit
19:40:50.858 T:547567643216  NOTICE: Using Release Kodi from Debian x64 build
19:40:50.858 T:547567643216  NOTICE: Kodi from Debian compiled from 2:17.1+dfsg1-3 by GCC 6.3.0 for Linux ARM 64-bit version 4.9.25 (264473)
19:40:50.858 T:547567643216  NOTICE: Running on Debian GNU/Linux 9 (stretch), kernel: Linux ARM 64-bit version 4.14.97-v8-0448a1dbea0f-bis+
19:40:50.865 T:547567643216  NOTICE: FFmpeg version/source: 3.2.12-1~deb9u1
19:40:50.865 T:547567643216  NOTICE: 4 CPU cores available
19:40:50.865 T:547567643216  NOTICE: ARM Features: Neon disabled
19:40:50.865 T:547567643216  NOTICE: special://xbmc/ is mapped to: /usr/share/kodi
19:40:50.865 T:547567643216  NOTICE: special://xbmcbin/ is mapped to: /usr/lib/aarch64-linux-gnu/kodi
19:40:50.865 T:547567643216  NOTICE: special://xbmcbinaddons/ is mapped to: /usr/lib/aarch64-linux-gnu/kodi/addons
19:40:50.865 T:547567643216  NOTICE: special://masterprofile/ is mapped to: /home/pi/.kodi/userdata
19:40:50.865 T:547567643216  NOTICE: special://envhome/ is mapped to: /home/pi
19:40:50.865 T:547567643216  NOTICE: special://home/ is mapped to: /home/pi/.kodi
19:40:50.865 T:547567643216  NOTICE: special://temp/ is mapped to: /home/pi/.kodi/temp
19:40:50.866 T:547567643216  NOTICE: special://logpath/ is mapped to: /home/pi/.kodi/temp
19:40:50.866 T:547567643216  NOTICE: The executable running is: /usr/lib/aarch64-linux-gnu/kodi/kodi.bin
19:40:50.866 T:547567643216  NOTICE: Local hostname: debian-stretch-64
19:40:50.866 T:547567643216  NOTICE: Log File is located: /home/pi/.kodi/temp//kodi.log
19:40:50.866 T:547567643216  NOTICE: -----------------------------------------------------------------------
19:40:51.102 T:547567643216   ERROR: DBus: Error org.freedesktop.DBus.Error.ServiceUnknown - The name org.freedesktop.UPower was not provided by any .service files
19:40:51.594 T:547567643216  NOTICE: load settings...
19:40:51.730 T:547567643216  NOTICE: Found 1 Lists of Devices
19:40:51.730 T:547567643216  NOTICE: Enumerated PULSE devices:
19:40:51.730 T:547567643216  NOTICE:     Device 1
19:40:51.730 T:547567643216  NOTICE:         m_deviceName      : Default
19:40:51.730 T:547567643216  NOTICE:         m_displayName     : Default
19:40:51.730 T:547567643216  NOTICE:         m_displayNameExtra: Default Output Device (PULSEAUDIO)
19:40:51.731 T:547567643216  NOTICE:         m_deviceType      : AE_DEVTYPE_PCM
19:40:51.731 T:547567643216  NOTICE:         m_channels        : FL,FR
19:40:51.731 T:547567643216  NOTICE:         m_sampleRates     : 5512,8000,11025,16000,22050,32000,44100,48000,64000,88200,96000,176400,192000,384000
19:40:51.731 T:547567643216  NOTICE:         m_dataFormats     : AE_FMT_U8,AE_FMT_S16NE,AE_FMT_S24NE3,AE_FMT_S24NE4,AE_FMT_S32NE,AE_FMT_FLOAT
19:40:51.731 T:547567643216  NOTICE:         m_streamTypes     : No passthrough capabilities
19:40:51.731 T:547567643216  NOTICE:     Device 2
19:40:51.731 T:547567643216  NOTICE:         m_deviceName      : alsa_output.platform-soc_audio.analog-stereo
19:40:51.731 T:547567643216  NOTICE:         m_displayName     : bcm2835 ALSA Analog Stereo
19:40:51.731 T:547567643216  NOTICE:         m_displayNameExtra: Analog Output (PULSEAUDIO)
19:40:51.731 T:547567643216  NOTICE:         m_deviceType      : AE_DEVTYPE_PCM
19:40:51.731 T:547567643216  NOTICE:         m_channels        : FL,FR
19:40:51.731 T:547567643216  NOTICE:         m_sampleRates     : 5512,8000,11025,16000,22050,32000,44100,48000,64000,88200,96000,176400,192000,384000
19:40:51.731 T:547567643216  NOTICE:         m_dataFormats     : AE_FMT_U8,AE_FMT_S16NE,AE_FMT_S24NE3,AE_FMT_S24NE4,AE_FMT_S32NE,AE_FMT_FLOAT
19:40:51.731 T:547567643216  NOTICE:         m_streamTypes     : No passthrough capabilities
19:40:51.744 T:547567643216  NOTICE: No settings file to load (special://xbmc/system/advancedsettings.xml)
19:40:51.744 T:547567643216  NOTICE: No settings file to load (special://masterprofile/advancedsettings.xml)
19:40:51.744 T:547567643216  NOTICE: Default Video Player: VideoPlayer
19:40:51.744 T:547567643216  NOTICE: Default Audio Player: paplayer
19:40:51.744 T:547567643216  NOTICE: Disabled debug logging due to GUI setting. Level 0.
19:40:51.744 T:547567643216  NOTICE: Log level changed to "LOG_LEVEL_NORMAL"
19:40:51.745 T:547567643216  NOTICE: Loading player core factory settings from special://xbmc/system/playercorefactory.xml.
19:40:51.747 T:547567643216  NOTICE: Loaded playercorefactory configuration
19:40:51.747 T:547567643216  NOTICE: Loading player core factory settings from special://masterprofile/playercorefactory.xml.
19:40:51.747 T:547567643216  NOTICE: special://masterprofile/playercorefactory.xml does not exist. Skipping.
10:40:51.862 T:547567643216  NOTICE: Running database version Addons27
10:40:52.197 T:547567643216  NOTICE: ADDONS: Using repository repository.xbmc.org
10:40:52.218 T:547474107824  NOTICE: PulseAudio: Opened device Default in pcm mode with Buffersize 150 ms
10:40:52.371 T:547567643216  NOTICE: Checking resolution 16
10:40:52.371 T:547567643216  NOTICE: CWinSystemX11GLESContext::CreateNewWindow
10:40:52.372 T:547567643216  NOTICE: CWinSystemX11GLESContext::GetVisual() m_pGLContext:(nil) GetVisual
10:40:52.372 T:547567643216  NOTICE: Create new CGLContextEGL at CWinSystemX11GLESContext::CreateNewWindow, m_dpy=0x55b304b780
10:40:56.522 T:547567643216   ERROR: Failed to find matching visual
10:40:56.522 T:547567643216  NOTICE: GL_VENDOR = NULL
10:40:56.522 T:547567643216  NOTICE: GL_RENDERER = NULL
10:40:56.522 T:547567643216  NOTICE: GL_VERSION = NULL
10:40:56.522 T:547567643216  NOTICE: GL_SHADING_LANGUAGE_VERSION = NULL
10:40:56.522 T:547567643216  NOTICE: GL_EXTENSIONS = NULL
10:40:56.527 T:547567643216   ERROR: GL: Error compiling vertex shader
10:40:56.527 T:547567643216   ERROR: B
10:40:56.527 T:547567643216   ERROR: GL: Error compiling vertex shader
10:40:56.527 T:547567643216   ERROR: GUI Shader [guishader_frag_default.glsl] - Initialise failed
10:40:56.528 T:547567643216   ERROR: GL: Error compiling vertex shader
10:40:56.528 T:547567643216   ERROR: B
10:40:56.528 T:547567643216   ERROR: GL: Error compiling vertex shader
10:40:56.528 T:547567643216   ERROR: GUI Shader [guishader_frag_texture.glsl] - Initialise failed
10:40:56.529 T:547567643216   ERROR: GL: Error compiling vertex shader
10:40:56.529 T:547567643216   ERROR: B
10:40:56.529 T:547567643216   ERROR: GL: Error compiling vertex shader
10:40:56.530 T:547567643216   ERROR: GUI Shader [guishader_frag_multi.glsl] - Initialise failed
10:40:56.531 T:547567643216   ERROR: GL: Error compiling vertex shader
10:40:56.531 T:547567643216   ERROR: B
10:40:56.531 T:547567643216   ERROR: GL: Error compiling vertex shader
10:40:56.531 T:547567643216   ERROR: GUI Shader [guishader_frag_fonts.glsl] - Initialise failed
10:40:56.532 T:547567643216   ERROR: GL: Error compiling vertex shader
10:40:56.532 T:547567643216   ERROR: B
10:40:56.532 T:547567643216   ERROR: GL: Error compiling vertex shader
10:40:56.532 T:547567643216   ERROR: GUI Shader [guishader_frag_texture_noblend.glsl] - Initialise failed
10:40:56.533 T:547567643216   ERROR: GL: Error compiling vertex shader
10:40:56.533 T:547567643216   ERROR: B
10:40:56.533 T:547567643216   ERROR: GL: Error compiling vertex shader
10:40:56.533 T:547567643216   ERROR: GUI Shader [guishader_frag_multi_blendcolor.glsl] - Initialise failed
10:40:56.534 T:547567643216   ERROR: GL: Error compiling vertex shader
10:40:56.535 T:547567643216   ERROR: B
10:40:56.535 T:547567643216   ERROR: GL: Error compiling vertex shader
10:40:56.535 T:547567643216   ERROR: GUI Shader [guishader_frag_rgba.glsl] - Initialise failed
10:40:56.536 T:547567643216   ERROR: GL: Error compiling vertex shader
10:40:56.536 T:547567643216   ERROR: B
10:40:56.536 T:547567643216   ERROR: GL: Error compiling vertex shader
10:40:56.536 T:547567643216   ERROR: GUI Shader [guishader_frag_rgba_blendcolor.glsl] - Initialise failed
10:40:56.537 T:547567643216   ERROR: GL: Error compiling vertex shader
10:40:56.537 T:547567643216   ERROR: B
10:40:56.537 T:547567643216   ERROR: GL: Error compiling vertex shader
10:40:56.537 T:547567643216   ERROR: GUI Shader [guishader_frag_rgba_bob.glsl] - Initialise failed
10:40:56.539 T:547567643216  NOTICE: CWinSystemX11GLESContext::GetVisual() m_pGLContext:0x55b301bf20 GetVisual
10:40:56.539 T:547567643216   ERROR: Failed to find matching visual
10:40:57.218 T:547567643216   ERROR: Invalid GUI Shader selected - [guishader_frag_texture_noblend.glsl]
10:40:57.218 T:547567643216   ERROR: GLES: Vertical Blank Syncing unsupported
10:40:57.774 T:547559226800  NOTICE: Running database version Addons27
10:40:57.785 T:547559226800  NOTICE: Running database version ViewModes6
10:40:57.795 T:547559226800  NOTICE: Running database version Textures13
10:40:57.807 T:547559226800  NOTICE: Running database version MyMusic60
10:40:57.835 T:547559226800  NOTICE: Running database version MyVideos107
10:40:57.842 T:547559226800  NOTICE: Running database version TV29
10:40:57.849 T:547559226800  NOTICE: Running database version Epg11
10:40:57.856 T:547567643216   ERROR: Invalid GUI Shader selected - [guishader_frag_texture_noblend.glsl]
10:40:57.856 T:547567643216  NOTICE: start dvd mediatype detection
10:40:57.918 T:547567643216   ERROR: Invalid GUI Shader selected - [guishader_frag_texture_noblend.glsl]
10:40:57.921 T:547567643216 WARNING: CSkinInfo: failed to load skin settings
10:40:58.071 T:547567643216   ERROR: GetCharacter: Unable to cache character (out of memory?)
10:40:58.587 T:547567643216 WARNING: JSONRPC: Could not parse type "Setting.Details.SettingList"
10:40:58.831 T:547567643216  NOTICE: initialize done
10:40:58.831 T:547567643216  NOTICE: Running the application...
10:40:58.850 T:547567643216  NOTICE: starting zeroconf publishing
10:40:58.850 T:547567643216  NOTICE: starting upnp client
10:40:58.873 T:546131930544  NOTICE: ES: Starting UDP Event server on port 9777
10:40:58.873 T:546131930544  NOTICE: UDP: Listening on port 9777 (ipv6 : false)
10:40:59.317 T:547567643216 WARNING: ReallocTexture: allocated new texture with height of 85, requested 128
10:40:59.331 T:547567643216 WARNING: Previous line repeats 4 times.
10:40:59.331 T:547567643216   ERROR: Invalid GUI Shader selected - [guishader_frag_texture.glsl]
10:40:59.332 T:547567643216   ERROR: Previous line repeats 1 times.
10:40:59.332 T:547567643216   ERROR: Invalid GUI Shader selected - [guishader_frag_fonts.glsl]
10:40:59.348 T:547567643216   ERROR: Previous line repeats 239 times.
10:40:59.349 T:547567643216   ERROR: Invalid GUI Shader selected - [guishader_frag_texture.glsl]
10:40:59.349 T:547567643216   ERROR: Invalid GUI Shader selected - [guishader_frag_fonts.glsl]
10:40:59.349 T:547567643216   ERROR: Previous line repeats 10 times.
10:40:59.349 T:547567643216   ERROR: Invalid GUI Shader selected - [guishader_frag_texture.glsl]
10:40:59.349 T:547567643216   ERROR: Invalid GUI Shader selected - [guishader_frag_fonts.glsl]
10:40:59.350 T:547567643216   ERROR: Previous line repeats 10 times.
10:40:59.350 T:547567643216   ERROR: Invalid GUI Shader selected - [guishader_frag_texture.glsl]
10:40:59.350 T:547567643216   ERROR: Previous line repeats 2 times.
10:40:59.350 T:547567643216   ERROR: Invalid GUI Shader selected - [guishader_frag_fonts.glsl]
10:40:59.351 T:547567643216   ERROR: Previous line repeats 11 times.
.
.
.
.

searched but cant fine the logs for kodi32

[sakaki]

First go at packaging up the kernel:
https://www.dropbox.com/s/ngjj4uo82wojl ... l.tar?dl=1

Pretty much identical to raspberrypi-kernel.

Thanks, looks good! I forgot Debian has diversions, don't get that luxury in Gentoo >< Nice way to handle the clashes with raspberrypi-kernel (which presumably would then stay installed)?

One question: shouldn't the control file specify arm64 architecture, not armhf?

Best, sakaki

PS which package owns the /boot/overlay/*.dtbo files on the default distribution image?

[ShiftPlusOne]

Thanks, looks good! I forgot Debian has diversions, don't get that luxury in Gentoo >< Nice way to handle the clashes with raspberrypi-kernel (which presumably would then stay installed)?

One question: shouldn't the control file specify arm64 architecture, not armhf?

Best, sakaki

PS which package owns the /boot/overlay/*.dtbo files on the default distribution image?

In this case diversions are actually a hacky workaround for another Debian issue. dpkg tries to chown files and fails if it's installing them to a fat32 partition, erroring out the install. Temporarily diverting the files during install works around that. I suppose it could be used to divert the clashing dtb files as well, so maybe that's something to look into to avoid the dtbo issue (addressed later on)

You'd set the architecture to arm64 if you wanted it to install in an arm64 userland, which isn't the case here. If it was set to arm64, it wouldn't build for, or get added into an armhf apt repo.

dtbo files get installed by the raspberrypi-kernel package (since they are built from the kernel source tree), which gets removed if you install this package, so the overlays go missing. Either they should be included in your pre-built tarball or the package should be installable alongside raspberrypi-kernel.

In the meanwhile, I've set up a vps to host a repo for all of this and automatically update everything when it's updated on github. More info on that after namecheap gets around to updating the DNS records.

[sakaki]

You'd set the architecture to arm64 if you wanted it to install in an arm64 userland, which isn't the case here. If it was set to arm64, it wouldn't build for, or get added into an armhf apt repo.

Ah OK, makes sense. Nicely illustrates why I'm not the right person to do Debian packaging ^-^

dtbo files get installed by the raspberrypi-kernel package (since they are built from the kernel source tree), which gets removed if you install this package, so the overlays go missing. Either they should be included in your pre-built tarball or the package should be installable alongside raspberrypi-kernel.

There's an argument for going either way I guess. However, the 'cleaner' way to do it would arguably be to add the dtbos to my kernel autobuild tarballs (for releases going forward) and then your package can fully replace raspberrypi-kernel, rather than being dependent upon it. Shouldn't be difficult to do; I'll look to address that point over the next week or so.
Best,
sakaki

[sakaki]

I've already got the 32bit and 64bit of my fave programs installed (thats why it took a long time to "install")

So you want me to test:

Clean boot32
<snip>
shutdown

Clean boot64
<snip>
shutdown

report results to this forum

if the above sequence is OK then I'll make time.

Yes, if you could, that'd be great - many thanks!

Also noted while writing halfway to this post I installed 64bit htop. System froze during the "processing triggers"
regained control after about a minute (give or take 10 secs).

Possibly your system was swapping at that point? Incidentally, I will probably re-work the icon-copying lines of the systemd-path-unit-triggered reflect-apps script; it currently uses a rather heavyweight way of first copying the icon tree inside the container, and then copying it out (to workaround packages like firefox-esr, which use root-relative symlinks outside of the gdm search path for certain icons). This could more quickly be done with a few rewrite rules and a direct host-filesystem copy; as this script gets triggered whenever the container's /usr/share/applications directory is updated during a new install, it can sometimes cause a little bit of "stutter" (but I've never seen 30secs delay).

Best, sakaki

[sakaki]

ShiftPlusOne,

I've updated the kernel autobuild script, and manually triggered it to check. So, from this release, there is another top-level folder overlays in the tarball, which contains the dtbo files (and the README). I've also added the bcm2710-rpi-cm3.dtb file to /boot.

Best,
sakaki

PS I chose not to make the overlays folder a subfolder of /boot, to avoid issues on the Gentoo side (where another package owns the dtbos).

Edited to reflect changed decision on this point: please see post below.

[sakaki]

ShiftPlusOne,

I've updated the kernel autobuild script, and manually triggered it to check. So, from this release, there is another top-level folder overlays in the tarball, which contains the dtbo files (and the README). I've also added the bcm2710-rpi-cm3.dtb file to /boot.

Best,
sakaki

PS I chose not to make the overlays folder a subfolder of /boot, to avoid issues on the Gentoo side (where another package owns the dtbos).

Actually, I realized on reflection that the above was probably not such a great design choice. Accordingly, I've dealt with the path ownership clash problem in the affected Gentoo ebuilds, which is where the fix belongs, and have modified the autobuild (and the contents of the first affected tarball) so that "overlays" now is now just a subdirectory of /boot (which seems more natural).

I think this should mean your deb will then just do the right thing and copy these across on installation?

Best,
sakaki

[ShiftPlusOne]

There are a few lines I need to add back to handle the overlays.

[sakaki]

Hi ShiftPlusOne,

as the default kernel branch appears to have just been bumped to rpi-4.19.y (from rpi-4.14.y), I've updated my kernel autobuilds to track this. Hopefully this won't cause any issues for your binary kernel deb?

PS no rush on the remaining packaging, as I will be tied up for the next week or two landing the updated release of gentoo-on-rpi3-64bit, which will (fingers crossed ^-^) be out of test and into imaging run-up very shortly.

Once that's out of the way I'll be able to circle back and put out a new version of the raspbian-nspawn-64 image, based on your debs (and therefore, updatable going forward, hopefully).

Best, sakaki

[ShiftPlusOne]

as the default kernel branch appears to have just been bumped to rpi-4.19.y (from rpi-4.14.y), I've updated my kernel autobuilds to track this. Hopefully this won't cause any issues for your binary kernel deb?

Shouldn't be a problem at all.

I've spent a bit of time on this yesterday. I've added the overlays and have been setting up a workflow to automate everything. We can set up a release github webhook to trigger a deb builds or I can just set up a cronjob to check for new releases.

Also, any thoughts on adding these debs to the official archive.raspberrypi.org repo once they're stable?

[sakaki]

Shouldn't be a problem at all.

I've spent a bit of time on this yesterday. I've added the overlays and have been setting up a workflow to automate everything. We can set up a release github webhook to trigger a deb builds or I can just set up a cronjob to check for new releases.

Perfect, many thanks!

Also, any thoughts on adding these debs to the official archive.raspberrypi.org repo once they're stable?

Wow! Well obviously that'd be fantastic ^-^ I have had a few thoughts on issues that should be addressed prior to any 'production' roll out; I'll try and put them into some some semblance of coherence, and post again here shortly.

Best,

sakaki

[ShiftPlusOne]

Latest update seems to work:

Code: Select all

# Add my repo signing key
wget -O - https://repo.xecdesign.com/public.key | sudo apt-key add -
# Add the repo
cat << EOF | sudo tee /etc/apt/sources.list.d/xec.list
deb http://repo.xecdesign.com/ stretch main
#deb-src http://repo.xecdesign.com/ stretch main
EOF
# Install the package
sudo apt update
sudo apt install bcmrpi3-kernel-bis
reboot

Or, it can be downloaded manually from here https://repo.xecdesign.com/pool/main/b/ ... _armhf.deb

Edit: Source package uploaded here: https://github.com/XECDesign/bcmrpi3-ke ... ter/debian

[sakaki]

ShiftPlusOne,

OK, so I've just tried installing your 64-bit kernel deb on a copy of my raspbian-nspawn-64 image, and it seems to work very well, automatically replacing the raspberrypi-kernel as one would expect.

Further, with this 4.19.y, and the /proc/cpuinfo hack discussed earlier in this thread, picamera (and friends [1]) can be now run under a 64-bit kernel too (from a 32-bit userland, but this shouldn't be too hard to extend to 64-bit userland...). Screenshot:



This is an RPi3B+ (in a PiTop v1 chassis) with an RPi camera module v2 attached, running raspbian-nspawn-64 with your 64-bit kernel package. The two little Python programs (client and server) shown are trivially adapted from here [2]. Done this way, you get a real-time (-ish, the frame rate isn't all that great) view of the camera even in true kms mode (which I booted under for this example). raspivid, raspistill etc. work too, but if you want them to display a preview, you need to use fkms instead iirc (which is mostly fine, incidentally, but still has a few rough edges under 4.19 at the time of writing) [3].


Productionization (of the Binary Kernel Package)

So the next steps for this part of the project would be, as you suggest, to craft either a git hook or cronjob, to ensure that a matching version of your bcmrpi3-kernel-bis package gets created whenever a new kernel tarball is released by my autobuild (usually on a weekly cadence, although I am kicking things off slightly more regularly atm with the shift to 4.19). Once that is done, I don't see too many reasons why this package couldn't be made available in the main repo for those who want to try booting under a 64-bit kernel (there's no need for all the 64-bit userland container stuff etc. to be installed just to do that; someone could do it starting on a stock Raspbian system).

Aside: it's nice that using your deb, an end user can simply apt-get install a 64-bit kernel package (thereby also removing raspberrypi-kernel), reboot, try stuff out, and then revert by apt-get installing raspberrypi-kernel again. In fact, as jdonald has suggested, this process could even be simplified further, as an option in raspi-config:

...
This got buried among the "brain dumps" but a few posts back I pushed for the idea of treating this as initially a capability mode rather than a new image. That is, include kernel8.img with Raspbian so people can boot into 64-bit mode. Following a similar pattern as the OpenGL driver, ideally there would be a flag in raspi-config to enable/disable it, with the default being off.

(To be clear, I'm only suggesting providing the kernel. Users who desire Eric's GL driver, gcc-aarch64-linux-gnu, or anything else from 64-bit userland don't get that preinstalled, but at least have a starting point to work with if they choose.)
...

As an alternative to his suggestion (of shipping Raspbian with kernel8.img (and module set) preinstalled), raspi-config (or the gui config tool) could simply kick off an apt-get install of bcmrpi3-kernel-bis should the "64-bit kernel" option be chosen, and apt-get install raspberrypi-kernel again whenever the option was unchecked.

I intend to keep the automated kernel builds running for the forseeable future (as the resulting tarballs are used in my own gentoo-on-rpi3-64bit image, and there is at least one downstream AUR consumer that I know of using the 'vanilla' variant), so the only remaining 'productionization' issues therefore would seem to be:

1. I assume you'd want some ability to have 'curated' versions released which have had at least some baseline testing? That's the approach I take with gentoo-on-rpi3-64bit image: masking the automatically created gentoo ebuilds and only unmasking a few tested variants. Not sure how this would best be arranged in Debian.
2. It would possibly make sense to also have a auto-created deb (patterned after your existing one) following bcmrpi3-kernel, as this uses an untweaked bcmrpi3_defconfig, which some may prefer (for testing, to keep things clean etc.).
3. If a version of the bcmrpi3-kernel-bis deb goes into the main repo, is there a way to avoid e.g. someone on an RPi2v1 installing it by accident and thereby ending up with an unbootable system (as the 32-bit kernel will be removed)?
4. In the limit, I assume the RPF would probably want to host their own 64-bit kernel autobuilds if this facility were to prove popular (for end-user assurance etc.) I have no problem with things migrating that way in time should policy dictate, of course.
5. Others?

Productionization (of the Userland Packages)

There are a number of points regarding development of the other two, userland packages (64-bit container image and 32-bit container support) that I've been thinking about a bit:

1. At the moment the container name is 'hardwired' to debian-stretch-64. It'd be nice to at least indirect this in the 32-bit support package scripts (ds64-run etc.) so that things don't break when e.g. buster rolls out etc. Shouldn't be too hard to do (will need to rename that 'ds64-' prefix used everywhere too, or at least find a backronym for it ^-^).
2. Extending this idea, do we want to support multiple containers at the same time? A mix of 64-bit and 32-bit containers at the same time? Doing so would have its uses, but on the downside a single-container-only (from the 'managed' perspective, nothing prevents users creating their own using standard tools) system has the advantage of simplicity. To go multi-container would probably require a management app (to start, stop etc.), maybe menu hierarchy (for launching container shells etc), and might get to feel quite 'fiddly'. There are arguments both ways (for example, on the plus side, having multiple installed containers is very useful for testing, and it opens up a wide range of different repos in which to find packages, and so on).
3. The 'reflector' script for menu items currently has a race condition on startup with the container boot (I've only seen it 'lose' once in about 50 times, but still, it's a bug). Underlying issue is that the ds64-running script essentially just checks that the target has started booting, not that it is ready to accept e.g. container shell commands. Should be easy to fix.
4. On a similar note, the icon copying process in the reflector is a bit clunky - it first copies the gdm icon tree locally within the container, resolving symlinks, then copies the result out. Again, simple to resolve - in fact, the whole operation can simply be done 'from the outside' with no shell invocation necessary.
5. As to the 64-bit container image itself, that should be made as vanilla as possible. Fortunately, there's very little that needs to be done relative to a debootstrapped starting point, since e.g. the reflect-passwd script will sort out setting up the "pi" user and baseline groups. I've moved a few other things 'to the outside' on my working copy (for example, dealing with the "passwordless sudo" requirement by just bind-mounting an entry into /etc/sudoers.d when the container is booted).
6. So... perhaps the container image deb should, in fact, just kick off a debootstrap, and then do a few post-install steps where necessary. Doing it this way would ensure users started always with an up-to-date set of packages in the container. On the other hand, an prebuilt image is more straightforward, and the process of creating that can always be automated. I guess we should just stick with the 'untar an image into /var/lib/machines/...' approach (as originally discussed) and take it from there.
7. Even if these images debs are not simultaneously installable, we'd still need to put them in a taxonomy able to handle e.g. stretch / buster type transitions (and possibly, other distros too).

Incidentally, I appreciate there may not be the same appetite to migrate the 64-bit userland elements into the official repo as the kernel. I hope something can be done (over time) though - as I've been playing with it over the last few weeks, the approach does genuinely seem to be a very low-touch way to support some mixing of 64-bit userland apps with standard Raspbian.

Best,

sakaki

[1] See e.g. 6by9's post here.
[2] As noted there, you can also do this with raspivid and nc, but it's nice to see picamera working!
[3] As the "display server" part does not itself use picamera, and the image uses a common network namespace, you could even run it inside the container, with the "stream sender" running outside, as a crude way of allow the local camera to be used from 64-bit userland.

[ShiftPlusOne]

Great to hear there are no major issues.

Since this isn't something I'm doing for work, adding anything to raspi-config isn't an option. Same goes for adding another 64 bit kernel. It starts to get complicated and overlap with work other people are doing.

If we do start to ship a stock 64bit kernel it would be along with the existing kernel.img and kernel7.img built as a part of the existing process. I'll leave all of that to the people currently handling kernel builds.

If a version of the bcmrpi3-kernel-bis deb goes into the main repo, is there a way to avoid e.g. someone on an RPi2v1 installing it by accident and thereby ending up with an unbootable system (as the 32-bit kernel will be removed)?

I think it's possible to notify the user they've made a mistake and tell them how to fix it, but I'm not aware of a mechanism to prevent it from happening.

I assume you'd want some ability to have 'curated' versions released which have had at least some baseline testing? That's the approach I take with gentoo-on-rpi3-64bit image: masking the automatically created gentoo ebuilds and only unmasking a few tested variants. Not sure how this would best be arranged in Debian.

Instead of uploading to 'main', we could add a 'testing' component and then move packages across when they're ready.

I don't really know how that would work in practice. How would we know which releases are ready to be marked stable? Would anybody use the untested packages?

As far as the userland packages go, I don't have too many thoughts on that.

Bootstrapping takes a long time, so I think it should happen when the package is built, not on install. It shouldn't be out of date if it's generated often enough.

I'm not really sure how to handle upgrades. I think we have to leave it up to the user or maybe provide helper commands. Maybe something to allow them to reset the container to the latest version, losing their changes or dist-upgrade the existing container.

Making things generic and allowing for multiple containers is definitely a good idea. Overall, it just adds flexibility for some of the stranger things people want to do - run 64bit applications, i386 applications under wine, buster versions of applications and so on.

Having said all that, those are just my thoughts. It's your project and I'm happy to do with whatever you think works best.

[sakaki]

Of course, no worries ^-^

I just got the wrong end of the stick I think, as you had said:

Also, any thoughts on adding these debs to the official archive.raspberrypi.org repo once they're stable?

so I assumed you were looking to upstream stuff at some point.

Best, sakaki

[ShiftPlusOne]

Of course, no worries ^-^

I just got the wrong end of the stick I think, as you had said:

Also, any thoughts on adding these debs to the official archive.raspberrypi.org repo once they're stable?

so I assumed you were looking to upstream stuff at some point.

Best, sakaki

Sorry I was a bit vague there.

I've got permission to add your kernel and userland packages to the official repo, so yeah there's no problem with that. If you're okay with it, that's the plan.

If I try to push it further and integrate it into raspi-config or other packages it starts to involve more people.

[sakaki]

Sorry I was a bit vague there.

I've got permission to add your kernel and userland packages to the official repo, so yeah there's no problem with that. If you're okay with it, that's the plan.

If I try to push it further and integrate it into raspi-config or other packages it starts to involve more people.

No problem, getting a version of this into the official repo is a fantastic opportunity in any event!

And a significant responsibility. On reflection, I think you're probably right that this needs to support multiple guest images, so the core design issue to be solved for the first 'real' release is how best to 'un-hardcode' to achieve that, allowing users to add/publish their own OS images for use with it, without getting sucked into something so complex people may as well use docker ^-^

I'm keen that the design process for this is a) time bounded and b) happens in the open, since the basis for this project so far has really been ideas from others in the community, rather than something I've come up with myself, and all the better for that.

So, how does the following rough schedule sound?

Next few days: I'll publish some high-level project goals / scope / principles for review in this thread; comments and suggestions from all actively invited. Next couple of weeks these can then be honed (and I can get the next gentoo-on-rpi3-64bit release out of the way, to free up some bandwidth).

Fri Mar 8: Goal review period ends. Hopefully also we'd be in a position where the current three demo deb packages were available as a baseline 'show me':

• The 64-bit kernel package - which seems to be in a pretty good shape right now and probably won't have to change that much - hopefully by this point getting auto-generated as required, via cron/hook?
• The debian-stretch-64 userland image package is just an untar, so hopefully not too complex to package (obviously, dealing with this properly for multiple images, versioning of images etc. will be something to deal with for the actual end system, but not required for the demo).
• And for the raspbian userland support package, I could provide a script to install to $DESTDIR or whatever, basis the GitHub source release tarball, if that would help.

Just to reiterate, I realize you're doing this in your own time, and so the demo packaging might not be possible by Fri 8. Absolutely no problem if so! I really appreciate the effort you're putting into this - thank you. Also, if you need to send any out-of-band comments, just ping me an email: [email protected].

Fri Mar 22: I'll post a more fully written up design spec to this thread for review. Comments and suggestions actively invited. The spec will primarily be a concrete proposal walking through how the current, functioning approach will be generalized to work with multiple OS images, potentially simultaneously, what it would take to add an image to the system, guest package installation / removal and effects on the Raspbian desktop menu, how available images would be 'published' (Gentoo overlay-style perhaps?), updated, and withdrawn, what one would need to do to a baseline OS image to prepare it for use (hopefully, not much!) etc.

Fri Apr 5: Spec review period ends (initial spec locked off).

Fri Apr 26: I'll try to put out an image (with any required supporting infrastructure) implementing the target design. This (apart from the kernel) will probably not be deb-packaged, but will (as the current raspbian-nspawn-64 image is) be a concrete, runnable system. Comments invited.

Fri May 31: Image review period ends. On the basis of 'plan to throw one away, you will anyhow', I'll then put together a tweaked version of the image, incorporating comments.

Fri Jun 14: Tweaked image version released for final review.

Fri Jun 28: Updated image review period ends. Initial packaging for upstreaming begins.

Fri Jul 26: First cut upstream packaging complete - alpha release for testing and review. At least one demo 32-bit and 64-bit container image available.

Fri Aug 30: Formal beta. With hopefully a few more, community contributed OS container images available.

Fri Sep 27: Lock off "1.0" packages and release to live tree.

I appreciate this is a bit finger-in-the-air, particularly towards the end, but does that sort of schedule seem realistic to you? Clearly, the final decision as to whether the components were stable enough to upstream would rest with you. And real-world constraints may intervene (work-wise, May is looking particularly horrible for me atm). Plus there may be some argument to release the 64-bit kernel package earlier than the others. TBD. But roughly-roughly sane?

Best,

sakaki