The start of raspbian (Debian Hard Float (armhf) for RPi)

[mpthompson]

Note to readers:

This thread follows the founding and early history of what became raspbian (though the name raspbian wasn't used until some time after the project began). Most early discussions happened here (though some happened on the debian-arm mailing list). The thread was originally posted in the "projects and collaboration general" forum and has moved forums a couple of times as the raspberry Pi guys reorganised and expanded their forums.

Some even earlier discussions (before the project really started) can be found in viewtopic.php?f=63&t=3641&start=25

I have now locked this topic because having new posts in it when we have a whole forum to use does doesn't serve to do anything except confuse.

-- plugwash

----------------------------------------

I'm creating a new topic in this forum in the hopes of getting some help around the project of creating a flavor of Debian armhf that is tuned to the specifics of the RPi.  I'm hoping that by doing some of the early work I can help kick off a broader effort to make Debian a first class, no compromise, Linux distribution for the RPi.

Right now, my immediate goal is to not create a full Debian port for the Raspberry Pi, but rather an experimental build of armhf tuned to the RPi that would have enough of the system in place that tests can be run to determine if a full blown porting effort would be worthwhile.  This would involve re-compiling enough of the armhf packages for ARMv6+VFPv2 to create a minimal Debian install using debootstrap, the build-essential packages and perhaps some other interesting packages that might demonstrate the performance benefits of hardfloat compilation.

As of now, I've purchased a Freescale iMX53 Quick Start Board with a ARM Cortex-A8 1 GHz processor and 1GB of RAM and went through the somewhat arduous task of getting u-boot, a working Linux Kernel and a basic install of Debian Wheezy armhf distribution running on it.  It is my intention to use this platform to develop the early RPi tuned Debian packages that I can share with others.

My next steps are to recompile the build-essential packages to, by default, output RPi friendly ARMv6+VFPv2 binaries rather than ARMv7-A+Thumb-2+VFPv3D16 binaries and then use these packages to recompile the 250, or so, packages required to create a minimal install of Debian.  I believe (and someone please confirm this if possible) that the ARMv6+VFPv2 packages I create should be fully compatible with the standard armhf packages, meaning I can mix them on a system which should help the porting process.  This is because they share the same ABI and the ARMv7 processor should run ARMv6 binaries fine (the passing of floating point values in the VFP registers is where some issue may crop up).

Once I get the 250 or so packages recompiled for a minimal install of Debian armhf for RPi, I should then be able to test these out by creating a minimal install for my iMX53 Quick Start Board.  By then, hopefully, I'll have actual RPi hardware in hand and use the same packages to create a Debian armhf install on the Raspberry Pi.  Obviously, such an install will be severely limited due to a lack of all the 10,000 other Debian ports being available, but it should at least be enough to run performance experiments against and have enough available to port any other Debian package to the RPi with hard float enabled.

The other route I could have gone is to stay with the armel ABI where floating point values are passed in integer registers and recompile key packages with hardware floating point enabled.  The advantage this has is that the 10,000s of existing armel packages for Debian could be used without change and the compilation process is narrowed down to just a few hundred packages that could really benefit from hard float compilation.  However, to me, this seems to be just a half step forward for RPi Debian users and the RPi would benefit from having a Linux distribution that takes maximum advantage of the computing resources of the platform.

I could use help in this effort by people more familiar with the Debian package build system giving me advice as to the most efficient way of getting the packages built.  First, beginning with the build-essential compilers and making sure I'm selecting the right options and configuring things so that the resulting binaries would indeed run on the Raspberry Pi.  Then, if there are other people who have iMX53 boards, perhaps sharing the task of building the initial set of packages needed for the minimal install of Debian.

What do other Debian fans think of this approach?  Comments or suggestions?

[Toad King]

I'm unsure if any hardfloat builds of the binary-only RPi components (OpenGL/OnenMAX/etc.) will be provided, so you may end up with a crippled system unless you go the softfp route. It's technically less optimized, but you get huge compatibility gains.

[mpthompson]

I was told in another thread that there were binaries compiled for both hard and soft float ABIs, but I'll definitely have to check to verify this on my own.  Thanks for the reminder.

[dom]

https://github.com/raspberrypi/firmware/tree/master/hardfp

has what you need.

[kasperl]

I wonder how much use you could get out of the multi-arch system, allowing some of the armel packages to be used for stuff that isn't done yet. It would make a fully-featured system available a lot quicker, and it might be easier for some packages that don't want to recompile. It would of course be a pain, especially at first, to get enough hf packages to make it worthwhile, but at least things would work in the meantime.

[mpthompson]

I don't know much about multiarch, but it seems like it would potentially be very interesting to run both armel and RPi flavored armhf packages on the same system.  Also, it's my understanding that multiarch is still a work in progress, but it is advanced enough to be workable under Wheezy.

In any case, it seems that my efforts as described would still dovetail nicely with a multiarch approach as it would build floating point packages that would be installed on a multiarch configured Debian system.

[mpthompson]

dom said:

https://github.com/raspberrypi/firmware/tree/master/hardfp

has what you need.

Hmmm. I checked those out and it appears the code is compiled using floating point instructions, but they are specifying the soft float ABI which allows them to be linked with armel code.  Found in the files are the relevant options the code seems to be compiled with:

------

-marm -mabi=aapcs-linux -mno-thumb-interwork -march=armv6 -mtune=arm1136j-s -msoft-float

------

It makes sense given their isn't yet a Linux for the RPi with hard float ABI to compile against.  At least, I don't believe there is.

However, this means these libraries would not work compiled against application code which specified the hard ABI.

I guess part of the hope here would be that when a hard float version of Linux is available, these libraries could be recompiled on that system by whomever created them in the first place.  This does seem to put a damper on things that these libraries wouldn't be immediately available.

[shirro]

Using my cross compiler toolchain's read-elf -A and TAG_VFP_ARGS is different between the hardfp and the normal libs. And objdump -S shows lots more happening with fp registers. I would have guessed it used the hardfloat abi.

Where did you find the compile options?

[mpthompson]

shirro said:

Where did you find the compile options?

If you dig down through the source tree to the sample application it looks like the compile options are included.  At least that is my interpretation.  An example is below:

https://github.com/raspberrypi/firmware/blob/master/hardfp/opt/vc/src/hello_pi/libs/interface/vchiq_arm/.vchiq_arm.o.cmd

Look at the first line which is several hundred characters long.

I don't doubt that the code is compiled with floating point hardware instructions, it just looks like they are using the soft floating point ABI the '-mabi=aapcs-linux' flag being the same specified for the Debian armel (softfp) ABI.  The difference is that when functions are called, the floating point values are marshaled via integer registers rather than passing the values in the floating point registers which is more efficient.

This makes sense as I don't think there is any hard float Linux distributions available for the RPi hardware, unless the Broadcom created one for internal use.

[shirro]

Yes, libilclient looks like softfloat abi to me. When I try and recompile the audio and video demos they fail to link because ilclient does not use VFP register arguments. The other demos don't need ilclient so I can change the Makefile to build them and no probs.

Perhaps this was just an oversight. Only the audio and video demo need ilclient so I guess it is just some higher level helper lib for OpenMax IL. Possibly nothing else out there needs it.

[EDIT: yep finally reading the include. ilclient is definately a helper lib to make OpenMax IL less painful. I will file a bug in github]

[shirro]

There are a few other libs that don't appear to be hardfloat in hardfp/opt/vc/lib

Will try and find out what they do and if they are important.

The ones that look suspect are libVG.so, libvcos.so, libvcos_generic.so, libvcos_static.a (libvcos.a looks ok)

[Not sure but I think libVG is probably just a misnamed libOpenVG and libvcos looks like the sort of thing that is probably statically linked into the broadcom code so will not cause any issues. I think some old stuff just got left in there when they built for hardfp.]

[mpthompson]

OK, it took me 24 hours, but I've got a rebuilt version of gcc-4.6 running under Debian armhf that, by default, produces hardfloat ARMv6+VFP binaries that theoretically will run on the Raspberry Pi.   I did some quick and dirty benchmarking with floating point calls on my iMX53 Quick Start Board and the code executes nearly as fast as the ARMv7+VFPv3-D16+Thumb2 code normally produced for armhf.  Also, these RPi flavored binaries seem to co-mingle with regular armhf packages fine so that should help make things easier.

Some things I'm looking at now include:

1. Create a list of the 250 or so packages that go into a minimal Debian install along with the additional build-essential packages.

2. Create a build process to rebuild the packages from the armhf source code packages so that they are all suitable for installing on the RPi.  Then store the packages in a local repository.  Potential to waste a lot of time here if it isn't done right.

3. Figure out some way of identifying whether any ARMv7 code has leaked through steps one and two -- that would prevent the package from running on the RPi.  Ideas on how to accomplish this would be terrific.

4. Share with the world a hard float version of Debian for the RPi and hope that there is a enough interest in the Debian and RPi communities to create a proper Debian port so the vast majority of all the 10,000s of Debian packages come available.

5. Figure out a way of getting the Broadcom or the RPi foundation to recompile hard float versions of the RPi proprietary binaries so they use the armhf hard float ABI.

As always, comments and insight are appreciated.  To a large extent, I'm doing this for fun and personal inquisitiveness, but hopefully it can help spark a larger effort around Debian for the RPi.

[shirro]

mpthompson said:

1. Create a list of the 250 or so packages that go into a minimal Debian install along with the additional build-essential packages.

Perhaps do a Debian install in a chroot and include every package you think anyone might want, then do a dpkg --get-slections as the basis for your packages to build. I guess the the debian distro build proceses does something like apt-get build-dep behind the scenes. If you were writing your own script to build the packages I would probably just iterate the get-selections list, get the dependencies, make the packages etc. Yeah, I clearly haven't build a Debian distro before.

2. Create a build process to rebuild the packages from the armhf source code packages so that they are all suitable for installing on the RPi.  Then store the packages in a local repository.  Potential to waste a lot of time here if it isn't done right.

Yes, I would have a local repo, and a public one for me . Makes it easy to add new versions of packages in and update your Pi and have people test before shipping out another image and it will all be so damn organised.

3. Figure out some way of identifying whether any ARMv7 code has leaked through steps one and two -- that would prevent the package from running on the RPi.  Ideas on how to accomplish this would be terrific.

The only way I can think of doing this would be to disassemble the output and search for ARMv7 instructions. Thumb code would be a giveaway. When I get my Armv7 I will compile some packages for v7 and v6 and write some sort of diagnostic if it isn't already done somewhere.

4. Share with the world a hard float version of Debian for the RPi and hope that there is a enough interest in the Debian and RPi communities to create a proper Debian port so the vast majority of all the 10,000s of Debian packages come available.

Built it and they will come.

5. Figure out a way of getting the Broadcom or the RPi foundation to recompile hard float versions of the RPi proprietary binaries so they use the armhf hard float ABI.

It is only one helper library required by two of the demos. The rest of the libraries that count look ok. Will know for sure when we get hardware.

[dom]

I've had another go at building the hardfp libs.

libilclient.a wasn't being built by my script, and the other softfp libs are obsolete ones.

github is updated.

[darkcity]

http://www.raspberrypi.org/for.....ppy-on-arm

your work could be useful to Rpi version puppy linux.  great stuff ; -)

[kghunt]

As for a temprary repo you can host the files in a dropbox public directory. I have done this with ubuntu with great success. Its were I keep all my debs that are not in the repos. Then I just add that repo to my system and viola.

[mpthompson]

kghunt said:

As for a temprary repo you can host the files in a dropbox public directory. I have done this with ubuntu with great success. Its were I keep all my debs that are not in the repos. Then I just add that repo to my system and viola.

I'll look into dropbox as you suggested.   Hopefully I'll get to the point where I need to distribute the debs.

[mpthompson]

The plot thickens.

While thinking about the next steps for getting a Debian armhf build for the RPi, it occurred to me that I was thinking about the porting process a little too simplistic.  An important objective is to create a version of Debian armhf that fully utilizes hard float compilation, but is free of ARMv7 code.  If any ARMv7 code leaks through in a package, the binary simply won't run on the RPi.

Therefore, not only do I need to rebuild any packages directly used in a Debian minimum installation, but any other package those packages may be statically linked against -- which could be a LOT.   To understand the scope of this issue, I need to come up with some sort of dependency tree that will tell me what specific packages need to be rebuilt and in what order to avoid the ARMv7 contamination in packages we want for ARMv6.

I've found some tools that do create dependency trees for Debian packages, but I'm not sure yet if they are producing the information I need.  As this is pretty much an issue any Debian port will face when creating a port to new hardware, this must be a path that has been trodden down many times before.  I just need to find out how other people determined what minimal packages need to be rebuilt in such circumstances.  This is an area I really don't want to re-invent the wheel.

If someone has insight into how to determine the minimal number of build-dep packages that need to be create for a given set of packages, please let me know.  It would be a great help.

[shirro]

Was just thinking while kernels are compiling what would happen if you compiled a .o for arm1176, vfpu and linked it with an object compiled for armv7, vfp3, thumb2 etc. Warnings? Errors? Nope worked fine. Had a v7 main call a function in v6 and the other way round.

One thing of interest was that readelf reported the resulting bin as having arch of v7 either way. If I create a library and put each architecture in it then each is reported correctly.

So I reckon you could use find, readelf and grep to locate any v7 libraries or bins in a system fairly easily.

[mpthompson]

shirro said:

One thing of interest was that readelf reported the resulting bin as having arch of v7 either way. If I create a library and put each architecture in it then each is reported correctly.
So I reckon you could use find, readelf and grep to locate any v7 libraries or bins in a system fairly easily.

Very, very interesting observation.  I think I follow what you did (I'm a bit groggy from a little lack of sleep .  I'll try to reproduce it on my system, but I may come back with some questions.

BTW, I finally got a functioning Debian armel system in QEMU running under an emulated ARMv6 CPU.  It looks to emulate VFP, but I haven't tested it out yet.  The proc information indicates VFP:

$ cat /proc/cpuinfo
Processor       : ARMv6-compatible processor rev 3 (v6l)
BogoMIPS        : 347.34
Features        : swp half thumb fastmult vfp edsp java
CPU implementer : 0x41
CPU architecture: 6TEJ
CPU variant     : 0x1
CPU part        : 0xb36
CPU revision    : 3
Hardware        : ARM-Versatile PB
Revision        : 0000
Serial          : 0000000000000000

Once I get the RPi flavored version of the armhf packages compiled, I should be able to use them on this QEMU system with debootstrap to create a chroot environment under the ARMv6.  That is one step away from booting an RPi flavored version of armhf.

[mpthompson]

shirro said:

So I reckon you could use find, readelf and grep to locate any v7 libraries or bins in a system fairly easily.

You just made my weekend!!! This has the potential to make it relatively trivial to ferret out v7 code in libraries and applications. I did the following test:

Below, in my armhf v6 chroot environment I compiled the hello.c program using my rebuilt v6 gcc and linked it against the rebuilt v6 standard C libraries.  Readelf does indeed properly report the resulting binary as v6.

---------------------------------------------
$ cat hello.c

#include <stdio.h>

int main()
{
printf("Hello world.\n");

return 0;
}

$ gcc -c hello.c
$ gcc -o hello hello.o

$ readelf -A hello
Attribute Section: aeabi
File Attributes
Tag_CPU_name: "6"
Tag_CPU_arch: v6
Tag_ARM_ISA_use: Yes
Tag_THUMB_ISA_use: Thumb-1
Tag_FP_arch: VFPv2
Tag_ABI_PCS_wchar_t: 4
Tag_ABI_FP_denormal: Needed
Tag_ABI_FP_exceptions: Needed
Tag_ABI_FP_number_model: IEEE 754
Tag_ABI_align_needed: 8-byte
Tag_ABI_align_preserved: 8-byte, except leaf SP
Tag_ABI_enum_size: int
Tag_ABI_HardFP_use: SP and DP
Tag_ABI_VFP_args: VFP registers
Tag_DIV_use: Not allowed
$
---------------------------------------------

Below, I went outside my chroot environment and simply linked the same v6 hello.o file created above with the armhf v7 standard C library.  Indeed, the resulting binary is now flagged as being v7.

---------------------------------------------
$ gcc -o hello hello.o

$ readelf -A hello
Attribute Section: aeabi
File Attributes
Tag_CPU_name: "7-A"
Tag_CPU_arch: v7
Tag_CPU_arch_profile: Application
Tag_ARM_ISA_use: Yes
Tag_THUMB_ISA_use: Thumb-2
Tag_FP_arch: VFPv3-D16
Tag_ABI_PCS_wchar_t: 4
Tag_ABI_FP_denormal: Needed
Tag_ABI_FP_exceptions: Needed
Tag_ABI_FP_number_model: IEEE 754
Tag_ABI_align_needed: 8-byte
Tag_ABI_align_preserved: 8-byte, except leaf SP
Tag_ABI_enum_size: int
Tag_ABI_HardFP_use: SP and DP
Tag_ABI_VFP_args: VFP registers
Tag_DIV_use: Not allowed
$
---------------------------------------------

This is terrific.  This means I no longer have to take it on faith that I'm actually producing pure v6 binaries.  A quick test already shows many of the binaries I created as I recompiled packages have v7 code leaking into them.  Something I suspected.  Now, if I can automate tracking down the source of the v7 code from static linking, that will just be icing on the cake.  However, just this puts me miles ahead in the game.

OK, it's time for bed for me.

[mpthompson]

Just an update for the people interested in a hard float version of Debian for the Raspberry Pi.

I now have a Freescale iMX53 Quick Start Board configured as a build system and I've made pretty good progress in the last week rebuilding Debian armhf packages specifically for the ARMv6+VFP capabilities of the RPi.  So far I have just under 100 packages built with another 200 to 300 more to go before I'll have enough to try and bootstrap a minimal version of Debian Linux.

A major aid to my progress has been creating a dpkg-v7clean utility that scans packages I've created to make sure they are free of ARMv7+VFPv3-D16+Thumb2 code produced for standard Debian armhf systems.  Of course, I won't know for sure until I run the packages on an actual ARMv6 system (or, actually a QEMU emulated ARMv6 system until RPi hardware ships) that I succeeded, but I'm pretty sure that I'm doing things right.

Hopefully I'll pass that milestone in about two weeks or so, assuming I don't hit major difficulties attempting to create pure ARMv6 binaries of the remaining packages needed to create a minimal Debian system. Once there, I'll look to figure out an efficient way of sharing these packages with other people who would like to experiment with them.

After that, I'll start working an actual RPi image in anticipation of hardware shipping as well as identifying what additional packages I should look to be building beyond those needed for just a minimal Debian system.

Mike

[shirro]

Wow, that is awesome progress for one little board. Well done.

[mpthompson]

Well, just compiling gcc and libc gave me about half of the packages.   Unfortunately, only about a dozen of those are needed for the minimal install.

[mpthompson]

BTW, I also have a Mele A1000 on order to see if I can get a basic port of Debian armhf running on it.

http://www.cnx-software.com/20.....droid-stb/

It runs the AllWinner A10 (Cortext A8) SOC which seems to be the hottest chip coming out of China these days.  Being half the cost of the iMX53, a 1 GHz processor, a decent 512M RAM and a real SATA interface, I'm hoping these can be converted into inexpensive build machines if there is enough interest in pushing through a full Debian armhf port for the Raspberry Pi.   A number of these along with a few iMX53 systems for heavy lifting could probably make a nice build cluster for Debian ARM.