The start of raspbian (Debian Hard Float (armhf) for RPi)

[mpthompson]

Bluemerlin, the package libatomic-ops has a challenging package failure that can be looked at.  A log of the failure I got when building it can be seen here.

http://pastebin.com/k6eDcLgX

The description for this package is:

Libatomic-ops implements a large collection of operations, each one of which is a combination of an (optional) atomic memory operation, and a memory barrier. It also implements associated feature-test macros that determine whether a particular operation is available on the current target hardware (either directly or by synthesis). Libatomic-ops attempts to replace various existing files with similar goals, since they usually do not handle differences in memory barrier styles with sufficient generality.

The error is:

gcc -DHAVE_CONFIG_H -I.   -D_FORTIFY_SOURCE=2 -fPIC -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat -Wformat-security -Werror=format-security -Wall -DNDEBUG -c atomic_ops_stack.c
/tmp/ccSYadB5.s: Assembler messages:
/tmp/ccSYadB5.s:181: Error: selected processor does not support ARM mode `ldrexd r2,[r0]'
/tmp/ccSYadB5.s:192: Error: selected processor does not support ARM mode `strexd r3,r6,[r0]'
make[4]: *** [atomic_ops_stack.o] Error 1

It's unclear to me whether the compiler itself is generating bogus assembly code or there are inline assembly statements or something that is causing this error.

I believe the package may believe it's being compiled on an armv7 armhf system and we need to tweak it to use what assembly code that might be used on an armv5 armel system.  But this is just a wild guess on my part what a fix might be.

It would be REALLY useful if you could come up with whatever changes are needed to get this package working.

mpthompson said:

It"s unclear to me whether the compiler itself is generating bogus assembly code or there are inline assembly statements or something that is causing this error.

I believe the package may believe it"s being compiled on an armv7 armhf system and we need to tweak it to use what assembly code that might be used on an armv5 armel system.  But this is just a wild guess on my part what a fix might be.

It would be REALLY useful if you could come up with whatever changes are needed to get this package working.

Hi.

I have probably missed something but ldrex is a valid armv6 (arm11) instruction and the Pi *HAS* an armv6 processor. Why do you refer to armv5 armel (which indeed does NOT have ldrex). I always assumed your Raspbian gcc was generating armv6 instructions, but it appears to generate armv5?

Anyway, the assembler used for building the package at least thinks it is generating for armv5.

[mpthompson]

tomtor said:

mpthompson said:

Hi.
I have probably missed something but ldrex is a valid armv6 (arm11) instruction and the Pi *HAS* an armv6 processor. Why do you refer to armv5 armel (which indeed does NOT have ldrex). I always assumed your Raspbian gcc was generating armv6 instructions, but it appears to generate armv5?

Anyway, the assembler used for building the package at least thinks it is generating for armv5.

No, you aren't missing anything.  It's actually my ignorance of the ARM instruction set.  The Raspbian gcc is (or should be) generating armv6 code by default.  I was assuming that ldrex was an armv7 instruction, but that doesn't seem to be case.  In all honesty, I shouldn't have even been speculating.  Forget I mentioned armv5 or whatever.

The package does include assembly files, but the error seems to be coming from a C file.  Whatever the issue, hopefully it's not something that is too difficult to find and fix.  I've come across some packages that do indeed use command options to tell the compiler to output code for a specific processor which of course needed to be fixed to get the package to build properly.  Perhaps that's the case here.

mpthompson said:

tomtor said:

mpthompson said:

Hi.
I have probably missed something but ldrex is a valid armv6 (arm11) instruction and the Pi *HAS* an armv6 processor. Why do you refer to armv5 armel (which indeed does NOT have ldrex). I always assumed your Raspbian gcc was generating armv6 instructions, but it appears to generate armv5?

Anyway, the assembler used for building the package at least thinks it is generating for armv5.

No, you aren't missing anything.  It's actually my ignorance of the ARM instruction set.  The Raspbian gcc is (or should be) generating armv6 code by default.  I was assuming that ldrex was an armv7 instruction, but that doesn't seem to be case.  In all honesty, I shouldn't have even been speculating.  Forget I mentioned armv5 or whatever.

The package does include assembly files, but the error seems to be coming from a C file.  Whatever the issue, hopefully it's not something that is too difficult to find and fix.  I've come across some packages that do indeed use command options to tell the compiler to output code for a specific processor which of course needed to be fixed to get the package to build properly.  Perhaps that's the case here.

ldrex is an armv6k instruction not in the original armv6 set.

Is the build/qemu image armv6 or armv6k?

Possible fix, if I change the configure.ac file so that it appends the CFLAGS with -march=armv6k it builds without issue.

Presumably the qemu image needs to be changed?

[plugwash]

The gcc defaults are currently set to plain armv6 (with vfp and hardfloat).

Doing some googling it seems that arm architectures are even more of a mess than I thought and armv6k was an extension introduced somewhere in the middle of the arm11 line (so some arm11 cores have it others don't).

We could change the gcc defaults but it's a change i'm reluctant to make as it is EXTREMELY difficult to revert if we discover it was a mistake. Do we know for sure if the Pi supports armv6k? what about other similar devices like the olinuxunio?

I only mention this in passing, but the kernel is built for armv6j: because the kernel guys think that gcc works better that way. Could that could be affecting your builds if they rely on uname ?

[plugwash]

Bluemerlin said:

Ummmmm it built fine first time with no errors ????

Sorry I neglected to say this but it's important that you do an "arch only" build (dpkg-buildpackage -B), Due to a recent change in dpkg a lot of packages are failing when doing an "arch only" build. Fetchmail seems to be one of those cases.

[plugwash]

fetchmail is now dealt with

Package:libatomic-ops

Looking at the instruction set for the ARM1176JZF-S included in the Pi SOC

http://infocenter.arm.com/help.....l#Cacicaih

If definately has the ldrexd and strexd armv6k instructions that are being complained about.

Is it the tool chain not seeing the correct emulated processor or that the package does not treat what it finds correctly.

I'll keep digging.

[plugwash]

Bluemerlin said:

Is it the tool chain not seeing the correct emulated processor or that the package does not treat what it finds correctly.

gcc (when invoked normally) does not and must not look at the system it is running on. If it did we couldn't compile our packages on armv7 build systems. The default settings for the compiler (the compiler passes the settings it is using onto the assembler) are defined in the gcc packaging.

As I said before it is possible to change the defaults in the gcc packaging but I really don't want to do that without better knowlege of the implications (the Pi may be our initial target but i'd really like to be able to say we support any armv6 device with vfpv2) since it is a descision that once made would be extremely difficult to back out of.

plugwash said:

Bluemerlin said:

Is it the tool chain not seeing the correct emulated processor or that the package does not treat what it finds correctly.

gcc (when invoked normally) does not and must not look at the system it is running on. If it did we couldn"t compile our packages on armv7 build systems. The default settings for the compiler (the compiler passes the settings it is using onto the assembler) are defined in the gcc packaging.

As I said before it is possible to change the defaults in the gcc packaging but I really don"t want to do that without better knowlege of the implications (the Pi may be our initial target but i"d really like to be able to say we support any armv6 device with vfpv2) since it is a descision that once made would be extremely difficult to back out of.

# gcc -c atomic_ops_stack.c -mcpu=arm1176jzf-s -O2
# gcc -c atomic_ops_stack.c -march=armv6 -O2
/tmp/ccfvBJ97.s: Assembler messages:
/tmp/ccfvBJ97.s:79: Error: selected processor does not support `ldrexd r6,[r1]"
/tmp/ccfvBJ97.s:88: Error: selected processor does not support `strexd ip,sl,[r1
]"
#

So only specifying -march=armv6 causes a mismatch between compiler output and assembler. Compiler emits armv6k instruction and assembler only handles armv6 instructions?

Specifying -mcpu=arm1176jzf-s works ok. My suggestion for building a Raspbian debian would be the using the -mcpu=arm1176jzf-s option (or -march=armv6k ) as suggested earlier.

Hello Plugwash,

Apologies if I fail to understand the purpose of Raspbian but aren't we trying to produce a build of Debian that makes the absolute most of the Raspberry Pi hardware available? If we fail to implement a compiler argument for the Pi that gives better performance why do we work towards the hard float version at all?

Now I understand why you want compatibility across a number of devices and that is good goal to have. But I think we are on a Raspberry Pi forum looking at the Raspberry Pi hardware. I think we would just be spreading ourselves thinly if you look at other future devices that may or may not be supported by this work.

Saying that if using the newer compiler flag produces results practically identical in respect of performance to the old one then maybe it is worth looking at future portability and not only Pi performance as the primary goal.

[plugwash]

tomtor said:

So only specifying -march=armv6 causes a mismatch between compiler output and assembler. Compiler emits armv6k instruction and assembler only handles armv6 instructions?

Do you have any evidence that it is the complier generating those instructions and not some inline assembler in the file being compiled?

Specifying -mcpu=arm1176jzf-s works ok.

Right

My suggestion for building a Raspbian debian would be the using the -mcpu=arm1176jzf-s option (or -march=armv6k ) as suggested earlier.

I'm not nessacerally saying this is a bad idea. However I am being cautious for the following reason.

As a general rule changing compiler settings to require a higher CPU is easy because code built with the old settings is not harmful. Changing compiler settings to support a lower CPU is much harder because you need to make sure absoloutely everything is rebuilt.

Changes that are easy to make but extremely difficult to back out should not be taken lightly.

john.mills said:

Saying that if using the newer compiler flag produces results practically identical in respect of performance to the old one then maybe it is worth looking at future portability and not only Pi performance as the primary goal.

My suspiscion is that this flag will make a negligable difference to performance and I have no idea how bad it's implications for portability are at this stage.

I could not find embedded assembler in either the c sources or in
/usr/include which have those specific instructions so they are probably gcc built-in. I would have to check the gcc source.

Specifying the 1136 as a lower class ARM is also an option.
I wonder if any ARM 11 processor exists which does have CFO and not the atomic instructions.

CFO should be VFP. Typo due to phone intelligence

[mpthompson]

Just getting caught up on this thread this morning (Pacific Daylight Time).

Just to give a little history, I chose the -march=armv6 and -mfp=vfp about six or seven weeks ago when I made the first changes to gcc just to see if I could build some functioning packages.  At the time I gave no more thought to it other than being unsure if the other specific armv6 (ie. armv6j, armv6t2, etc...) would work or not on the Raspberry Pi.  Since then, I haven't had a specific reason to go back and re-examine whether the -march flag should be set to something else.

In general, I like the idea of the binaries we are building being useful beyond the Raspberry Pi if it comes with very little cost in terms of performance.  Part of the genesis of Raspbian was unhappiness that the folks who created the Debian armhf port decided to draw the line that cpus below armv7-a wouldn't be supported.  [Please note, this is not a criticism of that decision at all as I understand why they made that call they way they did a year ago.  It's just that it didn't do those of us who are interested in the Raspberry Pi any good.]  I usually find some flexibility to be pretty good and allows serendipity to sometimes take place.

To solve the specific problem with libatomic-ops, I would suggest we make a Raspberry Pi specific change to the package to specify the -march=armv6k flag to the compiler and move on.  There are still 11,000 other packages to be built.  I had to make a similar change to the klibc package as that maintainer specifically set the compiler to produce armv7 code and I needed to reset that flag to a value friendly to the Raspberry Pi.  We can always go back and rebuild these smaller number of packages if a better solution is found.

With regards to QEMU image, is there another option that should be set for the -cpu setting?  If so, let me know and I'll update the image file to reflect that change.

I'm a bit confused:

gcc source gcc-4.7.0/gcc/config/arm/sync.md:

(define_insn "arm_load_exclusivedi"
[(set (match_operand:DI 0 "s_register_operand" "=r")
(unspec_volatile:DI
[(match_operand:DI 1 "mem_noofs_operand" "Ua")]
VUNSPEC_LL))]
"TARGET_HAVE_LDREXD"
{
rtx target = operands[0];
/* The restrictions on target registers in ARM mode are that the two
registers are consecutive and the first one is even; Thumb is
actually more flexible, but DI should give us this anyway.
Note that the 1st register always gets the lowest word in memory.  */
gcc_assert ((REGNO (target) & 1) == 0);
operands[2] = gen_rtx_REG (SImode, REGNO (target) + 1);
return "ldrexd%?\t%0, %2, %C1";
}
[(set_attr "predicable" "yes")])

and also in gcc 4.7.0:

/* Nonzero if this chip supports ldrexd and strexd.  */
#define TARGET_HAVE_LDREXD      (((arm_arch6k && TARGET_ARM) || arm_arch7) \
&& arm_arch_notm)

So gcc 4.7 emits the instruction and the conditions look ok.

I compiled on my arm box with an older gcc version 4.4.5 (Debian 4.4.5-8) but could not

find the ldrexd instructions in the compiler source. Weird?

mpthompson said:

To solve the specific problem with libatomic-ops, I would suggest we make a Raspberry Pi specific change to the package to specify the -march=armv6k flag to the compiler and move on.  There are still 11,000 other packages to be built.  I had to make a similar change to the klibc package as that maintainer specifically set the compiler to produce armv7 code and I needed to reset that flag to a value friendly to the Raspberry Pi.  We can always go back and rebuild these smaller number of packages if a better solution is found.

Sounds good to me!

I cannot imagine that there would be a substantial (any?) performance advantage for the Pi by using other flags. I expect that compiling with gcc4.7.0 would also work with the original armv6 flag, but I'm a bit mystified that I cannot find the corresponding code in the gcc 4.4.5 sources.

Ah, problem solved. Assembly IS in the libatomic-ops sources!

./atomic_ops/sysdeps/gcc/arm.h

has the embedded assembly.

Condition:

#if defined(__ARM_ARCH_6__) || defined(__ARM_ARCH_6J__)
|| defined(__ARM_ARCH_6K__) || defined(__ARM_ARCH_6ZK__)
|| defined(__ARM_ARCH_7__) || defined(__ARM_ARCH_7A__)
|| defined(__ARM_ARCH_7M__) || defined(__ARM_ARCH_7R__)

should be

#if defined(__ARM_ARCH_6K__) || defined(__ARM_ARCH_6ZK__)
|| defined(__ARM_ARCH_7__) || defined(__ARM_ARCH_7A__)
|| defined(__ARM_ARCH_7M__) || defined(__ARM_ARCH_7R__)

Either patch this file or use -march=armv6k

I vote for the last option because the same problem may exist in other packages.

Bluemerlin's discovery of the differences between armv6 and armv6k are hard to find and easy overlooked.

[mpthompson]

Oh no!!! I went to go order 3 more Freescale iMX53 Quick Start Boards from Digikey to complete the build cluster and they are out of stock!!!  And, they have a 12 week factory lead time putting deliver in late June or July!!! I tried Mouser, Newark, Avnet, Future Electronics and a few other places and they are all out of stock as well with fairly long delivery dates (Mouser may have some next week). I can't find any Internet distributor that has them in stock on either side of the Atlantic.

Well, I guess we'll have to make do with our five server cluster.  I'm glad I got the order for four in two weeks ago as I had no idea these boards were that hard to come by.  I'm just sorry I didn't order all seven at the time to build out the cluster of eight servers.

Perhaps someone else may have better luck locating stock.  The part number of the development board I need is: MCIMX53-START-R  Or, you happen to know a source for boards that are no longer in use, I'll pay the retail price of $149 for them.

I have a Mele A1000 on hand that I was thinking of pressing into service if I could get a working kernel/armhf install on it.  It has half the memory of the iMX53 QSB, but that should be good enough for all but a handful of packages.

[mpthompson]

tomtor said:

Either patch this file or use -march=armv6k
I vote for the last option.

Terrific.  I'll make the patch this afternoon after I get back from lunch.  If I go the -march=armv6k route, what's specific change do I need to make to which file?

BTW, this package was a low level dependency for the X windows packages.  Hopefully the fix will help shake loose some more X windows packages for compilation.

mpthompson said:

Oh no!!! I went to go order 3 more Freescale iMX53 Quick Start Boards from Digikey to complete the build cluster and they are out of stock!!!  And, they have a 12 week factory lead time putting deliver in late June or July!!! I tried Mouser, Newark, Avnet, Future Electronics and a few other places and they are all out of stock as well with fairly long delivery dates (Mouser may have some next week). I can't find any Internet distributor that has them in stock on either side of the Atlantic.

No worry. I'm sure your current cluster has finished the build before i get my Pi

mpthompson said:

tomtor said:

Either patch this file or use -march=armv6k
I vote for the last option.

Terrific.  I'll make the patch this afternoon after I get back from lunch.  If I go the -march=armv6k route, what's specific change do I need to make to which file?

BTW, this package was a low level dependency for the X windows packages.  Hopefully the fix will help shake loose some more X windows packages for compilation.

I'm not familiar with your repository build configuration and where the compiler options are located in the tree, that is plugwash territory.

If I remember correctly you specified the used flags in one of the earlier posts? Just changing armv6 to armv6k should do the trick, but patching the .h is also a good approach.

Let's just patch the arm.h file for now and if no other armv6 versus armv6k issues raise their head we can leave everything else unchanged. That would also be plugwash favorite approach.

I'll have a beer now and I'm heading towards my bed....

[plugwash]

tomtor said:

I"m not familiar with your repository build configuration and where the compiler options are located in the tree, that is plugwash territory.

If you want to make the change to the defaults for all builds then you would have to change and rebuild the gcc packages. If you want to use explicit for one particular package then the details will depend on the build system for that package. Usually putting something like "export CFLAGs += -march=armv6k" will do it.