hippy
Posts: 7435
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

[Solved] How do I prove I am generating ARMv6 code ?

Tue Jun 30, 2020 11:59 pm

I am using Cython and Cython3 to convert a Python program into C and then into '.so' libraries so they can be distributed with a Python wrapper.

I want to build on a Pi 3B (ARMv7) but generate '.so' files which will run on a Pi Zero (ARMv6), which I don't have to hand presently. That seems to be working.

The output using a Cython build, which includes "march=armv6" ...

Code: Select all

running build_ext
building 'core' extension
creating build/temp.linux-armv7l-2.7
arm-linux-gnueabihf-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -march=armv6 -fPIC -I/usr/include/python2.7 -c core.c -o build/temp.linux-armv7l-2.7/core.o
arm-linux-gnueabihf-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-InigCj/python2.7-2.7.16=. -fstack-protector-strong -Wformat -Werror=format-security -Wl,-z,relro -march=armv6 build/temp.linux-armv7l-2.7/core.o -o /home/pi/apps/xyzzy/core.so
And for a Cython3 build, which also includes "march=armv6" ....

Code: Select all

running build_ext
building 'core' extension
creating build/temp.linux-armv7l-3.7
arm-linux-gnueabihf-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -march=armv6 -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.7m -c core.c -o build/temp.linux-armv7l-3.7/core.o
arm-linux-gnueabihf-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,relro -march=armv6 -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-armv7l-3.7/core.o -o /home/pi/apps/xyzzy/core.cpython-37m-arm-linux-gnueabihf.so
And I have my two '.so' files ...

Code: Select all

-rwxr-xr-x 1 pi pi 1324668 Jul  1 00:48 core.cpython-37m-arm-linux-gnueabihf.so
-rwxr-xr-x 1 pi pi  939848 Jul  1 00:46 core.so
But how do I tell if the '.so' files are ARMv6, not ARMv7 ?

Code: Select all

pi@Pi3B:~/apps/xyzzy $ file core.cpython-37m-arm-linux-gnueabihf.so 
core.cpython-37m-arm-linux-gnueabihf.so: ELF 32-bit LSB pie executable, ARM, EABI5 version 1 (SYSV), dynamically linked, BuildID[sha1]=dd4ca9f3f5c90cdb4019473fff1b35414d6cd87a, with debug_info, not stripped

Code: Select all

pi@Pi3B:~/apps/xyzzy $ file core.so 
core.so: ELF 32-bit LSB pie executable, ARM, EABI5 version 1 (SYSV), dynamically linked, BuildID[sha1]=03e784158b32cbda354b3da537f66ddd6f672757, with debug_info, not stripped
Last edited by hippy on Wed Jul 01, 2020 2:31 pm, edited 1 time in total.

cleverca22
Posts: 545
Joined: Sat Aug 18, 2012 2:33 pm

Re: How do I prove I am generating ARMv6 code ?

Wed Jul 01, 2020 2:23 am

i was solving a similar problem a few years ago with: https://gist.github.com/cleverca22/d9fa ... 30f6518c9c

the answer is in some arch specific headers that you can dump with `readelf -A`

LdB
Posts: 1549
Joined: Wed Dec 07, 2016 2:29 pm

Re: How do I prove I am generating ARMv6 code ?

Wed Jul 01, 2020 3:22 am

Compile library using the proper full mcu flags rather than march ... code generated is MCU specific then not generic

PI1 hard floats
-mcpu=arm1176jzf-s -mfpu=vfp -mfloat-abi=hard
PI1 soft floats
-mcpu=arm1176jzf-s -mfpu=vfp -mfloat-abi=soft

hippy
Posts: 7435
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: How do I prove I am generating ARMv6 code ?

Wed Jul 01, 2020 2:29 pm

cleverca22 wrote:
Wed Jul 01, 2020 2:23 am
the answer is in some arch specific headers that you can dump with `readelf -A`
LdB wrote:
Wed Jul 01, 2020 3:22 am
-mcpu=arm1176jzf-s -mfpu=vfp -mfloat-abi=hard
-mcpu=arm1176jzf-s -mfpu=vfp -mfloat-abi=soft
Thanks both. That resolved the issue.

-march=armv6

Code: Select all

Tag_CPU_name: "6"
Tag_CPU_arch: v6
-mcpu=arm1176jzf-s -mfpu=vfp -mfloat-abi=hard

Code: Select all

Tag_CPU_name: "6KZ"
Tag_CPU_arch: v6KZ
And I guess it's that one as '-mfloat-abi=soft' fails to compile.

Oddly, with no '-march=', no '-mcpu=', it appears to behave as if '-march=armv6' anyway, so my assumption it would be generating ARMv7 code appears to have been faulty. But better to ensure it is ARMv6 rather than get bitten if that every does change.

tttapa
Posts: 31
Joined: Mon Apr 06, 2020 2:52 pm

Re: How do I prove I am generating ARMv6 code ?

Wed Jul 01, 2020 3:06 pm

hippy wrote:
Wed Jul 01, 2020 2:29 pm
Oddly, with no '-march=', no '-mcpu=', it appears to behave as if '-march=armv6' anyway, so my assumption it would be generating ARMv7 code appears to have been faulty. But better to ensure it is ARMv6 rather than get bitten if that every does change.
You can check what the compiler's default architecture options are using:

Code: Select all

arm-linux-gnueabihf-gcc -Q --help=target | grep 'mcpu\|march\|mfloat'
Pieter

hippy
Posts: 7435
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: [Solved] How do I prove I am generating ARMv6 code ?

Wed Jul 01, 2020 3:18 pm

Thanks Pieter,

Code: Select all

  -march=                              armv6+fp
  -mcpu=                      
  -mfloat-abi=                         hard
That does makes sense. Through the build directories being named 'armv7l' I simply presumed it may be generating code for ARMv7 and knew I needed to generate code for ARMv6.

GlowInTheDark
Posts: 583
Joined: Sat Nov 09, 2019 12:14 pm

Re: [Solved] How do I prove I am generating ARMv6 code ?

Wed Jul 01, 2020 4:34 pm

By the way, did you follow up on the "readelf" suggestion?

I did a little testing and that seems to work. readelf -A file | grep CPU_arch:

does the trick.

Notes:
1) As far as I can tell, the Raspbian userland is still all compiled with v6. This is so the same image works on the PiZero as on the regular PIs.
2) If you compile stuff yourself (on a regular Pi), it will probably be compiled with v7. (And thus won't run on the Zero).
GitD's list of things that are not ready for prime time:
1) IPv6
2) 64 bit OSes
3) USB 3
4) Bluetooth

hippy
Posts: 7435
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: [Solved] How do I prove I am generating ARMv6 code ?

Wed Jul 01, 2020 5:08 pm

GlowInTheDark wrote:
Wed Jul 01, 2020 4:34 pm
By the way, did you follow up on the "readelf" suggestion?
Yes; a couple of posts up.
GlowInTheDark wrote:
Wed Jul 01, 2020 4:34 pm
If you compile stuff yourself (on a regular Pi), it will probably be compiled with v7. (And thus won't run on the Zero).
That's what I was thinking but it seems they'd considered portability and default to v6, presumably so own compiled code does run on whatever board one has put the SD card in, copied the executable to.

To get ARMv7 exeutables one has to explicitly add '-march=armv7' or maybe some 'mcpu=' option, 'readelf' then shows -

Code: Select all

  Tag_CPU_name: "7"
  Tag_CPU_arch: v7

User avatar
Paeryn
Posts: 2952
Joined: Wed Nov 23, 2011 1:10 am
Location: Sheffield, England

Re: [Solved] How do I prove I am generating ARMv6 code ?

Wed Jul 01, 2020 5:42 pm

GlowInTheDark wrote:
Wed Jul 01, 2020 4:34 pm
2) If you compile stuff yourself (on a regular Pi), it will probably be compiled with v7. (And thus won't run on the Zero).
No, gcc supplied with RPiOS defaults to ARMv6 no matter which model you compile on, you have to specifically tell it to use ARMv7 if you want it.

The last thing RPF want is for users compiling programs with the supplied tools on their RPi4 and finding it doesn't work when they copy the program to their RPi0.
She who travels light — forgot something.

GlowInTheDark
Posts: 583
Joined: Sat Nov 09, 2019 12:14 pm

Re: [Solved] How do I prove I am generating ARMv6 code ?

Wed Jul 01, 2020 5:43 pm

That's what I was thinking but it seems they'd considered portability and default to v6, presumably so own compiled code does run on whatever board one has put the SD card in, copied the executable to.
Yes, this seems to be the case (at least in some versions), but, of course, there have been many revisions of Raspbian and so many versions of gcc floating around through the years, so hard to be sure of anything for certain.

Anyway, it seems I have at least one self-compiled program that is v7. I'm 98% sure that the configure script for that program tests for and sets the "march" option appropriately (i.e., the configure script detects that it is running on a v7 CPU, so selects to build for that CPU.
To get ARMv7 executables one has to explicitly add '-march=armv7' or maybe some 'mcpu=' option,
Yep. As above, a "configure" script may do this for you.
GitD's list of things that are not ready for prime time:
1) IPv6
2) 64 bit OSes
3) USB 3
4) Bluetooth

GlowInTheDark
Posts: 583
Joined: Sat Nov 09, 2019 12:14 pm

Re: [Solved] How do I prove I am generating ARMv6 code ?

Wed Jul 01, 2020 5:49 pm

The last thing RPF want is for users compiling programs with the supplied tools on their RPi4 and finding it doesn't work when they copy the program to their RPi0.
One could argue that this is backwards (note that merely suggesting this as a possibility might get me banned).

Most people who own a "regular" Pi don't even have a Pi0, so your reference to "their RPi0" is a lot like saying "Well, it might not work on their go-kart [or their Ferrari]".

I think a well-written configure script should set things up to build the best version possible for the CPU architecture upon which the configure sciprt is running. That seems obvious to me. Of course, the user can over-ride this if they like (and they know how to do it).

The configure script for GAWK, in fact, does just that. The result, of course, is a gawk executable that doesn't run on the Zero. This is as it should be.
GitD's list of things that are not ready for prime time:
1) IPv6
2) 64 bit OSes
3) USB 3
4) Bluetooth

jahboater
Posts: 5671
Joined: Wed Feb 04, 2015 6:38 pm
Location: West Dorset

Re: [Solved] How do I prove I am generating ARMv6 code ?

Wed Jul 01, 2020 6:02 pm

Well, if the user knows enough to fiddle with compiler options explicitly targeting a late model Pi (or enough to build their own compiler) then they would know that the resulting programs are not going to run on a Pi Zero!
Pi4 8GB running PIOS64

hippy
Posts: 7435
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: [Solved] How do I prove I am generating ARMv6 code ?

Wed Jul 01, 2020 6:18 pm

GlowInTheDark wrote:
Wed Jul 01, 2020 5:49 pm
I think a well-written configure script should set things up to build the best version possible for the CPU architecture upon which the configure sciprt is running. That seems obvious to me.
I doubt there's going to be any agreement on which is best and what the default should be.

I think the current default, 'compile so it runs on everything', ie ARMv6, is the most appropriate option in most cases. It's the path to 'least surprise', in that it will run on anything, and likely executes only a little less efficiently on ARMv7 than it could.

My problem was I didn't know what the default was, what I was generating, and didn't know how to find out either, I merely realised I could have a potential problem.

I'm not sure what happens when the Pi goes 64-bit. With a 32-bit userland I presume we carry on as now, no change. Will ARMv8 run ARMv6 executable code ?

GlowInTheDark
Posts: 583
Joined: Sat Nov 09, 2019 12:14 pm

Re: [Solved] How do I prove I am generating ARMv6 code ?

Wed Jul 01, 2020 6:20 pm

My problem was I didn't know what the default was, what I was generating, and didn't know how to find out either, I merely realised I could have a potential problem.
Totally agree. And I think we all learned something from this thread (namely, about "readelf -A").
GitD's list of things that are not ready for prime time:
1) IPv6
2) 64 bit OSes
3) USB 3
4) Bluetooth

jahboater
Posts: 5671
Joined: Wed Feb 04, 2015 6:38 pm
Location: West Dorset

Re: [Solved] How do I prove I am generating ARMv6 code ?

Wed Jul 01, 2020 10:40 pm

hippy wrote:
Wed Jul 01, 2020 6:18 pm
I'm not sure what happens when the Pi goes 64-bit. With a 32-bit userland I presume we carry on as now, no change. Will ARMv8 run ARMv6 executable code ?
PIOS64 is 64-bit userland.
And it can run ARMv6 32-bit executable code!

Code: Select all

pi@raspberrypi:~ $ uname -a
Linux raspberrypi 5.4.47-v8+ #1322 SMP PREEMPT Wed Jun 17 18:05:46 BST 2020 aarch64 GNU/Linux
pi@raspberrypi:~ $ file /bin/ls
/bin/ls: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=9ecc063cc78a0a8c15f950e5e8fc4a6954c734dc, stripped
pi@raspberrypi:~ $ file try
try: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, for GNU/Linux 3.2.0, not stripped
pi@raspberrypi:~ $ ./try
Hello, world
pi@raspberrypi:~ $ 
The compiler on PIOS64 produces 64-bit aarch64 executables by default which obviously cannot run on a Pi Zero:

Code: Select all

pi@raspberrypi:~ $ gcc-8 try.c -o try
pi@raspberrypi:~ $ file try
try: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=fc0ef0866868a81f1abd18096566a413f15e809b, not stripped
pi@raspberrypi:~ $ 
Pi4 8GB running PIOS64

hippy
Posts: 7435
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: [Solved] How do I prove I am generating ARMv6 code ?

Thu Jul 02, 2020 12:26 am

jahboater wrote:
Wed Jul 01, 2020 10:40 pm
PIOS64 is 64-bit userland.
And it can run ARMv6 32-bit executable code!
Thanks for testing.

I have my Cython3 generated 'core.cpython-37m-arm-linux-gnueabihf.so' (ARMv6) copied over to a Pi 1B and that runs as hoped for. So the 32-bit case appears to be solved for all Pi's; Zero, 1B through 4B.

Not quite so lucky with a Pi 4B using Aarch64, full 64-bit. Cython3 generates a '.so' (Aarch64) which my Python 3 code imports and calls, so that's fine.

But after copying over 'core.cpython-37m-arm-linux-gnueabihf.so' (ARMv6), Python 3 doesn't recognise it as a module available for import.

The Aarch64-GCC called by Cython3 won't accept -march=armv6 or -march=-armv7 so can't get a suitable '.so' that way.

So it looks like I'm sorted for ARM 32-bit and Aarch64 64-bit, just not 'one size fits all'. More investigation needed there. But that's not too much of a worry at the present time.

One thing I did note under Aarch64 is that 'readelf -A' doesn't return anything for my Aarch64 '.so' file, but 'readelf -a' does give a "Machine: Aarch64" which may be an equivalent.

LdB
Posts: 1549
Joined: Wed Dec 07, 2016 2:29 pm

Re: [Solved] How do I prove I am generating ARMv6 code ?

Thu Jul 02, 2020 1:52 am

I learnt to do it for the same reasons and it is and will become more and more common on future 64bit Pi's. The same thing occurs on Intel where you have 32bit and 64bit apps running into a 64 bit O/S.

However you might as well have the flags for all the old models

Pi2 ARM7 32 bit
-mcpu=cortex-a7 -mfpu=neon -mfloat-abi=hard

PI3 ARM8 32 bit
-mcpu=cortex-a53 -mfpu=neon-vfpv4 -mfloat-abi=hard

PI3 ARM8 64bit from a Pi4 default (AARCH64 is hard float always)
-mcpu=cortex-a53+fp+simd

jahboater
Posts: 5671
Joined: Wed Feb 04, 2015 6:38 pm
Location: West Dorset

Re: [Solved] How do I prove I am generating ARMv6 code ?

Thu Jul 02, 2020 2:50 am

LdB wrote:
Thu Jul 02, 2020 1:52 am
PI3 ARM8 64bit from a Pi4 default (AARCH64 is hard float always)
-mcpu=cortex-a53+fp+simd
For the Pi4 64-bit I now just use:

-mcpu=native
Pi4 8GB running PIOS64

LdB
Posts: 1549
Joined: Wed Dec 07, 2016 2:29 pm

Re: [Solved] How do I prove I am generating ARMv6 code ?

Thu Jul 02, 2020 6:29 am

The OP is cross compiling that is only useful in the one situation you are compiling code for the Pi4 on the Pi4.

The proper cross compile commands for the Pi4 64 bit are
-mcpu=cortex-a72+crypto

Pi4 32 bit comping
-mcpu=cortex-a72 -mfloat-abi=hard -mfpu=neon-fp-armv8

Return to “C/C++”