hydra3333
Posts: 207
Joined: Thu Jan 10, 2013 11:48 pm

is it possible to compile a Python program into an executable ?

Mon Aug 17, 2020 2:39 am

Hello.

Is it possible to compile a Python program into an executable ?

If so I'm tempted to look at and see if it would be possible to port hd-idle (it's go code) https://github.com/adelolmo/hd-idle to python3.

Thanks.

User avatar
HermannSW
Posts: 3775
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: is it possible to compile a Python program into an executable ?

Mon Aug 17, 2020 5:46 am

https://stamm-wilbrandt.de/2wheel_balancing_robot
https://stamm-wilbrandt.de/en#raspcatbot
https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://github.com/Hermann-SW/raspiraw
https://stamm-wilbrandt.de/en/Raspberry_camera.html

hippy
Posts: 9158
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: is it possible to compile a Python program into an executable ?

Mon Aug 17, 2020 1:09 pm

It's as simple as ...

fibo.py

Code: Select all

def Fibo(n):
  if n <= 1 : return n
  else      : return Fibo(n-1) + Fibo(n-2)
print(Fibo(24))

Code: Select all

sudo apt-get install cython3
cython3 --embed -o fibo.c fibo.py
gcc -Os -I /usr/include/python3.7m -lpython3.7m -o fibo fibo.c
./fibo

GlowInTheDark
Posts: 1353
Joined: Sat Nov 09, 2019 12:14 pm

Re: is it possible to compile a Python program into an executable ?

Mon Aug 17, 2020 1:34 pm

As always, we'd need to know why the OP wants/needs to "compile". What exactly does that word mean to him, and what is the long-range goal?

Usually, when people say "compile", they really mean "obfuscate".
GitD's list of things that are not ready for prime time:
1) IPv6
2) 64 bit OSes
3) USB 3
4) Bluetooth

Loves Linux; loves to dance.

hippy
Posts: 9158
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: is it possible to compile a Python program into an executable ?

Mon Aug 17, 2020 2:05 pm

GlowInTheDark wrote:
Mon Aug 17, 2020 1:34 pm
As always, we'd need to know why the OP wants/needs to "compile". What exactly does that word mean to him, and what is the long-range goal?
"Compile a Python program into an executable" seems pretty clear-cut to me.
GlowInTheDark wrote:
Mon Aug 17, 2020 1:34 pm
Usually, when people say "compile", they really mean "obfuscate".
I'm not convinced of that, but actually compiling should do a pretty good job of obfuscation, though I admit I've never tried to decompile a compiled Python program.

fanoush
Posts: 705
Joined: Mon Feb 27, 2012 2:37 pm

Re: is it possible to compile a Python program into an executable ?

Mon Aug 17, 2020 2:37 pm

GlowInTheDark wrote:
Mon Aug 17, 2020 1:34 pm
Usually, when people say "compile", they really mean "obfuscate".
OP is pretty clear - porting/rewriting specific program written in Go, obfuscation is quite unlikely here.
hippy wrote:
Mon Aug 17, 2020 1:09 pm
It's as simple as ...

fibo.py

Code: Select all

def Fibo(n):
  if n <= 1 : return n
  else      : return Fibo(n-1) + Fibo(n-2)
print(Fibo(24))

Code: Select all

sudo apt-get install cython3
cython3 --embed -o fibo.c fibo.py
gcc -Os -I /usr/include/python3.7m -lpython3.7m -o fibo fibo.c
./fibo
thanks, didn't work for me - gcc produced various errors, this one worked

Code: Select all

gcc -fPIC fibo.c `python3.7m-config --cflags --ldflags`  -o fibo
however there is runtime dependency to libpython3.7m.so.1.0

Code: Select all

$ ldd fibo
        linux-vdso.so.1 (0x00007fffe8fec000)
        libpython3.7m.so.1.0 => /usr/lib/x86_64-linux-gnu/libpython3.7m.so.1.0 (0x00007f7328a00000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f7328600000)
        libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007f73283c0000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f73281a0000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f7327f80000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f7327d70000)
        libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f7327b50000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f73277b0000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f7329400000)
So maybe it is not so good idea to write system stuff in python due to library dependencies? Not sure how Go is regarding this.

hippy
Posts: 9158
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: is it possible to compile a Python program into an executable ?

Mon Aug 17, 2020 3:04 pm

fanoush wrote:
Mon Aug 17, 2020 2:37 pm
didn't work for me - gcc produced various errors, this one worked

Code: Select all

gcc -fPIC fibo.c `python3.7m-config --cflags --ldflags`  -o fibo
however there is runtime dependency to libpython3.7m.so.1.0

Code: Select all

$ ldd fibo
        linux-vdso.so.1 (0x00007fffe8fec000)
        libpython3.7m.so.1.0 => /usr/lib/x86_64-linux-gnu/libpython3.7m.so.1.0 (0x00007f7328a00000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f7328600000)
        libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007f73283c0000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f73281a0000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f7327f80000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f7327d70000)
        libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f7327b50000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f73277b0000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f7329400000)
So maybe it is not so good idea to write system stuff in python due to library dependencies? Not sure how Go is regarding this.
Interesting. From the "x86_64-Linux" I'm guessing that's not on a Pi. For me it's similar, and has the 'libpython3.7m' dependency ...

Code: Select all

pi@Pi3B:~/tmp $ ldd fibo
        linux-vdso.so.1 (0x7ee83000)
        /usr/lib/arm-linux-gnueabihf/libarmmem-${PLATFORM}.so => /usr/lib/arm-linux-gnueabihf/libarmmem-v7l.so (0x76f50000)
        libpython3.7m.so.1.0 => /usr/lib/arm-linux-gnueabihf/libpython3.7m.so.1.0 (0x76af2000)
        libc.so.6 => /lib/arm-linux-gnueabihf/libc.so.6 (0x769a4000)
        libexpat.so.1 => /lib/arm-linux-gnueabihf/libexpat.so.1 (0x76963000)
        libz.so.1 => /lib/arm-linux-gnueabihf/libz.so.1 (0x76938000)
        libcrypt.so.1 => /lib/arm-linux-gnueabihf/libcrypt.so.1 (0x768f8000)
        libpthread.so.0 => /lib/arm-linux-gnueabihf/libpthread.so.0 (0x768ce000)
        libdl.so.2 => /lib/arm-linux-gnueabihf/libdl.so.2 (0x768bb000)
        libutil.so.1 => /lib/arm-linux-gnueabihf/libutil.so.1 (0x768a8000)
        libm.so.6 => /lib/arm-linux-gnueabihf/libm.so.6 (0x76826000)
        /lib/ld-linux-armhf.so.3 (0x76f65000)
        libgcc_s.so.1 => /lib/arm-linux-gnueabihf/libgcc_s.so.1 (0x767f9000)
As you say; perhaps not a good idea for system code.

I had never used 'Cython typing' where the type of parameters and returns can be specified. Trying it shows a massive speed improvement for my simplistic recursive Fibonacci benchmark above. for Fibo(42) it reduced runtime from over 8 minutes to below 8 seconds. That put Cython in second place, behind C but before Rust, for that benchmark.

Code: Select all

cdef int Fibo(int n):
  if n <= 1 : return n
  else      : return Fibo(n-1) + Fibo(n-2)
print(Fibo(24))

GlowInTheDark
Posts: 1353
Joined: Sat Nov 09, 2019 12:14 pm

Re: is it possible to compile a Python program into an executable ?

Mon Aug 17, 2020 3:18 pm

To be more clear, since it seems to have floated over people's heads the first time, there are two basic reasons why people want to "compile" programs written in script languages (e.g., Python):

1) Code security. I.e., I don't want the plebes looking at my source code. I.e., Obfuscation.

2) Performance gain

My experience is that most of the time, responders (aka, "help givers") assume #2 when #1 is actually what OP wants. Note that this may or may not be true in this instance case, but it is the way to bet.

That all said, I don't see what the connection is between compiling Python and porting whatever it was that OP was interested in porting. Those seem like unrelated topics. Unless...

Now we get to the real meat of it. When I first this thread, what I concluded was that OP was in some kind of corporate environment where he would only be allowed to work on the project of porting whatevr it was the OP was interested in porting if it were possible to compile. I.e., a corporate policy thing.
GitD's list of things that are not ready for prime time:
1) IPv6
2) 64 bit OSes
3) USB 3
4) Bluetooth

Loves Linux; loves to dance.

hippy
Posts: 9158
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: is it possible to compile a Python program into an executable ?

Mon Aug 17, 2020 3:37 pm

GlowInTheDark wrote:
Mon Aug 17, 2020 3:18 pm
That all said, I don't see what the connection is between compiling Python and porting whatever it was that OP was interested in porting. Those seem like unrelated topics.
Me too. It seems odd to want to take an open source program in one language which can be easily compiled on a Pi and other platforms to become an executable, want to rewrite that in Python and then compile that to be an executable.

That seems a whole lot of effort for little gain. But I just assumed the OP had their reasons.

And, to be honest, I was addressing the title of the thread which I expect will have drawn other Python users in, as well as the OP.

plugwash
Forum Moderator
Forum Moderator
Posts: 3737
Joined: Wed Dec 28, 2011 11:45 pm

Re: is it possible to compile a Python program into an executable ?

Mon Aug 17, 2020 3:44 pm

hippy wrote:
Mon Aug 17, 2020 2:05 pm
"Compile a Python program into an executable" seems pretty clear-cut to me.
The thing is while the cython invocation mentioned earlirer will "compile a python program into an executable" said executable will.

1. Be only marginally faster than the original script.
2. Require a specific version of libpython to run.
3. Require the python modules the original script required to run.

hippy
Posts: 9158
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: is it possible to compile a Python program into an executable ?

Mon Aug 17, 2020 4:25 pm

plugwash wrote:
Mon Aug 17, 2020 3:44 pm
hippy wrote:
Mon Aug 17, 2020 2:05 pm
"Compile a Python program into an executable" seems pretty clear-cut to me.
The thing is while the cython invocation mentioned earlirer will "compile a python program into an executable" said executable will.

1. Be only marginally faster than the original script.
Not true in my experience. While the simplistic recursive Fibonacci is a non-representative benchmark; just compiling with Cython gives a near two-fold improvement in execution over Python bytecode interpretation for Fibo(42). Adding Cython typing, as above, delivers a near 120-fold improvement in performance.

Code: Select all

                           Fibo(24)    Fibo(42)

C             optimised :    0.008s      5.022s
Cython 3          typed :    0.160s      7.887s
Rust          optimised :    0.013s      8.476s
C#                 mono :    0.168s     10.887s
Cython 3                :    0.240s   8m36.210s
Python 3         3.7.3  :    0.531s  15m42.762s
plugwash wrote:
Mon Aug 17, 2020 3:44 pm
2. Require a specific version of libpython to run.
3. Require the python modules the original script required to run.
Quite possibly but I'm not sure how much of a problem that would be.

It might be an issue if attempting to distribute a compiled Python executable, but not a problem if someone is simply compiling their own Python source to gain advantage through doing that.

fanoush
Posts: 705
Joined: Mon Feb 27, 2012 2:37 pm

Re: is it possible to compile a Python program into an executable ?

Mon Aug 17, 2020 6:19 pm

hippy wrote:
Mon Aug 17, 2020 3:37 pm
Me too. It seems odd to want to take an open source program in one language which can be easily compiled on a Pi and other platforms to become an executable, want to rewrite that in Python and then compile that to be an executable.
I can imagine that if someone who only knows Python well - not Go or C sees Go source but wants to modify/improve it, then it may somehow make sense. And reason for wanting executable - to have it working where python is not installed or to reduce memory requirements or make it smaller/faster.

GlowInTheDark
Posts: 1353
Joined: Sat Nov 09, 2019 12:14 pm

Re: is it possible to compile a Python program into an executable ?

Mon Aug 17, 2020 10:08 pm

Besides which, can't you just "compile" your Python code into .pyc files? That achieves both speedup (some - not a lot - and certainly not what you would get if you hand-re-coded the Python into, e.g., C and compiled that - but some) and also obfuscation.

There is a certain well-known Python application which is distributed as a single executable file, which is, in fact, a ZIP file - that is, it starts out with:

#!/usr/bin/env python
PK<much binary glop>

If you unpack the ZIP file contained therein, you will see all the .py files (uncompiled code). But if you then run:

$ python -m compileall ./

it will convert all the .py's to .pyc's. You then re-assemble the ZIP file from the .pyc's and you end up with something that is larger than the original, but which loads faster (because the component files are already compiled).

And, of course, since they are compiled, the original source is not on view. Thus, obfuscation.

P.S. Note, BTW: that both the before and after programs in the above process can be considered to be "executables" - so, in a sense, it is already the case that you can take a Python project (collection of Python source files) and make a single executable program file from it.
GitD's list of things that are not ready for prime time:
1) IPv6
2) 64 bit OSes
3) USB 3
4) Bluetooth

Loves Linux; loves to dance.

enedil
Posts: 80
Joined: Sat Feb 21, 2015 4:22 pm
Location: Toruń, Poland
Contact: Website

Re: is it possible to compile a Python program into an executable ?

Tue Aug 18, 2020 12:40 am

GlowInTheDark wrote:
Mon Aug 17, 2020 10:08 pm
Besides which, can't you just "compile" your Python code into .pyc files?
Every Python script before running is compiled to .pyc file so this is a noop (unless you frequently remove the pyc files and rerun your script, which would be wasteless).

GlowInTheDark wrote:
Mon Aug 17, 2020 10:08 pm
And, of course, since they are compiled, the original source is not on view. Thus, obfuscation.
I assume you never tried reverse engineering Python and native apps, but I assure you that Python reverse enginnering is several times easier. There are some really accurate decompilers for Python.


Now, the real question is, what does the OP want the decompilation for. Unless (s)he responds, it's a dead topic.
- What Can a Thoughtful Man Hope for Mankind on Earth, Given the Experience of the Past Million Years?
- Nothing.

Kurt Vonnegut, Cat's Cradle

GlowInTheDark
Posts: 1353
Joined: Sat Nov 09, 2019 12:14 pm

Re: is it possible to compile a Python program into an executable ?

Tue Aug 18, 2020 1:05 am

Every Python script before running is compiled to .pyc file so this is a noop (unless you frequently remove the pyc files and rerun your script, which would be wasteless).
It is *NOT* a no-op. As I said (several times, in fact), it makes it load and start executing faster because it doesn't have to compile them at runtime.

The difference is small, but definitely noticeable - and in my use-case, critical. It matters when you run a given script repeatedly; I'm guessing that your use-case is a single, long-running script.

Regarding the decompiling - sure, any nut can be cracked. You are right, of course, that we'll never really know unless/until OP posts again.
GitD's list of things that are not ready for prime time:
1) IPv6
2) 64 bit OSes
3) USB 3
4) Bluetooth

Loves Linux; loves to dance.

enedil
Posts: 80
Joined: Sat Feb 21, 2015 4:22 pm
Location: Toruń, Poland
Contact: Website

Re: is it possible to compile a Python program into an executable ?

Tue Aug 18, 2020 4:16 am

The .pyc files are generated automatically, they land in directory __pycache__ (unless you disable it by env variable). So if you run your code multiple times, withouth changing, the first run will generate the byte code and each subsequent will detect the existence of cache and load from there.
- What Can a Thoughtful Man Hope for Mankind on Earth, Given the Experience of the Past Million Years?
- Nothing.

Kurt Vonnegut, Cat's Cradle

hydra3333
Posts: 207
Joined: Thu Jan 10, 2013 11:48 pm

Re: is it possible to compile a Python program into an executable ?

Tue Aug 18, 2020 5:28 am

GlowInTheDark wrote:
Mon Aug 17, 2020 1:34 pm
As always, we'd need to know why the OP wants/needs to "compile". What exactly does that word mean to him, and what is the long-range goal?

Usually, when people say "compile", they really mean "obfuscate".
No. I'm an ancient programmer. I did mean compile into an executable so that no interpreters were invoked at runtime, eg hd-idle needs to be "low touch". If it were viable, and I did try to do a port of some kind, it'd be on github.

I'm not so sure about included python libraries, I suppose the compilation process takes care of that.

Now ... on to read the other posts !

edit: many thanks to everyone who responded.
Regarding de-compilation, no that's not on my agenda.
As someone mentioned, I wasn't comfortable with Go; Python is comparatively "easy" to my eyes.
Although, by the look of the responses I suppose I should just make an effort and learn Go.
At my age: so many languages, so little time :) If it were in fortran-77 or basic or PDP/Vax assembler ... ;)

GlowInTheDark
Posts: 1353
Joined: Sat Nov 09, 2019 12:14 pm

Re: is it possible to compile a Python program into an executable ?

Tue Aug 18, 2020 11:07 am

enedil wrote:
Tue Aug 18, 2020 4:16 am
The .pyc files are generated automatically, they land in directory __pycache__ (unless you disable it by env variable). So if you run your code multiple times, withouth changing, the first run will generate the byte code and each subsequent will detect the existence of cache and load from there.
Nope.
GitD's list of things that are not ready for prime time:
1) IPv6
2) 64 bit OSes
3) USB 3
4) Bluetooth

Loves Linux; loves to dance.

GlowInTheDark
Posts: 1353
Joined: Sat Nov 09, 2019 12:14 pm

Re: is it possible to compile a Python program into an executable ?

Tue Aug 18, 2020 11:15 am

Usually, when people say "compile", they really mean "obfuscate".
No...
First, point of order: I said "Usually, when ..."; if indeed your interest is no in obfuscation, it doesn't change the fact that I am still correct in saying that "Usually". I.e., an exception does not disprove a "Usually" type statement.

Anyway, it sounds like you are, indeed, interested in a performance gain. The problem, of course, is all the libraries, which still need to be present at runtime and within which most of the execution time will be spent. In real life code (as opposed to benchmarks), even if the code that you write gets converted to C and runs faster, you still have the libraries.
GitD's list of things that are not ready for prime time:
1) IPv6
2) 64 bit OSes
3) USB 3
4) Bluetooth

Loves Linux; loves to dance.

hippy
Posts: 9158
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: is it possible to compile a Python program into an executable ?

Tue Aug 18, 2020 11:53 am

GlowInTheDark wrote:
Tue Aug 18, 2020 11:07 am
enedil wrote:
Tue Aug 18, 2020 4:16 am
The .pyc files are generated automatically, they land in directory __pycache__ (unless you disable it by env variable). So if you run your code multiple times, withouth changing, the first run will generate the byte code and each subsequent will detect the existence of cache and load from there.
Nope.
Could you explain how 'enedil' is wrong in what they wrote ?

It's pretty much what I would have written myself. Python 2.7 created .pyc files in the same directory as the .py files, where Python 3 uses a __pycache__ sub-directory by default but that's all I'd add.

hippy
Posts: 9158
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: is it possible to compile a Python program into an executable ?

Tue Aug 18, 2020 11:58 am

Usually
Evidence-free arguments are usually entertaining ;)

User avatar
paddyg
Posts: 2612
Joined: Sat Jan 28, 2012 11:57 am
Location: UK

Re: is it possible to compile a Python program into an executable ?

Tue Aug 18, 2020 12:19 pm

...returning to the OPs question and later clarification! I would suggest that for the kind of application they're looking at (monitoring hard disks in the background) uncompiled python would be fine. The overhead is tiny if the application is dormant most of the time. IMO it's not worth thinking about trying to "lighten the touch" until you've written it and found it needs to be improved... didn't Donald Knuth say something about this years ago?
also https://groups.google.com/forum/?hl=en-GB&fromgroups=#!forum/pi3d

hydra3333
Posts: 207
Joined: Thu Jan 10, 2013 11:48 pm

Re: is it possible to compile a Python program into an executable ?

Tue Aug 18, 2020 4:55 pm

paddyg wrote:
Tue Aug 18, 2020 12:19 pm
...returning to the OPs question and later clarification! I would suggest that for the kind of application they're looking at (monitoring hard disks in the background) uncompiled python would be fine. The overhead is tiny if the application is dormant most of the time. IMO it's not worth thinking about trying to "lighten the touch" until you've written it and found it needs to be improved... didn't Donald Knuth say something about this years ago?
Thank you.

Whilst we did use an early LaTex for our documentation for a while (and thought we were made) ;) we weren't big on Knuth.

It was mainly to be an interesting exercise for me for my Pi4s running a web based chromecast server.
Light-touch programs are generally a good thing to do for code not frequently changed and/or which touches on performance related stuff ... learned that the hard way, early on.

Having said that, I just realised how much vbscript code I have hanging around here and there on my PCs :)

I may still do a port after having a go at learning go, but go seems to work OK to build stuff on Pi's.

Good Health to you all.

satineeraj
Posts: 4
Joined: Wed May 27, 2020 6:41 am

Re: is it possible to compile a Python program into an executable ?

Fri Aug 21, 2020 4:09 am

Yes you can compile the Python Code through the compilers check best Python Compilers here:http://www.techgeekbuzz.com/best-python-compiler/

enedil
Posts: 80
Joined: Sat Feb 21, 2015 4:22 pm
Location: Toruń, Poland
Contact: Website

Re: is it possible to compile a Python program into an executable ?

Fri Aug 21, 2020 12:18 pm

satineeraj wrote:
Fri Aug 21, 2020 4:09 am
Yes you can compile the Python Code through the compilers check best Python Compilers here:http://www.techgeekbuzz.com/best-python-compiler/
but they all come with some caveats, that were already mentioned
- What Can a Thoughtful Man Hope for Mankind on Earth, Given the Experience of the Past Million Years?
- Nothing.

Kurt Vonnegut, Cat's Cradle

Return to “Python”