is it possible to compile a Python program into an executable ?

[hydra3333]

Hello.

Is it possible to compile a Python program into an executable ?

If so I'm tempted to look at and see if it would be possible to port hd-idle (it's go code) https://github.com/adelolmo/hd-idle to python3.

Thanks.

[HermannSW]

Yes:
https://hackr.io/blog/best-python-compilers

[hippy]

It's as simple as ...

fibo.py

Code: Select all

def Fibo(n):
  if n <= 1 : return n
  else      : return Fibo(n-1) + Fibo(n-2)
print(Fibo(24))

Code: Select all

sudo apt-get install cython3
cython3 --embed -o fibo.c fibo.py
gcc -Os -I /usr/include/python3.7m -lpython3.7m -o fibo fibo.c
./fibo

[GlowInTheDark]

As always, we'd need to know why the OP wants/needs to "compile". What exactly does that word mean to him, and what is the long-range goal?

Usually, when people say "compile", they really mean "obfuscate".

[hippy]

As always, we'd need to know why the OP wants/needs to "compile". What exactly does that word mean to him, and what is the long-range goal?

"Compile a Python program into an executable" seems pretty clear-cut to me.

Usually, when people say "compile", they really mean "obfuscate".

I'm not convinced of that, but actually compiling should do a pretty good job of obfuscation, though I admit I've never tried to decompile a compiled Python program.

[fanoush]

Usually, when people say "compile", they really mean "obfuscate".

OP is pretty clear - porting/rewriting specific program written in Go, obfuscation is quite unlikely here.

It's as simple as ...

fibo.py

Code: Select all

def Fibo(n):
  if n <= 1 : return n
  else      : return Fibo(n-1) + Fibo(n-2)
print(Fibo(24))

Code: Select all

sudo apt-get install cython3
cython3 --embed -o fibo.c fibo.py
gcc -Os -I /usr/include/python3.7m -lpython3.7m -o fibo fibo.c
./fibo

thanks, didn't work for me - gcc produced various errors, this one worked

Code: Select all

gcc -fPIC fibo.c `python3.7m-config --cflags --ldflags`  -o fibo

however there is runtime dependency to libpython3.7m.so.1.0

Code: Select all

$ ldd fibo
        linux-vdso.so.1 (0x00007fffe8fec000)
        libpython3.7m.so.1.0 => /usr/lib/x86_64-linux-gnu/libpython3.7m.so.1.0 (0x00007f7328a00000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f7328600000)
        libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007f73283c0000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f73281a0000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f7327f80000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f7327d70000)
        libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f7327b50000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f73277b0000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f7329400000)

So maybe it is not so good idea to write system stuff in python due to library dependencies? Not sure how Go is regarding this.

[hippy]

didn't work for me - gcc produced various errors, this one worked

Code: Select all

gcc -fPIC fibo.c `python3.7m-config --cflags --ldflags`  -o fibo

however there is runtime dependency to libpython3.7m.so.1.0

Code: Select all

$ ldd fibo
        linux-vdso.so.1 (0x00007fffe8fec000)
        libpython3.7m.so.1.0 => /usr/lib/x86_64-linux-gnu/libpython3.7m.so.1.0 (0x00007f7328a00000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f7328600000)
        libexpat.so.1 => /lib/x86_64-linux-gnu/libexpat.so.1 (0x00007f73283c0000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f73281a0000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f7327f80000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f7327d70000)
        libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f7327b50000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f73277b0000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f7329400000)

So maybe it is not so good idea to write system stuff in python due to library dependencies? Not sure how Go is regarding this.

Interesting. From the "x86_64-Linux" I'm guessing that's not on a Pi. For me it's similar, and has the 'libpython3.7m' dependency ...

Code: Select all

pi@Pi3B:~/tmp $ ldd fibo
        linux-vdso.so.1 (0x7ee83000)
        /usr/lib/arm-linux-gnueabihf/libarmmem-${PLATFORM}.so => /usr/lib/arm-linux-gnueabihf/libarmmem-v7l.so (0x76f50000)
        libpython3.7m.so.1.0 => /usr/lib/arm-linux-gnueabihf/libpython3.7m.so.1.0 (0x76af2000)
        libc.so.6 => /lib/arm-linux-gnueabihf/libc.so.6 (0x769a4000)
        libexpat.so.1 => /lib/arm-linux-gnueabihf/libexpat.so.1 (0x76963000)
        libz.so.1 => /lib/arm-linux-gnueabihf/libz.so.1 (0x76938000)
        libcrypt.so.1 => /lib/arm-linux-gnueabihf/libcrypt.so.1 (0x768f8000)
        libpthread.so.0 => /lib/arm-linux-gnueabihf/libpthread.so.0 (0x768ce000)
        libdl.so.2 => /lib/arm-linux-gnueabihf/libdl.so.2 (0x768bb000)
        libutil.so.1 => /lib/arm-linux-gnueabihf/libutil.so.1 (0x768a8000)
        libm.so.6 => /lib/arm-linux-gnueabihf/libm.so.6 (0x76826000)
        /lib/ld-linux-armhf.so.3 (0x76f65000)
        libgcc_s.so.1 => /lib/arm-linux-gnueabihf/libgcc_s.so.1 (0x767f9000)

As you say; perhaps not a good idea for system code.

I had never used 'Cython typing' where the type of parameters and returns can be specified. Trying it shows a massive speed improvement for my simplistic recursive Fibonacci benchmark above. for Fibo(42) it reduced runtime from over 8 minutes to below 8 seconds. That put Cython in second place, behind C but before Rust, for that benchmark.

Code: Select all

cdef int Fibo(int n):
  if n <= 1 : return n
  else      : return Fibo(n-1) + Fibo(n-2)
print(Fibo(24))

[GlowInTheDark]

To be more clear, since it seems to have floated over people's heads the first time, there are two basic reasons why people want to "compile" programs written in script languages (e.g., Python):

1) Code security. I.e., I don't want the plebes looking at my source code. I.e., Obfuscation.

2) Performance gain

My experience is that most of the time, responders (aka, "help givers") assume #2 when #1 is actually what OP wants. Note that this may or may not be true in this instance case, but it is the way to bet.

That all said, I don't see what the connection is between compiling Python and porting whatever it was that OP was interested in porting. Those seem like unrelated topics. Unless...

Now we get to the real meat of it. When I first this thread, what I concluded was that OP was in some kind of corporate environment where he would only be allowed to work on the project of porting whatevr it was the OP was interested in porting if it were possible to compile. I.e., a corporate policy thing.

[hippy]

That all said, I don't see what the connection is between compiling Python and porting whatever it was that OP was interested in porting. Those seem like unrelated topics.

Me too. It seems odd to want to take an open source program in one language which can be easily compiled on a Pi and other platforms to become an executable, want to rewrite that in Python and then compile that to be an executable.

That seems a whole lot of effort for little gain. But I just assumed the OP had their reasons.

And, to be honest, I was addressing the title of the thread which I expect will have drawn other Python users in, as well as the OP.

[plugwash]

"Compile a Python program into an executable" seems pretty clear-cut to me.

The thing is while the cython invocation mentioned earlirer will "compile a python program into an executable" said executable will.

1. Be only marginally faster than the original script.
2. Require a specific version of libpython to run.
3. Require the python modules the original script required to run.

[hippy]

"Compile a Python program into an executable" seems pretty clear-cut to me.

The thing is while the cython invocation mentioned earlirer will "compile a python program into an executable" said executable will.

1. Be only marginally faster than the original script.

Not true in my experience. While the simplistic recursive Fibonacci is a non-representative benchmark; just compiling with Cython gives a near two-fold improvement in execution over Python bytecode interpretation for Fibo(42). Adding Cython typing, as above, delivers a near 120-fold improvement in performance.

Code: Select all

                           Fibo(24)    Fibo(42)

C             optimised :    0.008s      5.022s
Cython 3          typed :    0.160s      7.887s
Rust          optimised :    0.013s      8.476s
C#                 mono :    0.168s     10.887s
Cython 3                :    0.240s   8m36.210s
Python 3         3.7.3  :    0.531s  15m42.762s

2. Require a specific version of libpython to run.
3. Require the python modules the original script required to run.

Quite possibly but I'm not sure how much of a problem that would be.

It might be an issue if attempting to distribute a compiled Python executable, but not a problem if someone is simply compiling their own Python source to gain advantage through doing that.

[fanoush]

Me too. It seems odd to want to take an open source program in one language which can be easily compiled on a Pi and other platforms to become an executable, want to rewrite that in Python and then compile that to be an executable.

I can imagine that if someone who only knows Python well - not Go or C sees Go source but wants to modify/improve it, then it may somehow make sense. And reason for wanting executable - to have it working where python is not installed or to reduce memory requirements or make it smaller/faster.

[GlowInTheDark]

Besides which, can't you just "compile" your Python code into .pyc files? That achieves both speedup (some - not a lot - and certainly not what you would get if you hand-re-coded the Python into, e.g., C and compiled that - but some) and also obfuscation.

There is a certain well-known Python application which is distributed as a single executable file, which is, in fact, a ZIP file - that is, it starts out with:

#!/usr/bin/env python
PK<much binary glop>

If you unpack the ZIP file contained therein, you will see all the .py files (uncompiled code). But if you then run:

$ python -m compileall ./

it will convert all the .py's to .pyc's. You then re-assemble the ZIP file from the .pyc's and you end up with something that is larger than the original, but which loads faster (because the component files are already compiled).

And, of course, since they are compiled, the original source is not on view. Thus, obfuscation.

P.S. Note, BTW: that both the before and after programs in the above process can be considered to be "executables" - so, in a sense, it is already the case that you can take a Python project (collection of Python source files) and make a single executable program file from it.

[enedil]

Besides which, can't you just "compile" your Python code into .pyc files?

Every Python script before running is compiled to .pyc file so this is a noop (unless you frequently remove the pyc files and rerun your script, which would be wasteless).

And, of course, since they are compiled, the original source is not on view. Thus, obfuscation.

I assume you never tried reverse engineering Python and native apps, but I assure you that Python reverse enginnering is several times easier. There are some really accurate decompilers for Python.


Now, the real question is, what does the OP want the decompilation for. Unless (s)he responds, it's a dead topic.

[GlowInTheDark]

Every Python script before running is compiled to .pyc file so this is a noop (unless you frequently remove the pyc files and rerun your script, which would be wasteless).

It is *NOT* a no-op. As I said (several times, in fact), it makes it load and start executing faster because it doesn't have to compile them at runtime.

The difference is small, but definitely noticeable - and in my use-case, critical. It matters when you run a given script repeatedly; I'm guessing that your use-case is a single, long-running script.

Regarding the decompiling - sure, any nut can be cracked. You are right, of course, that we'll never really know unless/until OP posts again.

[enedil]

The .pyc files are generated automatically, they land in directory __pycache__ (unless you disable it by env variable). So if you run your code multiple times, withouth changing, the first run will generate the byte code and each subsequent will detect the existence of cache and load from there.

[hydra3333]

As always, we'd need to know why the OP wants/needs to "compile". What exactly does that word mean to him, and what is the long-range goal?

Usually, when people say "compile", they really mean "obfuscate".

No. I'm an ancient programmer. I did mean compile into an executable so that no interpreters were invoked at runtime, eg hd-idle needs to be "low touch". If it were viable, and I did try to do a port of some kind, it'd be on github.

I'm not so sure about included python libraries, I suppose the compilation process takes care of that.

Now ... on to read the other posts !

edit: many thanks to everyone who responded.
Regarding de-compilation, no that's not on my agenda.
As someone mentioned, I wasn't comfortable with Go; Python is comparatively "easy" to my eyes.
Although, by the look of the responses I suppose I should just make an effort and learn Go.
At my age: so many languages, so little time If it were in fortran-77 or basic or PDP/Vax assembler ...

[GlowInTheDark]

The .pyc files are generated automatically, they land in directory __pycache__ (unless you disable it by env variable). So if you run your code multiple times, withouth changing, the first run will generate the byte code and each subsequent will detect the existence of cache and load from there.

Nope.

[GlowInTheDark]

Usually, when people say "compile", they really mean "obfuscate".

No...

First, point of order: I said "Usually, when ..."; if indeed your interest is no in obfuscation, it doesn't change the fact that I am still correct in saying that "Usually". I.e., an exception does not disprove a "Usually" type statement.

Anyway, it sounds like you are, indeed, interested in a performance gain. The problem, of course, is all the libraries, which still need to be present at runtime and within which most of the execution time will be spent. In real life code (as opposed to benchmarks), even if the code that you write gets converted to C and runs faster, you still have the libraries.

[hippy]

The .pyc files are generated automatically, they land in directory __pycache__ (unless you disable it by env variable). So if you run your code multiple times, withouth changing, the first run will generate the byte code and each subsequent will detect the existence of cache and load from there.

Nope.

Could you explain how 'enedil' is wrong in what they wrote ?

It's pretty much what I would have written myself. Python 2.7 created .pyc files in the same directory as the .py files, where Python 3 uses a __pycache__ sub-directory by default but that's all I'd add.

[hippy]

Usually

Evidence-free arguments are usually entertaining

[paddyg]

...returning to the OPs question and later clarification! I would suggest that for the kind of application they're looking at (monitoring hard disks in the background) uncompiled python would be fine. The overhead is tiny if the application is dormant most of the time. IMO it's not worth thinking about trying to "lighten the touch" until you've written it and found it needs to be improved... didn't Donald Knuth say something about this years ago?

[hydra3333]

...returning to the OPs question and later clarification! I would suggest that for the kind of application they're looking at (monitoring hard disks in the background) uncompiled python would be fine. The overhead is tiny if the application is dormant most of the time. IMO it's not worth thinking about trying to "lighten the touch" until you've written it and found it needs to be improved... didn't Donald Knuth say something about this years ago?

Thank you.

Whilst we did use an early LaTex for our documentation for a while (and thought we were made) we weren't big on Knuth.

It was mainly to be an interesting exercise for me for my Pi4s running a web based chromecast server.
Light-touch programs are generally a good thing to do for code not frequently changed and/or which touches on performance related stuff ... learned that the hard way, early on.

Having said that, I just realised how much vbscript code I have hanging around here and there on my PCs

I may still do a port after having a go at learning go, but go seems to work OK to build stuff on Pi's.

Good Health to you all.

[satineeraj]

Yes you can compile the Python Code through the compilers check best Python Compilers here:http://www.techgeekbuzz.com/best-python-compiler/

[enedil]

Yes you can compile the Python Code through the compilers check best Python Compilers here:http://www.techgeekbuzz.com/best-python-compiler/

but they all come with some caveats, that were already mentioned