Page 1 of 2

What can be done about Python 3.5 slowness?

Posted: Tue Oct 17, 2017 1:58 am
by SlowBro
I'm using Python 3.5 in Stretch Lite on a B+ with a 32GB Samsung Evo+ SD and it's slowwwwwww. More than three times slower than 2.7.13 when doing a simple pass statement. What can be done about this?

Found some posts (this and this) discussing the slowness with no apparent resolution. I see from this Python mailing list discussion that 3.6 is up to 54% slower... which I would more readily accept. I'm seeing 348% slower. Though on my Windows desktop with Cygwin, 3.6 is faster?

Code: Select all

root@raspberrypi:~# time python2.7 -c pass

real    0m0.242s
user    0m0.200s
sys     0m0.030s
root@raspberrypi:~# time python3.5 -c pass

real    0m0.843s   <-- 348% slower
user    0m0.710s
sys     0m0.090s
root@raspberrypi:~#

Can it be compiled to bytecode? I tried py_compile... which made it worse.

Code: Select all

root@raspberrypi:~# time ./pass.py

real    0m0.762s
user    0m0.720s
sys     0m0.030s
root@raspberrypi:~# python3.5
Python 3.5.3 (default, Jan 19 2017, 14:11:04)
[GCC 6.3.0 20170124] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import py_compile
>>> py_compile.compile('pass.py')
'__pycache__/pass.cpython-35.pyc'
>>>
root@raspberrypi:~# time ./pass.py

real    0m0.791s
user    0m0.730s
sys     0m0.040s
root@raspberrypi:~#

Re: What can be done about Python 3.5 slowness?

Posted: Tue Oct 17, 2017 2:31 am
by Imperf3kt
Have you considered Cython?
http://cython.org/

Re: What can be done about Python 3.5 slowness?

Posted: Tue Oct 17, 2017 2:43 am
by SlowBro
Imperf3kt wrote:
Tue Oct 17, 2017 2:31 am
Have you considered Cython?
http://cython.org/
Hmmmmmmmmmm. So I tried to compile the following file, and it's even worse. However, my actual code may be faster. I'll try it later.

Code: Select all

root@raspberrypi:~# cat pass.py
#!/usr/bin/env python3.5
pass
root@raspberrypi:~# cython3 --embed pass.py -o pass.c; gcc $(pkg-config --libs --cflags python3) pass.c -o pass
root@raspberrypi:~# time python3.5 -m pass

real    0m0.916s
user    0m0.830s
sys     0m0.070s
root@raspberrypi:~#

Re: What can be done about Python 3.5 slowness?

Posted: Tue Oct 17, 2017 3:01 am
by SlowBro
Oh. Perhaps it's not loading the file in the current directory.

Code: Select all

root@raspberrypi:~# mv pass pass.old
root@raspberrypi:~# time python3.5 -m pass

real    0m0.935s
user    0m0.880s
sys     0m0.040s
root@raspberrypi:~#
Edit: No, it is.

Code: Select all

root@raspberrypi:~# cython3 --embed pass_test.py -o pass_test.c; gcc $(pkg-config --libs --cflags python3) pass.c -o pass_test
root@raspberrypi:~# time python3.5 -m pass_test

real    0m1.016s
user    0m0.870s
sys     0m0.120s
root@raspberrypi:~#

Re: What can be done about Python 3.5 slowness?

Posted: Tue Oct 17, 2017 9:55 am
by jamesh
Testing with just a 'pass' statement isn't particularly valid. You need some actual code in there, otherwise all you are measuring is the startup/shutdown time of the interpreter/program/whatever. And I would expect that to increase slightly with each version, since they are generally more complex each time, and will require more work to get going.

Re: What can be done about Python 3.5 slowness?

Posted: Tue Oct 17, 2017 10:26 am
by SlowBro
That’s why I tested the same on Windows as mentioned earlier. It’s actually faster there. But fair enough, I will test a slow part of my actual code to see if it makes any difference with 2.7.

I am very interested in Cython though. It promises dramatic increases that go beyond downgrading. Will seek to benchmark all three.

How about you? What are your times when running code under either interpreter?

Re: What can be done about Python 3.5 slowness?

Posted: Tue Oct 17, 2017 10:34 am
by jamesh
SlowBro wrote:
Tue Oct 17, 2017 10:26 am
That’s why I tested the same on Windows as mentioned earlier. It’s actually faster there. But fair enough, I will test a slow part of my actual code to see if it makes any difference with 2.7.

I am very interested in Cython though. It promises dramatic increases that go beyond downgrading. Will seek to benchmark all three.

How about you? What are your times when running code under either interpreter?
Sorry, don't generally use Python. C and C++ mostly due to working on the VC4 firmware and Linux kernel.

Re: What can be done about Python 3.5 slowness?

Posted: Tue Oct 17, 2017 10:57 am
by jahboater
SlowBro wrote:
Tue Oct 17, 2017 10:26 am
That’s why I tested the same on Windows as mentioned earlier. It’s actually faster there.
Thats probably because the Windows computer has a faster x86 processor, more memory, and faster disks. I thing you should only compare things on identical machines with identical settings.
SlowBro wrote:
Tue Oct 17, 2017 10:26 am
I will test a slow part of my actual code to see if it makes any difference with 2.7.
Definately, I agree with JamesH. Try to find some representative code that takes a significant amount of time to execute.

Re: What can be done about Python 3.5 slowness?

Posted: Tue Oct 17, 2017 11:01 am
by hippy
SlowBro wrote:
Tue Oct 17, 2017 1:58 am
I'm using Python 3.5 in Stretch Lite on a B+ with a 32GB Samsung Evo+ SD and it's slowwwwwww. More than three times slower than 2.7.13 when doing a simple pass statement. What can be done about this?
Not a lot. The general consensus is that Python 3.x is slower than Python 2.7 and that is unlikely to change unless Python developers or other contributors find some way to improve its performance.

You can look at using cython and other options but it may simply be easier to target your Python code at 2.7, run it with 2.7, while ensuring 3.x compatibility for others. It then becomes user choice whether they use the faster 2.7 or slower 3.x to run it. That's the approach I have taken.

Re: What can be done about Python 3.5 slowness?

Posted: Tue Oct 17, 2017 12:03 pm
by jahboater
I suppose there is a reason why the OP is using an old B+?

Changing to a Pi3 (around 10x faster) and using Python 3 would be my choice.

Re: What can be done about Python 3.5 slowness?

Posted: Tue Oct 17, 2017 1:35 pm
by Paeryn
SlowBro wrote:
Tue Oct 17, 2017 3:01 am

Code: Select all

root@raspberrypi:~# cat pass.py
#!/usr/bin/env python3.5
pass
root@raspberrypi:~# cython3 --embed pass.py -o pass.c; gcc $(pkg-config --libs --cflags python3) pass.c -o pass
root@raspberrypi:~# time python3.5 -m pass

real    0m0.916s
user    0m0.830s
sys     0m0.070s
root@raspberrypi:~#
Oh. Perhaps it's not loading the file in the current directory.

Code: Select all

root@raspberrypi:~# mv pass pass.old
root@raspberrypi:~# time python3.5 -m pass

real    0m0.935s
user    0m0.880s
sys     0m0.040s
root@raspberrypi:~#
Edit: No, it is.

Code: Select all

root@raspberrypi:~# cython3 --embed pass_test.py -o pass_test.c; gcc $(pkg-config --libs --cflags python3) pass.c -o pass_test
root@raspberrypi:~# time python3.5 -m pass_test

real    0m1.016s
user    0m0.870s
sys     0m0.120s
root@raspberrypi:~#
You are compiling your test program with cython but then go on to time Python running the original script, at no point are you timing the running of the compiled version (though in such a simple case there isn't going to be much difference since the overheads of starting the environment far outweigh the time spent running your program). In fact in the last attempt you have cython generating a C file pass_test.c but then go on to compile pass.c instead. To time the compiled version you want

Code: Select all

time ./pass_test
Is there any particular reason you are doing all this whilst logged in as root rather than as a standard user? Whilst you are free to do that if you want, it's generally not wise unless you really have to as it is all too easy to accidentally mess the system up.

Re: What can be done about Python 3.5 slowness?

Posted: Tue Oct 17, 2017 2:21 pm
by paddyg
It does depend critically on what your program is doing as well as what you're running it on. I've found that a few programs on RPi run faster on 3.5 that 2.7 and some faster with pypy. Generally writing a module in C, cython or roping something in using ctypes is quite a bit faster than vanilla python.. BUT If speed is an issue it's worth refactoring your code, if at all possible, to use a ready made optimised module. In my experience this will be miles faster. (To write as well as to run!)

I'm curious to know what your program is doing that runs so much slower on python 3.5

Re: What can be done about Python 3.5 slowness?

Posted: Tue Oct 17, 2017 3:06 pm
by SlowBro
jahboater wrote:
Tue Oct 17, 2017 10:57 am
SlowBro wrote:
Tue Oct 17, 2017 10:26 am
That’s why I tested the same on Windows as mentioned earlier. It’s actually faster there.
Thats probably because the Windows computer has a faster x86 processor, more memory, and faster disks.
No no, I mean Python 3.6 is faster on Windows than 2.7, negating the idea that it's the newer version causing the problem. If the cause is the newer version then 3.6 would be slower even on Windows.

In other words, 2.7 ran in a certain amount of time on Cygwin, and 3.6 ran in less time. (Sorry I don't have the times in front of me at the moment.) This is the opposite behavior observed on the Pi.

Re: What can be done about Python 3.5 slowness?

Posted: Tue Oct 17, 2017 3:08 pm
by SlowBro
Paeryn wrote:
Tue Oct 17, 2017 1:35 pm
You are compiling your test program with cython but then go on to time Python running the original script, at no point are you timing the running of the compiled version
Ahh my mistake, I forgot I also had renamed the source .py, will try it as you said.
Paeryn wrote:
Tue Oct 17, 2017 1:35 pm
Is there any particular reason you are doing all this whilst logged in as root rather than as a standard user?
Long story but I definitely am aware of the risks; thanks for looking out though.

Re: What can be done about Python 3.5 slowness?

Posted: Tue Oct 17, 2017 3:14 pm
by SlowBro
paddyg wrote:
Tue Oct 17, 2017 2:21 pm
I'm curious to know what your program is doing that runs so much slower on python 3.5
It's several CGI scripts but it's also very slow on the command line, not even in the web server yet. Before I sought to profile and optimize it I thought I should try a simple pass command and lo, it too is far slower.

At your suggestion I'm going to try the four different methods (2.7, 3.5, PyPy, Cython) and see which works best for my situation. Might be a combination of the four.

Alright thanks all!

Re: What can be done about Python 3.5 slowness?

Posted: Tue Oct 17, 2017 4:37 pm
by bensimmo
Is it possible to preload the environment etc. And just test the code itself.
Of course that would depend on how you'll be using it, if the setup is actually a problem.

Re: What can be done about Python 3.5 slowness?

Posted: Tue Oct 17, 2017 4:44 pm
by SlowBro
bensimmo wrote:
Tue Oct 17, 2017 4:37 pm
Is it possible to preload the environment etc. And just test the code itself.
Of course that would depend on how you'll be using it, if the setup is actually a problem.
Can you give an example?

The web app I'm building is mostly for managing the Wi-Fi config, the service account, and reviewing error logs. I can preload a list of access points and have it refresh periodically in the background, but I don't think there is much more that can be preloaded.

Re: What can be done about Python 3.5 slowness?

Posted: Tue Oct 17, 2017 5:07 pm
by jahboater
Just running it twice should preload much of the stuff - it will remain in the disk cache.

Re: What can be done about Python 3.5 slowness?

Posted: Tue Oct 17, 2017 5:25 pm
by MarkR
Yes, Python3 startup time is worse than Python2. Deal with it. You're getting new features.

If startup time is such a big problem, refactor your code so it starts the interpreter less often (e.g. do more work in a single run).

Re: What can be done about Python 3.5 slowness?

Posted: Tue Oct 17, 2017 5:50 pm
by bensimmo
SlowBro wrote:
Tue Oct 17, 2017 4:44 pm
bensimmo wrote:
Tue Oct 17, 2017 4:37 pm
Is it possible to preload the environment etc. And just test the code itself.
Of course that would depend on how you'll be using it, if the setup is actually a problem.
Can you give an example?

The web app I'm building is mostly for managing the Wi-Fi config, the service account, and reviewing error logs. I can preload a list of access points and have it refresh periodically in the background, but I don't think there is much more that can be preloaded.
No that's why I was asking.
It seems a bit silly to have everything load on demand if you call it all the time and it has to re-setup and not just sit there idle waiting for the next python code to run.
Use the memory and gain time.

Basically what I'm saying is once you turned the thing on everything should be ready to use if you trigger some more code.
(I hate putting thing in draws if I'm going to be using it again soon, unlike who I work with who wastes time, putting it away and getting it out the next-day or so.
Of course I pack away as needed if the space is needed. Looks a bit messier all the time though.)

Why should you worry about initialisation etc.

I don't know python3 well enough to know if you can alter its priorities.

Re: What can be done about Python 3.5 slowness?

Posted: Tue Oct 17, 2017 5:59 pm
by SlowBro
Would anyone be so kind as to post their real world stats? time for various scripts when run on both 2.7 and 3.5.

Re: What can be done about Python 3.5 slowness?

Posted: Tue Oct 17, 2017 6:07 pm
by SlowBro
jahboater wrote:
Tue Oct 17, 2017 5:07 pm
Just running it twice should preload much of the stuff - it will remain in the disk cache.
Not in my experience. Just as slow the first and subsequent runs.

Re: What can be done about Python 3.5 slowness?

Posted: Tue Oct 17, 2017 6:09 pm
by SlowBro
MarkR wrote:
Tue Oct 17, 2017 5:25 pm
Yes, Python3 startup time is worse than Python2. Deal with it. You're getting new features.
Not on other environments. When running in Cygwin the situation is completely backwards; 3.6 is faster than 2.7. Per the initial testing it's 3x slower on Pi -- but that may be a fault with the way I am testing. More testing on my part will be needed. What stats do you see? Kindly share some time benchmarks from one of your scripts under both interpreters.

Re: What can be done about Python 3.5 slowness?

Posted: Tue Oct 17, 2017 6:10 pm
by SlowBro
bensimmo wrote:
Tue Oct 17, 2017 5:50 pm
It seems a bit silly to have everything load on demand if you call it all the time and it has to re-setup and not just sit there idle waiting for the next python code to run.
Use the memory and gain time.
I can't think how to do that. Examples would be wonderful.

Re: What can be done about Python 3.5 slowness?

Posted: Tue Oct 17, 2017 7:37 pm
by paddyg
Well, if you host your webpages using http.server.CGIHTTPRequestHandler then the python interpreter will be loaded and running already. What kind of thing will your CGI scripts be doing?