User avatar
SlowBro
Posts: 138
Joined: Sat Feb 18, 2017 1:30 am

Re: What can be done about Python 3.5 slowness?

Tue Oct 17, 2017 8:12 pm

I'm not using http.server; I would use lighttpd but even on the command line (no web server involved) it's a dog.

The CGI app I'm building is mostly for managing the Wi-Fi config, the service account, and reviewing error logs.

User avatar
SlowBro
Posts: 138
Joined: Sat Feb 18, 2017 1:30 am

Re: What can be done about Python 3.5 slowness?

Wed Oct 18, 2017 12:33 am

SlowBro wrote:
Tue Oct 17, 2017 3:08 pm
Paeryn wrote:
Tue Oct 17, 2017 1:35 pm
You are compiling your test program with cython but then go on to time Python running the original script, at no point are you timing the running of the compiled version
Ahh my mistake, I forgot I also had renamed the source .py, will try it as you said.
Same difference.

Code: Select all

[email protected]:~# time ./pass_test

real    0m0.817s
user    0m0.730s
sys     0m0.080s
[email protected]:~#

User avatar
SlowBro
Posts: 138
Joined: Sat Feb 18, 2017 1:30 am

Re: What can be done about Python 3.5 slowness?

Wed Oct 18, 2017 1:33 am

OK so when running some more complex code the 3.5 interpreter is only 15% slower. This code uses gpiozero which itself takes about 8 seconds to start.

Now I will try PyPy and Cython.

Each test was run twice to allow for caching.

2.7 interpreter:

Code: Select all

[email protected]spberrypi:/SB# time tests/test_leds.py
Starting tests/test_leds.py
[SUCCESS] Set the good LED
[SUCCESS] Back to the default
[SUCCESS] Set the warn LED
[SUCCESS] Set the err LED
[SUCCESS] Back to the default
[SUCCESS] All off

real    0m7.920s
user    0m7.570s
sys     0m0.270s
[email protected]:/SB#
3.5 interpreter:

Code: Select all

[email protected]:/SmartBird# time tests/test_leds.py
Starting tests/test_leds.py
[SUCCESS] Set the good LED
[SUCCESS] Back to the default
[SUCCESS] Set the warn LED
[SUCCESS] Set the err LED
[SUCCESS] Back to the default
[SUCCESS] All off

real    0m9.152s
user    0m8.850s
sys     0m0.210s
[email protected]:/SB#

jahboater
Posts: 2721
Joined: Wed Feb 04, 2015 6:38 pm

Re: What can be done about Python 3.5 slowness?

Wed Oct 18, 2017 7:37 am

Have you got sleep() or delays in that code? 8-9 seconds seems a ridiculously long time to turn on and off a couple of led's.

User avatar
bensimmo
Posts: 2903
Joined: Sun Dec 28, 2014 3:02 pm
Location: East Yorkshire

Re: What can be done about Python 3.5 slowness?

Wed Oct 18, 2017 7:58 am

I don't remember gpiozero taking that long to just setup it led module part. It is however designed to make life easy.

If it is, bypass it and move to just pigpio direct commands.

Post the code so others can see (or help speed it up)

User avatar
paddyg
Posts: 2123
Joined: Sat Jan 28, 2012 11:57 am
Location: UK

Re: What can be done about Python 3.5 slowness?

Wed Oct 18, 2017 8:38 am

I'm not convinced that your search for a brand of python that is a bit quicker to start will be anywhere near as productive as finding a way for python to stay running. Even if your main web server isn't python based, maybe you could run a python cgi server locally and communicate with your main server using http requests, or have your python app running and communicate with your web server using sockets. https://redmine.lighttpd.net/projects/l ... PythonWSGI It also seems possible that lighttpd has functionality to keep python running rather than stop and restart it on every request. At this stage I think I would concentrate on the server end and do the python tuning later.

PS out of curiosity I ran the following on my RPi3

Code: Select all

import time
tm = time.time()
with open('junk.txt', 'w') as f:
  for i in range(1000000):
    f.write('{:07d}'.format(i))
print(time.time() - tm)
import numpy as np
from PIL import Image
tm = time.time()
im = np.array(Image.open('textures/straw1.jpg'))
im = np.array([255, 255, 255]) - im[:,:]
Image.fromarray(im.astype(np.uint8)).save('junk.jpg')
print(time.time() - tm)
[email protected]:~/pi3d_demos $ time python3 temp.py
10.550017833709717
0.27675461769104004
real 0m12.221s

[email protected]:~/pi3d_demos $ time python2 temp.py
8.8112680912
0.300981998444
real 0m9.767s

[email protected]rrypi:~/pi3d_demos $ time pypy temp.py
3.19162988663
1.65918898582
real 0m6.355s

i.e. for something that is very much just a for loop python2 is quite a bit faster than python3 (because of representation of integers which you probably already know) but pypy is much faster again. But pypy and python3 take quite a bit longer to load on start up, and to load modules. If you do the timing over just the code execution python3 is often faster than python2 and pypy. It depends on what your code is doing.
also https://groups.google.com/forum/?hl=en-GB&fromgroups=#!forum/pi3d

jahboater
Posts: 2721
Joined: Wed Feb 04, 2015 6:38 pm

Re: What can be done about Python 3.5 slowness?

Wed Oct 18, 2017 9:17 am

For amusement I counted the machine instructions needed to execute a trivial Python program
a = 1
b = 1
c = a + b

Python 2
Total instructions: 23834029

Python 3
Total instructions: 47543516

C
Total instructions: 109117

Assembler
Total instructions: 5

C++
Total instructions: 3045025
Last edited by jahboater on Wed Oct 18, 2017 10:56 am, edited 2 times in total.

hippy
Posts: 3166
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: What can be done about Python 3.5 slowness?

Wed Oct 18, 2017 10:19 am

jahboater wrote:
Wed Oct 18, 2017 9:17 am
Python 2 - Total instructions: 23834029
Python 3 - Total instructions: 47543516
That seems to fit with the suggestion that Python 3 is simply doing more than Python 2 and doing things differently. A reflection of how the code has been refactored and developed, the addition of checks or changes, how data is accessed and handled, even small changes adding up.
jahboater wrote:
Wed Oct 18, 2017 9:17 am
C - Total instructions: 109117
And presumably assembler would be quite a bit less.

User avatar
SlowBro
Posts: 138
Joined: Sat Feb 18, 2017 1:30 am

Re: What can be done about Python 3.5 slowness?

Wed Oct 18, 2017 10:23 am

jahboater wrote:
Wed Oct 18, 2017 7:37 am
8-9 seconds seems a ridiculously long time to turn on and off a couple of led's.
Yeah, I thought so as well. Here's some stripped-bare code. LOL Cython is actually the slowest. Leads me to conclude that the solution will be a combination of interpreters.

Code: Select all

[email protected]:~# cat test_gpiozero.py
from gpiozero import LED
led = LED(17)
led.on()
[email protected]:~# time python2.7 test_gpiozero.py

real    0m7.705s
user    0m7.420s
sys     0m0.250s
[email protected]:~# time python3.5 test_gpiozero.py

real    0m8.999s
user    0m8.500s
sys     0m0.400s
[email protected]:~# cython3 --embed test_gpiozero.py -o test_gpiozero.c; gcc $(pkg-config --libs --cflags python3) test_gpiozero.c -o test_gpiozero
[email protected]:~# time ./test_gpiozero

real    0m9.655s
user    0m9.370s
sys     0m0.220s
[email protected]:~#

User avatar
SlowBro
Posts: 138
Joined: Sat Feb 18, 2017 1:30 am

Re: What can be done about Python 3.5 slowness?

Wed Oct 18, 2017 10:31 am

bensimmo wrote:
Wed Oct 18, 2017 7:58 am
I don't remember gpiozero taking that long to just setup it led module part. It is however designed to make life easy.

If it is, bypass it and move to just pigpio direct commands.

Post the code so others can see (or help speed it up)
You are correct, gpiozero is the pig. I'm going to switch. And again, Cython is the loser.

Code: Select all

[email protected]:~# cat test_pigpio.py
import pigpio
pi = pigpio.pi()
pi.set_mode(17, pigpio.OUTPUT)
pi.write(17, 1)
[email protected]:~# time python2.7 test_pigpio.py

real    0m0.595s
user    0m0.420s
sys     0m0.090s
[email protected]:~# time python3.5 test_pigpio.py

real    0m1.719s
user    0m1.320s
sys     0m0.130s
[email protected]:~# cython3 --embed test_pigpio.py -o test_pigpio.c; gcc $(pkg-config --libs --cflags python3) test_pigpio.c -o test_pigpio
[email protected]:~# time ./test_pigpio

real    0m1.785s
user    0m1.460s
sys     0m0.060s
[email protected]:~#

jahboater
Posts: 2721
Joined: Wed Feb 04, 2015 6:38 pm

Re: What can be done about Python 3.5 slowness?

Wed Oct 18, 2017 10:49 am

hippy wrote:
Wed Oct 18, 2017 10:19 am
And presumably assembler would be quite a bit less.
Yes - 5 instructions!

The C compiler did of course remove the entire program (because it does nothing).
That is something an interpreter can never do.

C probably has to load libc, zero the bss section?, run its startup code, then call main (the program itself is only 2 instructions, one of which is not executed).
C++ is 3045025 so who knows what that is doing.
Last edited by jahboater on Wed Oct 18, 2017 11:19 am, edited 3 times in total.

User avatar
SlowBro
Posts: 138
Joined: Sat Feb 18, 2017 1:30 am

Re: What can be done about Python 3.5 slowness?

Wed Oct 18, 2017 10:56 am

paddyg wrote:
Wed Oct 18, 2017 8:38 am
I'm not convinced that your search for a brand of python that is a bit quicker to start will be anywhere near as productive as finding a way for python to stay running. (..) At this stage I think I would concentrate on the server end and do the python tuning later.
That's sound advice. I think what I'll do is drop gpiozero and either launch a pure http.server Python script or follow the advice in that link you shared, finding a way to ensure it is always running from boot.

Question 1: How would I pre-cache all my modules into memory at startup? Should I just import all from the main script or is there another way?
paddyg wrote:
Wed Oct 18, 2017 8:38 am
PS out of curiosity I ran the following on my RPi3
Same test on my B+. I didn't have your straw1.jpg so I used a suitably large picture of straw from Google images.

Couldn't figure out how to load the necessary modules into PyPy so here's 2.7, 3.5, and Cython. Again, 2.7 is the winner by far.

Question 2: How do I import the necessary modules into PyPy?

Code: Select all

[email protected]:~# time python2.7 paddyg_test.py
43.0744729042
19.3129739761

real    1m4.493s
user    0m53.290s
sys     0m3.250s
[email protected]:~# time python3.5 paddyg_test.py
65.92904710769653
19.037921667099

real    1m30.641s
user    1m15.060s
sys     0m4.170s
[email protected]:~# export test_file=paddyg_test; cython3 --embed ${test_file}.py -o ${test_file}.c; gcc $(pkg-config --libs --cflags python3) ${test_file}.c -o ${test_file}
[email protected]:~# time ./${test_file}
79.019864320755
18.89829993247986

real    1m43.753s
user    1m26.570s
sys     0m4.130s
[email protected]:~#
Last edited by SlowBro on Wed Oct 18, 2017 11:27 am, edited 1 time in total.

User avatar
SlowBro
Posts: 138
Joined: Sat Feb 18, 2017 1:30 am

Re: What can be done about Python 3.5 slowness?

Wed Oct 18, 2017 11:02 am

jahboater wrote:
Wed Oct 18, 2017 9:17 am
For amusement I counted the machine instructions needed to execute a trivial Python program
a = 1
b = 1
c = a + b
My time results for your test. 2.7 wins by far. PyPy is the slowest.

Code: Select all

[email protected]:~# time python2.7 jahboater_test.py

real    0m0.351s
user    0m0.220s
sys     0m0.070s
[email protected]:~# time python3.5 jahboater_test.py

real    0m1.026s
user    0m0.790s
sys     0m0.090s
[email protected]:~# export test_file=jahboater_test; cython3 --embed ${test_file}.py -o ${test_file}.c; gcc $(pkg-config --libs --cflags python3) ${test_file}.c -o ${test_file}
[email protected]:~# time ./${test_file}                                                                    
real    0m1.079s
user    0m0.850s
sys     0m0.060s
[email protected]:~# time pypy jahboater_test.py                                                            
real    0m1.081s
user    0m0.740s
sys     0m0.180s
[email protected]:~#

User avatar
SlowBro
Posts: 138
Joined: Sat Feb 18, 2017 1:30 am

Re: What can be done about Python 3.5 slowness?

Wed Oct 18, 2017 11:32 am

Rpi.GPIO is even faster than pigpio.

Code: Select all

[email protected]:~# time python2.7 test_pigpio.py

real    0m0.601s
user    0m0.420s
sys     0m0.080s
[email protected]:~# time python2.7 test_rpi.gpio.py

real    0m0.345s
user    0m0.240s
sys     0m0.040s
[email protected]:~#

User avatar
paddyg
Posts: 2123
Joined: Sat Jan 28, 2012 11:57 am
Location: UK

Re: What can be done about Python 3.5 slowness?

Wed Oct 18, 2017 12:12 pm

Python will only do the import once for each instance of python that's running. The issue is getting just one python process running and staying running. I've just done a quick experiment with a python cgi server and got it to serve several pages running a python cgi and, sure enough, just one instance of python is visible in top (running 100% as the cgi is basically for i in range(200000000): b = i ** 0.5) I think this is the way you should go. Info on the page I linked to before about how to serve cgi info from a python server.

Easiest way to install modules for pypy is to install pip for pypy see http://pi3d.github.io/html/FAQ.html#pypy

PS with the python cgi server experiment I had sudden misgivings about my assertion so I then tried adding
tm = time.time()
import numpy
imptm = time.time() - tm
and the first cgi page to be requested gave a time of 1.28s but all subsequent requests give 0.10s to 0.15s
Last edited by paddyg on Wed Oct 18, 2017 1:07 pm, edited 1 time in total.
also https://groups.google.com/forum/?hl=en-GB&fromgroups=#!forum/pi3d

User avatar
SlowBro
Posts: 138
Joined: Sat Feb 18, 2017 1:30 am

Re: What can be done about Python 3.5 slowness?

Wed Oct 18, 2017 12:56 pm

By switching to a pure http.server and refactoring some code in the background (switched to pigpio for one) the device admin home page loads in about seven seconds under Py 3.5, which is about half the time from before. I'm going to test it under PyPy and 2.7 but this is looking good.

User avatar
paddyg
Posts: 2123
Joined: Sat Jan 28, 2012 11:57 am
Location: UK

Re: What can be done about Python 3.5 slowness?

Wed Oct 18, 2017 1:14 pm

How long does the admin home page take for subsequent loads (after the first time after the server has started)?
also https://groups.google.com/forum/?hl=en-GB&fromgroups=#!forum/pi3d

User avatar
SlowBro
Posts: 138
Joined: Sat Feb 18, 2017 1:30 am

Re: What can be done about Python 3.5 slowness?

Wed Oct 18, 2017 2:38 pm

paddyg wrote:
Wed Oct 18, 2017 1:14 pm
How long does the admin home page take for subsequent loads (after the first time after the server has started)?
Don't have my Pi in front of me right now, but if I recall reloading the page shaved off about two seconds, so from seven to five. So maybe I ought to import my home page (and sub-pages) with the server start, wouldn't you agree?

User avatar
SlowBro
Posts: 138
Joined: Sat Feb 18, 2017 1:30 am

Re: What can be done about Python 3.5 slowness?

Thu Oct 19, 2017 1:05 am

No, it's consistently about 7 seconds to load no matter how many times I refresh.

Return to “Python”

Who is online

Users browsing this forum: No registered users and 22 guests