User avatar
SlowBro
Posts: 93
Joined: Sat Feb 18, 2017 1:30 am

Re: What can be done about Python 3.5 slowness?

Tue Oct 17, 2017 8:12 pm

I'm not using http.server; I would use lighttpd but even on the command line (no web server involved) it's a dog.

The CGI app I'm building is mostly for managing the Wi-Fi config, the service account, and reviewing error logs.

User avatar
SlowBro
Posts: 93
Joined: Sat Feb 18, 2017 1:30 am

Re: What can be done about Python 3.5 slowness?

Wed Oct 18, 2017 12:33 am

SlowBro wrote:
Tue Oct 17, 2017 3:08 pm
Paeryn wrote:
Tue Oct 17, 2017 1:35 pm
You are compiling your test program with cython but then go on to time Python running the original script, at no point are you timing the running of the compiled version
Ahh my mistake, I forgot I also had renamed the source .py, will try it as you said.
Same difference.

Code: Select all

root@raspberrypi:~# time ./pass_test

real    0m0.817s
user    0m0.730s
sys     0m0.080s
root@raspberrypi:~#

User avatar
SlowBro
Posts: 93
Joined: Sat Feb 18, 2017 1:30 am

Re: What can be done about Python 3.5 slowness?

Wed Oct 18, 2017 1:33 am

OK so when running some more complex code the 3.5 interpreter is only 15% slower. This code uses gpiozero which itself takes about 8 seconds to start.

Now I will try PyPy and Cython.

Each test was run twice to allow for caching.

2.7 interpreter:

Code: Select all

root@raspberrypi:/SB# time tests/test_leds.py
Starting tests/test_leds.py
[SUCCESS] Set the good LED
[SUCCESS] Back to the default
[SUCCESS] Set the warn LED
[SUCCESS] Set the err LED
[SUCCESS] Back to the default
[SUCCESS] All off

real    0m7.920s
user    0m7.570s
sys     0m0.270s
root@raspberrypi:/SB#
3.5 interpreter:

Code: Select all

root@raspberrypi:/SmartBird# time tests/test_leds.py
Starting tests/test_leds.py
[SUCCESS] Set the good LED
[SUCCESS] Back to the default
[SUCCESS] Set the warn LED
[SUCCESS] Set the err LED
[SUCCESS] Back to the default
[SUCCESS] All off

real    0m9.152s
user    0m8.850s
sys     0m0.210s
root@raspberrypi:/SB#

jahboater
Posts: 1873
Joined: Wed Feb 04, 2015 6:38 pm

Re: What can be done about Python 3.5 slowness?

Wed Oct 18, 2017 7:37 am

Have you got sleep() or delays in that code? 8-9 seconds seems a ridiculously long time to turn on and off a couple of led's.

bensimmo
Posts: 1824
Joined: Sun Dec 28, 2014 3:02 pm
Location: East Yorkshire

Re: What can be done about Python 3.5 slowness?

Wed Oct 18, 2017 7:58 am

I don't remember gpiozero taking that long to just setup it led module part. It is however designed to make life easy.

If it is, bypass it and move to just pigpio direct commands.

Post the code so others can see (or help speed it up)

User avatar
paddyg
Posts: 2002
Joined: Sat Jan 28, 2012 11:57 am
Location: UK

Re: What can be done about Python 3.5 slowness?

Wed Oct 18, 2017 8:38 am

I'm not convinced that your search for a brand of python that is a bit quicker to start will be anywhere near as productive as finding a way for python to stay running. Even if your main web server isn't python based, maybe you could run a python cgi server locally and communicate with your main server using http requests, or have your python app running and communicate with your web server using sockets. https://redmine.lighttpd.net/projects/l ... PythonWSGI It also seems possible that lighttpd has functionality to keep python running rather than stop and restart it on every request. At this stage I think I would concentrate on the server end and do the python tuning later.

PS out of curiosity I ran the following on my RPi3

Code: Select all

import time
tm = time.time()
with open('junk.txt', 'w') as f:
  for i in range(1000000):
    f.write('{:07d}'.format(i))
print(time.time() - tm)
import numpy as np
from PIL import Image
tm = time.time()
im = np.array(Image.open('textures/straw1.jpg'))
im = np.array([255, 255, 255]) - im[:,:]
Image.fromarray(im.astype(np.uint8)).save('junk.jpg')
print(time.time() - tm)
pi@raspberrypi:~/pi3d_demos $ time python3 temp.py
10.550017833709717
0.27675461769104004
real 0m12.221s

pi@raspberrypi:~/pi3d_demos $ time python2 temp.py
8.8112680912
0.300981998444
real 0m9.767s

pi@raspberrypi:~/pi3d_demos $ time pypy temp.py
3.19162988663
1.65918898582
real 0m6.355s

i.e. for something that is very much just a for loop python2 is quite a bit faster than python3 (because of representation of integers which you probably already know) but pypy is much faster again. But pypy and python3 take quite a bit longer to load on start up, and to load modules. If you do the timing over just the code execution python3 is often faster than python2 and pypy. It depends on what your code is doing.
also https://groups.google.com/forum/?hl=en-GB&fromgroups=#!forum/pi3d

jahboater
Posts: 1873
Joined: Wed Feb 04, 2015 6:38 pm

Re: What can be done about Python 3.5 slowness?

Wed Oct 18, 2017 9:17 am

For amusement I counted the machine instructions needed to execute a trivial Python program
a = 1
b = 1
c = a + b

Python 2
Total instructions: 23834029

Python 3
Total instructions: 47543516

C
Total instructions: 109117

Assembler
Total instructions: 5

C++
Total instructions: 3045025
Last edited by jahboater on Wed Oct 18, 2017 10:56 am, edited 2 times in total.

hippy
Posts: 2261
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: What can be done about Python 3.5 slowness?

Wed Oct 18, 2017 10:19 am

jahboater wrote:
Wed Oct 18, 2017 9:17 am
Python 2 - Total instructions: 23834029
Python 3 - Total instructions: 47543516
That seems to fit with the suggestion that Python 3 is simply doing more than Python 2 and doing things differently. A reflection of how the code has been refactored and developed, the addition of checks or changes, how data is accessed and handled, even small changes adding up.
jahboater wrote:
Wed Oct 18, 2017 9:17 am
C - Total instructions: 109117
And presumably assembler would be quite a bit less.

User avatar
SlowBro
Posts: 93
Joined: Sat Feb 18, 2017 1:30 am

Re: What can be done about Python 3.5 slowness?

Wed Oct 18, 2017 10:23 am

jahboater wrote:
Wed Oct 18, 2017 7:37 am
8-9 seconds seems a ridiculously long time to turn on and off a couple of led's.
Yeah, I thought so as well. Here's some stripped-bare code. LOL Cython is actually the slowest. Leads me to conclude that the solution will be a combination of interpreters.

Code: Select all

root@raspberrypi:~# cat test_gpiozero.py
from gpiozero import LED
led = LED(17)
led.on()
root@raspberrypi:~# time python2.7 test_gpiozero.py

real    0m7.705s
user    0m7.420s
sys     0m0.250s
root@raspberrypi:~# time python3.5 test_gpiozero.py

real    0m8.999s
user    0m8.500s
sys     0m0.400s
root@raspberrypi:~# cython3 --embed test_gpiozero.py -o test_gpiozero.c; gcc $(pkg-config --libs --cflags python3) test_gpiozero.c -o test_gpiozero
root@raspberrypi:~# time ./test_gpiozero

real    0m9.655s
user    0m9.370s
sys     0m0.220s
root@raspberrypi:~#

User avatar
SlowBro
Posts: 93
Joined: Sat Feb 18, 2017 1:30 am

Re: What can be done about Python 3.5 slowness?

Wed Oct 18, 2017 10:31 am

bensimmo wrote:
Wed Oct 18, 2017 7:58 am
I don't remember gpiozero taking that long to just setup it led module part. It is however designed to make life easy.

If it is, bypass it and move to just pigpio direct commands.

Post the code so others can see (or help speed it up)
You are correct, gpiozero is the pig. I'm going to switch. And again, Cython is the loser.

Code: Select all

root@raspberrypi:~# cat test_pigpio.py
import pigpio
pi = pigpio.pi()
pi.set_mode(17, pigpio.OUTPUT)
pi.write(17, 1)
root@raspberrypi:~# time python2.7 test_pigpio.py

real    0m0.595s
user    0m0.420s
sys     0m0.090s
root@raspberrypi:~# time python3.5 test_pigpio.py

real    0m1.719s
user    0m1.320s
sys     0m0.130s
root@raspberrypi:~# cython3 --embed test_pigpio.py -o test_pigpio.c; gcc $(pkg-config --libs --cflags python3) test_pigpio.c -o test_pigpio
root@raspberrypi:~# time ./test_pigpio

real    0m1.785s
user    0m1.460s
sys     0m0.060s
root@raspberrypi:~#

jahboater
Posts: 1873
Joined: Wed Feb 04, 2015 6:38 pm

Re: What can be done about Python 3.5 slowness?

Wed Oct 18, 2017 10:49 am

hippy wrote:
Wed Oct 18, 2017 10:19 am
And presumably assembler would be quite a bit less.
Yes - 5 instructions!

The C compiler did of course remove the entire program (because it does nothing).
That is something an interpreter can never do.

C probably has to load libc, zero the bss section?, run its startup code, then call main (the program itself is only 2 instructions, one of which is not executed).
C++ is 3045025 so who knows what that is doing.
Last edited by jahboater on Wed Oct 18, 2017 11:19 am, edited 3 times in total.

User avatar
SlowBro
Posts: 93
Joined: Sat Feb 18, 2017 1:30 am

Re: What can be done about Python 3.5 slowness?

Wed Oct 18, 2017 10:56 am

paddyg wrote:
Wed Oct 18, 2017 8:38 am
I'm not convinced that your search for a brand of python that is a bit quicker to start will be anywhere near as productive as finding a way for python to stay running. (..) At this stage I think I would concentrate on the server end and do the python tuning later.
That's sound advice. I think what I'll do is drop gpiozero and either launch a pure http.server Python script or follow the advice in that link you shared, finding a way to ensure it is always running from boot.

Question 1: How would I pre-cache all my modules into memory at startup? Should I just import all from the main script or is there another way?
paddyg wrote:
Wed Oct 18, 2017 8:38 am
PS out of curiosity I ran the following on my RPi3
Same test on my B+. I didn't have your straw1.jpg so I used a suitably large picture of straw from Google images.

Couldn't figure out how to load the necessary modules into PyPy so here's 2.7, 3.5, and Cython. Again, 2.7 is the winner by far.

Question 2: How do I import the necessary modules into PyPy?

Code: Select all

root@raspberrypi:~# time python2.7 paddyg_test.py
43.0744729042
19.3129739761

real    1m4.493s
user    0m53.290s
sys     0m3.250s
root@raspberrypi:~# time python3.5 paddyg_test.py
65.92904710769653
19.037921667099

real    1m30.641s
user    1m15.060s
sys     0m4.170s
root@raspberrypi:~# export test_file=paddyg_test; cython3 --embed ${test_file}.py -o ${test_file}.c; gcc $(pkg-config --libs --cflags python3) ${test_file}.c -o ${test_file}
root@raspberrypi:~# time ./${test_file}
79.019864320755
18.89829993247986

real    1m43.753s
user    1m26.570s
sys     0m4.130s
root@raspberrypi:~#
Last edited by SlowBro on Wed Oct 18, 2017 11:27 am, edited 1 time in total.

User avatar
SlowBro
Posts: 93
Joined: Sat Feb 18, 2017 1:30 am

Re: What can be done about Python 3.5 slowness?

Wed Oct 18, 2017 11:02 am

jahboater wrote:
Wed Oct 18, 2017 9:17 am
For amusement I counted the machine instructions needed to execute a trivial Python program
a = 1
b = 1
c = a + b
My time results for your test. 2.7 wins by far. PyPy is the slowest.

Code: Select all

root@raspberrypi:~# time python2.7 jahboater_test.py

real    0m0.351s
user    0m0.220s
sys     0m0.070s
root@raspberrypi:~# time python3.5 jahboater_test.py

real    0m1.026s
user    0m0.790s
sys     0m0.090s
root@raspberrypi:~# export test_file=jahboater_test; cython3 --embed ${test_file}.py -o ${test_file}.c; gcc $(pkg-config --libs --cflags python3) ${test_file}.c -o ${test_file}
root@raspberrypi:~# time ./${test_file}                                                                    
real    0m1.079s
user    0m0.850s
sys     0m0.060s
root@raspberrypi:~# time pypy jahboater_test.py                                                            
real    0m1.081s
user    0m0.740s
sys     0m0.180s
root@raspberrypi:~#

User avatar
SlowBro
Posts: 93
Joined: Sat Feb 18, 2017 1:30 am

Re: What can be done about Python 3.5 slowness?

Wed Oct 18, 2017 11:32 am

Rpi.GPIO is even faster than pigpio.

Code: Select all

root@raspberrypi:~# time python2.7 test_pigpio.py

real    0m0.601s
user    0m0.420s
sys     0m0.080s
root@raspberrypi:~# time python2.7 test_rpi.gpio.py

real    0m0.345s
user    0m0.240s
sys     0m0.040s
root@raspberrypi:~#

User avatar
paddyg
Posts: 2002
Joined: Sat Jan 28, 2012 11:57 am
Location: UK

Re: What can be done about Python 3.5 slowness?

Wed Oct 18, 2017 12:12 pm

Python will only do the import once for each instance of python that's running. The issue is getting just one python process running and staying running. I've just done a quick experiment with a python cgi server and got it to serve several pages running a python cgi and, sure enough, just one instance of python is visible in top (running 100% as the cgi is basically for i in range(200000000): b = i ** 0.5) I think this is the way you should go. Info on the page I linked to before about how to serve cgi info from a python server.

Easiest way to install modules for pypy is to install pip for pypy see http://pi3d.github.io/html/FAQ.html#pypy

PS with the python cgi server experiment I had sudden misgivings about my assertion so I then tried adding
tm = time.time()
import numpy
imptm = time.time() - tm
and the first cgi page to be requested gave a time of 1.28s but all subsequent requests give 0.10s to 0.15s
Last edited by paddyg on Wed Oct 18, 2017 1:07 pm, edited 1 time in total.
also https://groups.google.com/forum/?hl=en-GB&fromgroups=#!forum/pi3d

User avatar
SlowBro
Posts: 93
Joined: Sat Feb 18, 2017 1:30 am

Re: What can be done about Python 3.5 slowness?

Wed Oct 18, 2017 12:56 pm

By switching to a pure http.server and refactoring some code in the background (switched to pigpio for one) the device admin home page loads in about seven seconds under Py 3.5, which is about half the time from before. I'm going to test it under PyPy and 2.7 but this is looking good.

User avatar
paddyg
Posts: 2002
Joined: Sat Jan 28, 2012 11:57 am
Location: UK

Re: What can be done about Python 3.5 slowness?

Wed Oct 18, 2017 1:14 pm

How long does the admin home page take for subsequent loads (after the first time after the server has started)?
also https://groups.google.com/forum/?hl=en-GB&fromgroups=#!forum/pi3d

User avatar
SlowBro
Posts: 93
Joined: Sat Feb 18, 2017 1:30 am

Re: What can be done about Python 3.5 slowness?

Wed Oct 18, 2017 2:38 pm

paddyg wrote:
Wed Oct 18, 2017 1:14 pm
How long does the admin home page take for subsequent loads (after the first time after the server has started)?
Don't have my Pi in front of me right now, but if I recall reloading the page shaved off about two seconds, so from seven to five. So maybe I ought to import my home page (and sub-pages) with the server start, wouldn't you agree?

User avatar
SlowBro
Posts: 93
Joined: Sat Feb 18, 2017 1:30 am

Re: What can be done about Python 3.5 slowness?

Thu Oct 19, 2017 1:05 am

No, it's consistently about 7 seconds to load no matter how many times I refresh.

Return to “Python”

Who is online

Users browsing this forum: No registered users and 13 guests