Cpython, Cython and PyPy

[Hove]

My sensors churn out data at 1kHz over I2C. I use a combination of RPIO and GPIO to catch hardware interrupts and generate hardware PWM. Every 10 samples, I do some motion processing.

With CPython (interpreted), with every trick in the book that I know, I can squeeze out about 950Hz processing on my A+ overclocked to 900MHz.

That means I'm missing 5% of my data some of which is integrated, and therefore these missed data may lead to permanent integration errors. This is most likely due to the motion processing happening every 10 samples (~100Hz).

Both Cython (C compiled) and PyPy (JIT compiler) should more than give me the extra speed I need so that motion processing happens in under 1ms and I don't miss the next batch of data.

What I'm unsure of is whether Cython / PyPy are capable of handing the GPIO and RPIO 'C' libraries and if so, do I need to do anything extra to make these work? Both libraries are Python wrappers for pure 'C' code directly accessing registers / DMA or can I just change my call from sudo python ./mycode.py to sudo cython ./mycode.py (or the equivalent of PyPy).

TIA

[MarkHaysHarris777]

Both Cython (C compiled) and PyPy (JIT compiler) should more than give me the extra speed I need so that motion processing happens in under 1ms and I don't miss the next batch of data.

If you study Python, hang around the python forum(s), and have used Python extensively, then you know that Python is ill-suited for performance and|or fine timing applications. Keep in mind that the overclocked Raspberry PI isn't really that fast in the first place (by modern standards). It would be much better to build your project with C|C++ and|or assembler. You might also want to try to build your 'own' gpio libraries for the functions you need, or build C modules to call the C libraries which you then embed into your python scripts. But to repeat, Python has 'never' been advertised as a performance tool (its just not a primary goal of Python).

What I'm unsure of is whether Cython / PyPy are capable of handing the GPIO and RPIO 'C' libraries and if so, do I need to do anything extra to make these work?

A properly compiled C library for Python should work fine; assuming the version of Python you're using complies with the C calling conventions and that the C libraries used were compiled for the Arm.

[Hove]

If you study Python, hang around the python forum(s), and have used Python extensively, then you know that Python is ill-suited for performance and|or fine timing applications. Keep in mind that the overclocked Raspberry PI isn't really that fast in the first place (by modern standards). It would be much better to build your project with C|C++ and|or assembler. You might also want to try to build your 'own' gpio libraries for the functions you need, or build C modules to call the C libraries which you then embed into your python scripts. But to repeat, Python has 'never' been advertised as a performance tool (its just not a primary goal of Python).

One of the aims of this project is to prove that python _is_ more than fast enough to sample data and process it at the IMU sampling frequency of 1kHz. If it weren't for the complex motion processing, the answer would be an unqualified yes. Currently it's missing about 5% of samples while it does motion processing of batches of 10 samples. If the RPi A2 launch was nearer, then I could spread the code over CPU's and it would just work using just interpreted CPython.

A properly compiled C library for Python should work fine;...

With what? Cython, PyPy? Both?

[paddyg]

Well cython is C code - it converts pythonish code to C then compiles it! It's not too difficult to use, here's an example of getting GPIO digital input and output using cython. I found it increased the speed over normal python by a factor of 150.
BUT
1. I had to take the "raw" C code for basic GPIO input output and wrap it with my cython routine (but wasn't too difficult as you can see)
2. The speed improvements only work while the looping is kept inside the cython loop. If a normal python loop called the cython code as a function it would show scarcely any improvement. This is because of the overhead of converting to and from python variables. i.e. you would need to have your cythonised routine running all the time in it's own thread and pass the info out as requested (I've not done anything like that and don't know if queue, for instance can be used)
3. I'm not sure that this is the best way to solve your problem
3.1 If the readings from the accelerometer were in a FIFO buffer then you wouldn't miss any of them (you might have a timing error still though, but better than nothing at all). I don't know what chip you are using but you can buy ones with FIFO buffers for only a few quid and I don't think it would count as cheating.
3.2 The main reason your quadcopter goes out of control is that it starts to tilt but thinks it's still horizontal. Once its frame of reference it tilted nothing it does will sort it out. Most applications using accelerometers now correct their orientation using the earth's magnetic field and prevent integration errors (in 3 degrees of freedom anyway).

[Hove]

Hi Paddy,

I'll have a look at what it takes to Cythonize my code - we've actually chatted this over a year or so ago, but I found other ways to improve performance. Did you have to cythonize the GPIO python library or did you just use the 'C' code of the library directly?

I'm using the MPU-9250 but timing rules out the use of the FIFO in interpreted Python: if I run sampling at 1kHz, the FIFO will overflow as the code loops about 950Hz but if I run sampling at 500Hz, then timing can have huge errors, and I integrate the accelerometer to get velocities (and actually that's the reason for this question). By using the data registers and a hardware data ready interrupt, timing accuracy is as good as it can be.

I've just tried PyPy quickly but it stalled at importing RPi.GPIO - but it looked like it may just be a naming problem. I've got the RPi.GPIO code so I can easily tweak its names to see if that makes PyPy happy.

Cheers,

[paddyg]

As you say, it's a while ago but I think I looked at original example C code that Dom and Gert put up and did a basic rearrangement so setup_io() and pin_out() were functions, and got rid of main() (you would need a pin reading function obviously). I could then do a cython style import (cdef extern...) from the cython code and pass info to and read from the C functions.

I think it would be hard work to cythonize your whole code - much better to make a small cython function of the time-critical bits and run it in another thread. The main code would then get a list of time-stamped value asynchronously. Or, better still, the cython function would do the integration and as much of the number crunching as possible then just put the resultant figures into a global or queue at a lower rate.

I'm surprised about the issue with the FIFO buffer. Can you not empty it each time you read from it and just assume each reading is 1ms apart? Are the delays in the python loop long enough for the whole buffer to fill? I don't suppose the buffers are very big but I imagined they could hold a few hundred readings, obviously not enough.

[Hove]

As you say, it's a while ago but I think I looked at original example C code that Dom and Gert put up and did a basic rearrangement so setup_io() and pin_out() were functions, and got rid of main() (you would need a pin reading function obviously). I could then do a cython style import (cdef extern...) from the cython code and pass info to and read from the C functions.

I think it would be hard work to cythonize your whole code - much better to make a small cython function of the time-critical bits and run it in another thread. The main code would then get a list of time-stamped value asynchronously. Or, better still, the cython function would do the integration and as much of the number crunching as possible then just put the resultant figures into a global or queue at a lower rate.

That could work, yes. First though, I'm going to spend a bit of time with PyPy as it seems a simpler solution because it doesn't need the convertion to "C" format.

I'm surprised about the issue with the FIFO buffer. Can you not empty it each time you read from it and just assume each reading is 1ms apart? Are the delays in the python loop long enough for the whole buffer to fill? I don't suppose the buffers are very big but I imagined they could hold a few hundred readings, obviously not enough.

Hadn't thought of it that way - I guess it's a bit of a balancing act: my python code can't read the registers directly at 1kHz (I'm about 5% short of that), but _if_ reading the FIFO is quicker, and quick enough to do occasional motion processing then that could work. Each read could just empty the FIFO, averaging / integrating the data assuming 1ms intervals, and doing the motion processing each time FIFO is emptied. But it does rely on the fact the FIFO reads are faster than the register reads to give that little bit of time for the periodic motion processing.

Swapping to the FIFO is a big code change and disposes of a lot of hardware interrupt code performance I've put into my custom RPi.GPIO python library, so I'm a bit cautious about proceeding based upon a hope that reading the FIFO is faster. But desperation is a good motivator

[Hove]

I now have the GPIO library working under PyPy simply by installing it as a pypy module based on the RPi.GPIO 'C' source code:

Code: Select all

sudo pypy ./setup.py install

At the moment, the test code I'm using is not aimed at performance checking, so I can't tell if there's any improvement. Certainly there's a delay of a few seconds while the JIT compiler does its business, and there are comments that the performance may be reduced importing 'C' libraries: http://pypy.readthedocs.org/en/latest/f ... mporterror

More anon.

[paddyg]

Yes, major changes are definitely to be undertaken with caution! Ideally there would be some way of testing the read speed in isolation before committing to a major rewrite. Probably the most efficient way would be to use your existing program unchanged with cProfile, start the program as per

Code: Select all

$ cd ~/quadcopter_or_whatever
$ python -m cProfile ~/quadcopter_or_whatever/Quadcopter.py > result.txt

and after a few minutes running stop it and look in the result.txt which you will be able to parse with a spreadsheet. You can then look to see how many times I2C.readList() was called and what the execution time was. That way you can work out the resultant I2C read speed. I would expect it to be capable of MB/s if the I2C is clocked as fast as it will go.

It's definitely worth spending a bit of effort analysing and profiling before starting to modify your code.

PS I note your sceptical silence on my doubts as to whether adding in these missing readings will really make that much difference. Maybe this will explain my thinking better. Below is a copied bit of spreadsheet, the first column is the actual acceleration, really it's a continuous variable so there should be an infinite number of values but I just represent it as having significantly more values than the accelerometer can read: whatever the sample speed there is still some acceleration happening between readings that you haven't measured. The second column is the cumulative sum of accelerations or velocity (really it's infinitesimal integral). The third column is regular sampling. The fourth column is regular sampling but with some readings missed (and "filled in" using the next reading). If you make something similar and play around with it you will see that missing readings do have a detrimental effect but not much. In fact, though I don't know the formula for things like this, it is almost always something over root n.

So if your estimate of the velocity based on getting all the readings was 0.100% out, if you only had 95% of the readings your estimate would be 0.103% out. Or, to put it another way, if the quadcopter went out of control after 10s using 95% of the accelerometer readings I would expect it to go out of control after 10.25s if you used 100% of the readings.

Of course this is assuming all that's happening is sampling, if there is a systematic error that faster reading will solve then that would change things.

Code: Select all

-12.9	-12.9			
-12.4	-25.3			
-11.8	-37.1			
-11.9	-49.0			
-12.1	-61.1	-121.3	-121.3	
-11.5	-72.6			
-11.6	-84.2			
-11.8	-96.0			
-11.4	-107.5			
-11.8	-119.3			
-11.6	-130.9			
-10.9	-141.8			
-11.0	-152.8			
-11.0	-163.8			
-10.6	-174.4	-227.6	-227.6	
-11.3	-185.7			
-10.5	-196.2			
-10.4	-206.6			
-10.6	-217.2			
-10.8	-228.0			
-10.6	-238.6			
-10.2	-248.8			
-10.6	-259.4			
-9.9	-269.2			
-10.0	-279.2	-327.8	-327.8	
-10.0	-289.3			
-9.3	-298.5			
-9.6	-308.1			
-9.2	-317.3			
-9.0	-326.3			
-9.1	-335.4			
-9.0	-344.4			
-9.2	-353.6			
-8.8	-362.4			
-8.8	-371.2	-416.2	-327.8	missed
-8.5	-379.7			
-8.8	-388.5			
-8.5	-397.0			
-8.3	-405.3			
-8.0	-413.3			
-7.8	-421.1			
-8.0	-429.1			
-7.8	-436.9			
-7.9	-444.8			
-8.1	-452.9	-497.6	-490.7	
-8.2	-461.1			
-8.0	-469.1			
-7.9	-477.0			
-7.2	-484.2			
-6.9	-491.1			
-7.3	-498.4			
-7.5	-505.9			
-6.5	-512.4			
-6.8	-519.2			
-6.9	-526.2	-566.9	-490.7	missed
-6.5	-532.6			
-6.4	-539.1			
-6.0	-545.1			
-6.4	-551.5			
-6.0	-557.5			
-6.4	-563.9			
-6.1	-569.9			
-5.6	-575.5			
-5.6	-581.1			
-5.4	-586.5	-621.0	-599.0	
-5.9	-592.4			
-5.6	-597.9			
-5.2	-603.1			
-5.0	-608.1			
-5.3	-613.4			
-5.2	-618.6			
-4.9	-623.6			
-5.2	-628.7			
-4.7	-633.4			
-5.0	-638.4	-671.2	-649.2	
-4.9	-643.3			
-3.9	-647.3			
-4.3	-651.6			
-4.5	-656.1			
-4.4	-660.5			
-4.2	-664.7			
-3.9	-668.6			
-3.9	-672.5			
-4.0	-676.5			
-3.8	-680.2	-708.9	-686.9	
-3.1	-683.3			
-3.3	-686.6			
-2.8	-689.4			
-2.8	-692.2			
-2.4	-694.7			
-3.3	-697.9			
-3.0	-700.9			
-2.9	-703.8			
-2.0	-705.8			
-2.6	-708.4	-735.4	-686.9	missed
-2.7	-711.2			
-2.3	-713.5			
-1.9	-715.4			
-2.1	-717.5			
-1.6	-719.1			
-1.9	-721.0			
-1.3	-722.3			
-1.9	-724.3			
-1.7	-726.0			
-1.5	-727.5	-750.6	-717.3	
-0.9	-728.4			
-1.0	-729.4			
-1.1	-730.6			
-1.1	-731.7			
-0.8	-732.5			
-0.8	-733.3			
-0.2	-733.6			
-0.8	-734.3			
-0.4	-734.7			
-0.4	-735.1	-754.3	-721.0	
0.3	-734.8			
-0.2	-735.0			
-0.2	-735.2			
0.1	-735.1			
0.6	-734.5			
0.2	-734.3			
0.1	-734.2			
0.7	-733.4			
0.7	-732.7			
1.0	-731.7	-743.9	-710.6	
1.1	-730.6			
1.0	-729.7			
1.3	-728.3			
1.7	-726.6			
1.8	-724.8			
1.4	-723.4			
1.9	-721.5			
2.2	-719.3			
1.6	-717.7			
1.6	-716.1	-728.0	-694.7	
2.7	-713.5			
2.4	-711.0			
2.0	-709.0			
2.2	-706.8			
2.4	-704.5			
2.9	-701.6			
2.6	-699.0			
3.2	-695.8			
3.4	-692.4			
3.4	-689.0	-693.9	-660.6	
3.6	-685.5			
3.0	-682.5			
3.4	-679.1			
3.6	-675.5			
3.4	-672.1			
4.3	-667.8			
4.1	-663.7			
4.1	-659.6			
3.7	-655.9			
3.9	-652.0	-655.1	-621.8	
4.7	-647.3			
4.8	-642.5			
4.4	-638.1			
4.9	-633.2			
4.7	-628.4			
5.0	-623.5			
5.1	-618.4			
5.3	-613.1			
5.5	-607.6			
5.6	-602.0	-599.4	-621.8	missed
5.1	-596.9			
5.2	-591.7			
6.0	-585.7			
6.0	-579.7			
5.7	-574.0			
5.7	-568.3			
6.2	-562.1			
6.2	-555.9			
6.2	-549.7			
6.1	-543.6	-538.8	-500.6	
6.3	-537.3			
6.6	-530.7			
7.2	-523.5			
7.2	-516.3			
7.0	-509.3			
6.9	-502.4			
7.4	-495.0			
7.4	-487.6			
7.4	-480.2			
7.6	-472.6	-463.0	-424.9	
7.3	-465.4			
8.2	-457.1			
8.1	-449.1			
8.1	-441.0			
8.4	-432.6			
8.5	-424.0			
8.3	-415.7			
8.4	-407.3			
8.1	-399.1			
8.9	-390.2	-373.7	-335.6	
9.1	-381.1			
9.2	-371.9			
8.5	-363.3			
9.2	-354.1			
9.2	-344.9			
				
	-344.9	-373.7	-335.6	
	    Error = 	-28.8	9.3	

[Hove]

PS I note your sceptical silence on my doubts as to whether adding in these missing readings will really make that much difference.

The silence was more of a "I'm going to be an obstinate Yorkshireman and try it anyway" type of silence. This project has taken so long and I'm pretty much trying anything I can to see if it improves matters. To be honest, as you say, I don't think this will, but at the same time, by doing it, it'll rule out another possible cause for the horizontal drift I'm seeing.

[Hove]

I now have my quadcopter code running using PyPy - I just needed to rebuild my GPIO and RPIO code using PyPy to run the setup.py install scripts. Sadly, performance dropped down to 550 loops per second rather than 950 I was able to get from CPython. This was using PyPy version 2.2.1 from the standard Raspian distribution and PyPy is now up to 2.6.0, so I'll get myself a local copy and rebuild.

On the plus side, I updated the RPi distribution in passing, and that improved performance so that I now get about 98% compared to the 95 I was seeing previously.

[paddyg]

Interesting about pypy. I think cProfile works for pypy so you could get some very useful info by comparing the two. I bet it's down to just one or two bottle-neck processes.

I understand what you mean about not wanting to swap around until you've exhausted a line of enquiry - logical, not obstinate.

[Hove]

@paddyg

Just an update - I couldn't get Cprofile or profile working - no idea why - I just ended up with an empty profiling file.

I implemented FIFO, but that's turned out infinitely worse; the MPU6050 FIFO size register doesn't seem to decrement as you read data from it. It seems the only option is to reset the FIFO after reading the data into my own FIFO. But here, there's a risk of losing data so the sensor data shifts, and what should be an accelerometer byte turns out to be a gyro byte (based on tracking values for Z-axis gravity). At the same time, I can only read the FIFO a byte at a time over I2C, whereas I read 14 bytes at a time directly from the data registers. The python smbus is hugely inefficient even at 400kbps baudrate and I think the FIFO is filling fast than I can empty it without resetting it (and thereby losing sync and data).

I could try SPI as its data rate can be pushed much higher, but that means a rework of the custom PCBs the sensors are mounted on to use the SPI rather than the I2C pins.

I'll persevere with pypy for the moment, I think there may have been an i2c trick I missed with it.

[joan]

...
I'll persevere with pypy for the moment, I think there may have been an i2c trick I missed with it.

If you are using the Python SMBus module you'd probably gain a bit by using the raw I2C device instead.

[Hove]

...
I'll persevere with pypy for the moment, I think there may have been an i2c trick I missed with it.

If you are using the Python SMBus module you'd probably gain a bit by using the raw I2C device instead.

I am using the SMBus module - what's the 'raw I2C device' - is this effectively reading / writing to /dev/i2c* via Python?

[joan]

...
I am using the SMBus module - what's the 'raw I2C device' - is this effectively reading / writing to /dev/i2c* via Python?

Yes, that's what I meant. The following code shows one way of talking to the raw device.

Code: Select all

#!/usr/bin/env python

import io
import fcntl

# i2c.py
# 2014-09-21
# Public Domain

I2C_SLAVE=0x0703

class i2c:

   def __init__(self, device, bus):

      self.fr = io.open("/dev/i2c-"+str(bus), "rb", buffering=0)
      self.fw = io.open("/dev/i2c-"+str(bus), "wb", buffering=0)

      # set device address

      fcntl.ioctl(self.fr, I2C_SLAVE, device)
      fcntl.ioctl(self.fw, I2C_SLAVE, device)

   def write(self, bytes):
      self.fw.write(bytes)

   def read(self, bytes):
      return self.fr.read(bytes)

   def close(self):
      self.fw.close()
      self.fr.close()

if __name__ == "__main__":

   import time

   import struct

   import i2c

   import smbus

   dev = i2c.i2c(0x53, 0) # device 0x53, bus 0

   dev.write("\x2D\x00") # POWER_CTL reset
   dev.write("\x2D\x08") # POWER_CTL measure
   dev.write("\x31\x00") # DATA_FORMAT reset
   dev.write("\x31\x0B") # DATA_FORMAT full res +/- 16g

   num_samples=5000

   sample=[(0,0,0)]*num_samples

   start1 = time.time()

   for s in xrange(num_samples):

      dev.write("\x32")
      sample[s] = struct.unpack('3h', dev.read(6))
      # print("x={} y={} z={}".format(buf[0], buf[1], buf[2]))

   end1 = time.time()

   bus = smbus.SMBus(0)
   sensor_address = 0x53

   start2 = time.time()

   for s in xrange(num_samples):

      buf = bus.read_i2c_block_data(sensor_address, 0x32, 6)
      tup=(buf[0]|buf[1]<<8, buf[2]|buf[3]<<8, buf[4]|buf[5]<<8)
      sample[s] = tup

   end2 = time.time()

   print(end1-start1)

   print(end2-start2)

   dev.close()

[Hove]

Thanks Jean - I'll add this to the list of the bits to try imminently!

[Hove]

Yes, that's what I meant. The following code shows one way of talking to the raw device. <snip>

Out of interest do you have the performance stats from that script which used I2C dev and smbus to do the same thing? Or even just a vague memory how much faster direct I2C was compared to SMBus? Just sorting out what to investigate next with my very limited free time

Thanks

[joan]

...
Out of interest do you have the performance stats from that script which used I2C dev and smbus to do the same thing? Or even just a vague memory how much faster direct I2C was compared to SMBus? Just sorting out what to investigate next with my very limited free time
...

I'm afraid not. I don't remember there being much of a difference. The real benefit is not having to restrict read/writes to the constraints of the SMBus command set.