real-time image processing

[simpa]

Hi everyone,

I'm trying real-time image processing right now and i ran into troubles concerning performance issues. My script divides an 480x640 image array into a bunch of rectangles an then merges the pixels in these areas into the average pixel. i measured time consumption for the main blocks of my code:

Code: Select all

#numbers are marks for time measurement. I erased the code for readability.
#bounds is an array of rectangle coordinates. it has the following structure: [ [x1,x2,y1,y2],[x1,x2,y1,y2], ... ]

while True:
#1
                        # get frame from camera
                        ret, frame = cap.read()
#2
                        layer = []
                        #iterate over areas
                        for a,areaX in enumerate(areas[0]):
                                col = []
                                for b,areaY in enumerate(areas[1]):
#3
                                        b = bounds[a][b]
                                        area = frame[b[0]:b[1],b[2]:b[3]]
                                        #convert to 1 dimension to simplify code
                                        area1d = np.concatenate(area[:])
                                        count = len(area1d)
#4
                                        blue = np.sum(area1d[:,0])
                                        green = np.sum(area1d[:,1])
                                        red = np.sum(area1d[:,2])
#5
                                        averagePixel = np.array([red,green,blue])/count
                                        col.append(averagePixel)
#6
                                layer.append(col)

time consumption:

Code: Select all

#1-#2: 0.12s
#3-#4: 0.65s
#4-#5: 1.3s
#5-#6: 0.75s

all: 2.8 s

this is far to much for real-time. I'm wondering how people can run face-detection algorithms that fast.
I know my code is not top level, I'm a beginner at python and stuff. So my question is if somebody can help me improving this code. I was told numpy is surprisingly fast, but 1.3s to add some numbers is not what I expected. My goal are at least 3-5 fps, I'd like 10fps.

I'm using:
raspi model b rev2 @ 800Mhz
4vl2 driver (measured about 8fps, supposed to be 30fps or something. why? i dont really changed anything in the settings)

and yeah I know python is not the fastest language. But its easy and I dont have enough time and the will to jump into C. I think its possible to do this in python.
Help is appreciated

regards,
simpa

ps: I've tried pypy, but it seems it doesn't support opencv. Any other ways to get a video frame in python?

[jamesh]

Likely to get some performance increase by rewriting the code in C. Python is much slower than C, I would expect 5-10 times improvement at least.

But, there may be stuff you can improve in the Python. Do you need the np.concatenate? That takes a while. And can the 4->5 section be rewritten using more efficient code?

[PiGraham]

It's probably faster to use an imaging library to do the grunt work on pixels.
Take a look at Pillow.
http://www.pythonforbeginners.com/gui/how-to-use-pillow
One way to average rectangles of pixels is to generate a thumbnail. Each pixel in the thumbnail is the average of a rectangular block of pixels in the original.

Also look at OpenCV. There is a Python wrapper for it and its functions should be much faster than pixel access in Python.

[paddyg]

numpy can be very fast (for certain things it's supposed to be faster than C as it uses 'magic' fortran based code discovered in ancient peat bogs) however it's quite difficult to figure out. I did some experiments a while ago and found that for multiplying arrays of matrices together the einsum function was much faster than other methods. i.e. some numpy is faster than others.

As @pigraham says, use the graphics module where you can. Pillow has superseded PIL, runs with python3 and pypy and is maintained. I'm pretty sure that numpy can do what you want on the relatively small image size very quickly and if you carry on with numpy I would suggest it's a good idea to convert frame into a np.array just once and do all your work on numpy slices of that, rather than creating new lists and python slices of lists to convert to numpy arrays within the for loops.

PS some tips about optimised numpy slices here

[simpa]

thanks for the replies! I think this will help me a lot.
I'm gonna take a look now and come back if I have problems

[paddyg]

@simpa, how did you get on with this? Out of interest I did some experiments comparing numpy.einsum('ijk->k', im[l:r, t:b]) with Pillow resize(). Although einsum is much faster at doing the averaging for large rectangles (i.e. average rgb for 4x256x256 in a 512x512 image 0.008s c.f. im.resize((2, 2)) 0.160s, all the benefit is eaten up by converting the image to a numpy array. And when the rectangles get smaller than 32x32 the array slicing makes it much slower (though as_strided might cure that). Anyway it looks like 3-5 fps is quite feasible.

[simpa]

Well, I did some changes on the code (reduced the count of areas from 1200 on a 640x480 frame to 400 on 420x420, integrated Pillow) and now I'm running at 3fps, so thanks very much for the help!
Btw: I used Pillow resize() considering your test results (I have lots of small areas in my frame)

I measured time again:
take frame: 0.16s
crop areas: 0.06s
average areas: 0.07s

So the algorithm does good performance now, but the framerate of the camera is not satisfying. I will search for alternatives getting video captures and take a look at the v4l documentation (I'm not very familiar with that). Another point is I want to try pypy to get a few extra secs, therefore I need a way to get a capture without openCV (=>pypy compatible).
Any thoughts on this will be helpful, anyway I'll google around this week and in case I end up with no result I will open another thread.

Thanks again,
simpa