Gelu
Posts: 13
Joined: Tue Oct 28, 2014 2:57 pm

picamera rapid capture and processing

Tue Mar 17, 2015 1:38 pm

I am using the raspberry pi with the picamera and opencv python modules trying to do some rapid capture and processing. Currently I am using the recipe in http://picamera.readthedocs.org/en/late ... processing to capture each image to a BytesIO stream. Then I have added the code inside the ImageProccessor class to convert each stream to an opencv object and do some analysis "on the fly". My objective is to take images as fast as possible, I do background substraction (from an image taken at the beginning) and then detect objects in each image and save sub-images with the detected objects.

My current code threfore looks like:

Code: Select all

import io
import time
import threading
import picamera
import cv2
import picamera.array
import numpy as np


# Create a pool of image processors
done = False
lock = threading.Lock()
pool = []

class ImageProcessor(threading.Thread):
    def __init__(self):
        super(ImageProcessor, self).__init__()
        self.stream = io.BytesIO()
        self.event = threading.Event()
        self.terminated = False
        self.start()

    def run(self):
        # This method runs in a separate thread
        global frames
        global start
        global done
        global grayback
        while not self.terminated:
            # Wait for an image to be written to the stream
            if self.event.wait(1):
                try:
                    minpixel=1
                    self.stream.seek(0)
                    frames+=1
                    currentframe=frames ## Because frame can be incremented by other threads while proccesing

                    # Read the image and do some processing on it
                    # Construct a numpy array from the stream
                    data = np.fromstring(self.stream.getvalue(), dtype=np.uint8)
                    # "Decode" the image from the array, preserving colour
                    image = cv2.imdecode(data, 1)

                    gray = cv2.cvtColor(image,cv2.COLOR_BGR2GRAY)
                    #cv2.imwrite('fullimage'+str(currentframe)+".png",image)
                    d = cv2.absdiff(gray,grayback)
                    
                    # Gaussian blur works better but it is too slow
                    # d= cv2.GaussianBlur(d,(55,55),0) 

                    d= cv2.blur(d,(55,55))
                    # cv2.imwrite('fullimage'+str(currentframe)+"_blurnormal_55.png",d)

                    # I've tried different thresholding procedures, but the best results are with a simple binary threshold
                    #                    ret, thresh = cv2.threshold(d,0,255,cv2.THRESH_BINARY+cv2.THRESH_OTSU)

                    ret, thresh = cv2.threshold(d,5,255,cv2.THRESH_BINARY)
                    # cv2.imwrite('fullimage'+str(currentframe)+"_thres_bin_5.png",thresh)

                    contours, hierarchy = cv2.findContours(thresh,cv2.RETR_TREE,cv2.CHAIN_APPROX_SIMPLE)
                    # cv2.drawContours(image, contours, -1, (0,255,0), 3)
                    # cv2.imwrite('fullimage'+str(currentframe)+"_contour.png",image)

                    for i,cnt in enumerate(contours):
                        x,y,w,h = cv2.boundingRect(cnt)
                        if (w>minpixel | h>minpixel):
                            cv2.imwrite('image_'+str(currentframe)+"_"+str(i)+'.png',image[y:(y+h),x:(x+w)])
                            print("Wrote an image "+str(currentframe)+"_"+str(i))

                    finish = time.time()
                    print('Captured %d frames at %.2ffps' % (
                            currentframe,
                            currentframe / (finish - start)))
                    #...
                    #...
                    # Set done to True if you want the script to terminate
                    # at some point
                    #done=True
                finally:
                    # Reset the stream and event
                    self.stream.seek(0)
                    self.stream.truncate()
                    self.event.clear()
                    # Return ourselves to the pool
                    with lock:
                        pool.append(self)

def streams():
    while not done:
        with lock:
            if pool:
                processor = pool.pop()
            else:
                processor = None
        if processor:
            yield processor.stream
            processor.event.set()
        else:
            # When the pool is starved, wait a while for it to refill
            print ("Waiting")            
            time.sleep(0.1)

with picamera.PiCamera() as camera:
    pool = [ImageProcessor() for i in range(4)]

    print ("Starting fixed settings setup")
    camera.resolution = (1280, 720)
    camera.framerate = 30
    # Wait for analog gain to settle on a higher value than 1
    while camera.analog_gain <= 1:
        time.sleep(0.1)

    print ("Fixing the values")
    # Now fix the values
    camera.shutter_speed = camera.exposure_speed
    camera.exposure_mode = 'off'
    g = camera.awb_gains
    camera.awb_mode = 'off'
    camera.awb_gains = g

    camera.start_preview()
    time.sleep(2)
    backstream = io.BytesIO()
    camera.capture(backstream,format='jpeg', use_video_port=True)
    databack = np.fromstring(backstream.getvalue(), dtype=np.uint8)
    # "Decode" the image from the array, preserving colour
    background = cv2.imdecode(databack, 1)
    grayback = cv2.cvtColor(background,cv2.COLOR_BGR2GRAY)
    cv2.imwrite('background'+".png",background)

    start = time.time()
    frames=0
    camera.capture_sequence(streams(), use_video_port=True)

# Shut down the processors in an orderly fashion
while pool:
    with lock:
        processor = pool.pop()
    processor.terminated = True
    processor.join()

I am looking for suggestions to speed up the proccess (which would hopefully allow to run at even higher image resolutions).
I think one way to speed up a little is to (instead of capturing as JPEG and then decoding of each image to an opencv object which is lossy and time consuming) follow the suggested alternative capturing directly to a picamera.array: http://picamera.readthedocs.org/en/late ... ncv-object , for a single image the code:

Code: Select all

import time
import picamera
import picamera.array
import cv2

with picamera.PiCamera() as camera:
    camera.start_preview()
    time.sleep(2)
    with picamera.array.PiRGBArray(camera) as stream:
        camera.capture(stream, format='bgr')
        # At this point the image is available as stream.array
        image = stream.array
works great but I do not know how to combine these two pieces of code so that the ImageProcessor class defines a picamera.array instead of a BytesIO stream. The need to use a "with" statement to generate the stream for the picamera.array confuses me (I am new to python... ;) ).
Thanks for any pointers.
Angel

P.S: I posted a very similar question in http://stackoverflow.com/questions/2906 ... era-opencv without any answer, so I am reposting here, I am unsure where is the best place to ask this type of questions, maybe I should post on the github of picamera, but I believe here it would have more visibility.

User avatar
paddyg
Posts: 2493
Joined: Sat Jan 28, 2012 11:57 am
Location: UK

Re: picamera rapid capture and processing

Tue Mar 17, 2015 6:02 pm

I don't know if this is the case (but you could test it quite easily with timeit) but could you not simply change the argument from "jpeg" to "bgr" in line 124? I would expect the algorithm to capture the image without compression to be faster and the conversion to numpy.ndarray to be faster.

PS also python is not great with threads where time is of the essence so I think that trying to run four processes at once like this will not be optimal (To be explicit: python threads will not run process "in parallel" even on multi-core machines and will take a significant amount of extra processing not to do so!). As it says on the site where you got the original code "Using a generator function, we can maintain a queue of objects to store the captures, and have parallel threads accept and process the streams as captures come in. Provided the processing runs at a faster frame rate than the captures, the encoder won’t stall." In which case...!
also https://groups.google.com/forum/?hl=en-GB&fromgroups=#!forum/pi3d

Gelu
Posts: 13
Joined: Tue Oct 28, 2014 2:57 pm

Re: picamera rapid capture and processing

Tue Mar 17, 2015 7:10 pm

Thanks for the input. I have tried modifying the jpg argument to bgr but it seems I am not able to convert the stream to a correct numpy array, i.e if I do:

Code: Select all

stream2 = io.BytesIO()
camera.capture(stream2, format='bgr')
# Construct a numpy array from the stream                                                                                                                                   
data2 = np.fromstring(stream2.getvalue(), dtype=np.uint8)
# "Decode" the image from the array, preserving colour                                                                                                                      
image2 = cv2.imdecode(data2, 1)
cv2.imshow('image2',image2)
it returns an error saying that the image size is zero...
maybe I shouldn't be using the np.fromstring as http://picamera.readthedocs.org/en/late ... ra.capture states that argument bgr returns 24bit format, but then I don't know how to convert into opencv..

As for the comment on python, I found it convenient because of the picamera module but maybe I should port the code to C. I don't know enough of python to understand the paralellization limitation, the 4 cores of my raspberry pi 2 are working with the above code though. The bottleneck is in the proccessing part of the code (which also includes the conversion to numpy array and opencv object), but I feel that you know something I dont :lol: ... and my ignorance makes me unable to fully grasp your "in which case!..." :roll:

User avatar
paddyg
Posts: 2493
Joined: Sat Jan 28, 2012 11:57 am
Location: UK

Re: picamera rapid capture and processing

Tue Mar 17, 2015 8:36 pm

Hi, 24 bits sound like 3 x uint8 bytes which is what you look to be defining as your numpy array, but maybe fromstring() is expecting something different. In python-imaging you can just pass the image numpy.array(im). numpy.fromfile() might work as it can take a file object that probably behaves like BytesIO, but possibly not!

This seems like it might bypass the issue of loading into numpy (the second example 'how to avoid jpeg and de-jpeg')
https://github.com/waveform80/picamera/ ... ncv-object

This is the ref on threading in python http://www.dabeaz.com/python/UnderstandingGIL.pdf But actually I think that file, and probably other io, is not governed by the GIL so there could well be a sensible reason for running multiple threads. My ellipsis was meant to imply: if the processing runs at a faster frame rate than the capture then surely you only need one additional thread to do it.
also https://groups.google.com/forum/?hl=en-GB&fromgroups=#!forum/pi3d

Gelu
Posts: 13
Joined: Tue Oct 28, 2014 2:57 pm

Re: picamera rapid capture and processing

Tue Mar 17, 2015 9:25 pm

Thanks,
Yes, the second example you mention in https://github.com/waveform80/picamera/ ... ncv-object is what I was attempting to use (it is the second part of code in my first post above). The problem is that I don't know how to incorporate the "with picamera.array.PiRGBArray(camera) as stream:" and the "camera.capture(stream, format='bgr')" lines in the threaded code (the first part of code in my first post). The stream definition is inside the imageProccesing Class but the camera.capture is in the main thread and using a "with" forces me to have both lines of code (i.e. stream definition and capture to stream) in the same block.

User avatar
paddyg
Posts: 2493
Joined: Sat Jan 28, 2012 11:57 am
Location: UK

Re: picamera rapid capture and processing

Wed Mar 18, 2015 9:37 am

OK, sorry about that; I'll catch up with you eventually! It's a bit hard to figure out exactly how the streams are passed around in the first one without actually getting to grips with it (and probably putting in the RPi with the camera set up etc etc). Which I might get chance to do tonight.

Did you try any other methods of inputting to an dnarray such as fromfile() or frombuffer() or alternative outputs from io.BytesIO (or none i.e. fromfile() probably expects a file/io object)? I think I would try to debug at what point cv ended up with a zero length image by putting stream2.getvalue() into a variable (and printing info about it, type, size etc) then do the same thing for data2. I would see what seemed to change when the "jpeg" was swapped with "bgr".
also https://groups.google.com/forum/?hl=en-GB&fromgroups=#!forum/pi3d

Gelu
Posts: 13
Joined: Tue Oct 28, 2014 2:57 pm

Re: picamera rapid capture and processing

Wed Mar 18, 2015 6:43 pm

I guess that frombuffer is the way to go, looking at the code in class PiRGBArray I get the definition as:

Code: Select all

def flush(self):
super(PiRGBArray, self).flush()
self.array = bytes_to_rgb(self.getvalue(), self.size or self.camera.resolution)
and then :

Code: Select all

def bytes_to_rgb(data, resolution):
"""
Converts a bytes objects containing RGB/BGR data to a `numpy`_ array.
"""
width, height = resolution
fwidth, fheight = raw_resolution(resolution)
if len(data) != (fwidth * fheight * 3):
raise PiCameraValueError(
'Incorrect buffer length for resolution %dx%d' % (width, height))
# Crop to the actual resolution
return np.frombuffer(data, dtype=np.uint8).\
reshape((fheight, fwidth, 3))[:height, :width, :]
I'll give it a try and will get back!

Gelu
Posts: 13
Joined: Tue Oct 28, 2014 2:57 pm

Re: picamera rapid capture and processing

Wed Mar 18, 2015 7:22 pm

OK, doing:

Code: Select all

  
    stream = io.BytesIO()
    start = time.time()
    camera.capture(stream, format='bgr')
    # I have got this code from picamera array.py :                                                                                                                                         
    # class PiRGBArray(PiArrayOutput):                                                                                                                                          
    # Produces a 3-dimensional RGB array from an RGB capture.                                                                                                                   
    # Round a (width, height) tuple up to the nearest multiple of 32 horizontally                                                                                               
    # and 16 vertically (as this is what the Pi's camera module does for                                                                                                        
    # unencoded output).                                                                                                                                                        
    width, height = camera.resolution
    fwidth = (width + 31) // 32 * 32
    fheight = (height + 15) // 16 * 16
    if len(stream.getvalue()) != (fwidth * fheight * 3):
        raise PiCameraValueError('Incorrect buffer length for resolution %dx%d' % (width, height))
    image= np.frombuffer(stream.getvalue(), dtype=np.uint8).\
        reshape((fheight, fwidth, 3))[:height, :width, :]
    cv2.imwrite('bgrnumpy.png',image)
does the work of converting directly to an opencv so I can take part of this code into the thread class above. Thanks!!

User avatar
paddyg
Posts: 2493
Joined: Sat Jan 28, 2012 11:57 am
Location: UK

Re: picamera rapid capture and processing

Wed Mar 18, 2015 9:29 pm

Well done. I remember seeing the rgb size limitation before but had forgotten about it, hopefully doing the slicing like that all on one line will allow numpy to resize it very quickly.

It would interesting to know if this method is significantly faster than the original recipe, in which case it could usefully go in the picamera docs as this must be a pretty standard requirement. However, I did read that the jpeg compression is a hardware feature of the camera chip so the smaller number of bytes passed through the io stream might compensate for the time saved decompressing.
also https://groups.google.com/forum/?hl=en-GB&fromgroups=#!forum/pi3d

Gelu
Posts: 13
Joined: Tue Oct 28, 2014 2:57 pm

Re: picamera rapid capture and processing

Thu Mar 19, 2015 8:55 am

ummm, I would expect the code I wrote to be as fast as the picamera.array (basically because it is the same code). It is definitely faster than the jpeg conversion (as the original recipe stated).
I've done some timing and that is the case,

jpeg>picamera.array=np.frombuffer

going through jpeg is slower. What I am puzzled now :roll: is that the first time that camera.capture is called with format='bgr' the capture takes longer than subsequent calls (no matter if I do it with picamera.array or directly with then np.frombuffer)

This are the timings I get from the code bellow:
Time with jpeg+stream+numpy = 1.4922
Time with picamera.array = 1.2804
Time with picamera.array = 0.9746
Time with bgr+stream+numpy = 0.9922
Time with bgr+stream+numpy = 0.9954

The code:

Code: Select all

import io
import time
import picamera
import cv2
import picamera.array
import numpy as np

with picamera.PiCamera() as camera:
    camera.resolution = (1280, 720)
    camera.framerate = 30
    print("Selecting adequate fixed settings")
    # Wait for analog gain to settle on a higher value than 1
    while camera.analog_gain <= 1:
        time.sleep(0.1)
    # Now fix the values
    camera.shutter_speed = camera.exposure_speed
    camera.exposure_mode = 'off'
    g = camera.awb_gains
    camera.awb_mode = 'off'
    camera.awb_gains = g

    # start and wait for camera setup
    camera.start_preview()
    time.sleep(2)



    # Capture using jpeg
    stream = io.BytesIO()
    start = time.time()
    camera.capture(stream, format='jpeg')
    # Construct a numpy array from the stream
    data = np.fromstring(stream.getvalue(), dtype=np.uint8)
    # "Decode" the image from the array, preserving colour
    image = cv2.imdecode(data, 1)
    cv2.imwrite('jpegnumpy.png',image)
    print("Time with jpeg+stream+numpy = %.4f" % (time.time()-start))


    # capture using picamera.array
    start = time.time()
    with picamera.array.PiRGBArray(camera) as stream:
        camera.capture(stream, format='bgr')
        # At this point the image is available as stream.array
        background = stream.array
    cv2.imwrite('pimcamera.png',background)

    print("Time with picamera.array = %.4f" % (time.time()-start))


    # Lets do it again

    # capture using picamera.array
    start = time.time()
    with picamera.array.PiRGBArray(camera) as stream:
        camera.capture(stream, format='bgr')
        # At this point the image is available as stream.array
        background = stream.array
    cv2.imwrite('pimcamera.png',background)

    print("Time with picamera.array = %.4f" % (time.time()-start))


    # capture using bgr stream and converting to numpy 
    # basically the same as picamera.arry but written here directly
    start = time.time()
    stream = io.BytesIO()
    camera.capture(stream, format='bgr')
    # This code is from 
    # class PiRGBArray(PiArrayOutput):
    # Produces a 3-dimensional RGB array from an RGB capture.
    # Construct a numpy array from the stream
    # I have got this code from picamera array.py :
    # Round a (width, height) tuple up to the nearest multiple of 32 horizontally
    # and 16 vertically (as this is what the Pi's camera module does for
    # unencoded output).
    width, height = camera.resolution
    fwidth = (width + 31) // 32 * 32
    fheight = (height + 15) // 16 * 16
    if len(stream.getvalue()) != (fwidth * fheight * 3):
        raise PiCameraValueError('Incorrect buffer length for resolution %dx%d' % (width, height))

    image= np.frombuffer(stream.getvalue(), dtype=np.uint8).\
        reshape((fheight, fwidth, 3))[:height, :width, :]
    cv2.imwrite('bgrnumpy.png',image)

    print("Time with bgr+stream+numpy = %.4f" % (time.time()-start))



    # And again

    # capture using bgr stream and converting to numpy 
    # basically the same as picamera.arry but written here directly
    start = time.time()
    stream = io.BytesIO()
    camera.capture(stream, format='bgr')
    # This code is from 
    # class PiRGBArray(PiArrayOutput):
    # Produces a 3-dimensional RGB array from an RGB capture.
    # Construct a numpy array from the stream
    # I have got this code from picamera array.py :
    # Round a (width, height) tuple up to the nearest multiple of 32 horizontally
    # and 16 vertically (as this is what the Pi's camera module does for
    # unencoded output).
    width, height = camera.resolution
    fwidth = (width + 31) // 32 * 32
    fheight = (height + 15) // 16 * 16
    if len(stream.getvalue()) != (fwidth * fheight * 3):
        raise PiCameraValueError('Incorrect buffer length for resolution %dx%d' % (width, height))

    image= np.frombuffer(stream.getvalue(), dtype=np.uint8).\
        reshape((fheight, fwidth, 3))[:height, :width, :]
    cv2.imwrite('bgrnumpy.png',image)

    print("Time with bgr+stream+numpy = %.4f" % (time.time()-start))

Gelu
Posts: 13
Joined: Tue Oct 28, 2014 2:57 pm

Re: picamera rapid capture and processing

Thu Mar 19, 2015 10:46 am

OK, for the records, all of the above was documented in:
https://picamera.readthedocs.org/en/rel ... rgb-format
(shame on me for not reading slow enough...)

but I have another problem here!! (and it seems related to your comment on the jpeg compression).
When I try to capture with use_video_port=TRUE the jpeg capture is in fact faster than the rgb->numpy procedure using the still port:
Time with jpeg+stream+numpy = 0.7484 (with video_port=TRUE)
against the 0.9-1 timing using the rgb array.

The "problem" is that it also seems that using the video port and bgr capture is not done correctly:
camera.capture(stream, format='bgr',use_video_port=True)
returns a completely black image ¿?
this behaviour is not very well documented (or at least I could not find it).

Return to “Python”