wachrno
Posts: 3
Joined: Mon Mar 23, 2020 6:14 am

Opencv + Python 60fps 1280x720, how to? :/

Mon Mar 23, 2020 6:41 am

Hi!
I feel like this might have been asked before but after spending some time jumping between places in the forum, I really didn't manage to get a clear answer... :/
I have been trying to get images from picamera as fast as I can in an opencv ready format, but currently I can't get past the 7/8fps... For 1280x720... I'm just not sure what I'm doing wrong. Have been trying to follow different tutorials, but just can't seem to get more than that. This is what I have currently... Does anyone have a better way of doing this? The code I currently have is below, but it isn't anything particularly fancy... and it's from some tutorials online. Any help would be more than appreciated!

Code: Select all

    
    with picamera.PiCamera() as camera:

        # initialize the camera and grab a reference to the raw camera capture
        camera.resolution = (width, height)
        camera.framerate = fps
        rawCapture = PiRGBArray(camera, size=(width, height))
        # allow the camera to warmup
        time.sleep(2.0)

        # capture frames from the camera
        startTime = time.time()
        frameCounter = 0
        for frame in camera.capture_continuous(rawCapture, format="bgr", use_video_port=True):
            image = frame.array
            
	    *place image in a thread and process it in another place*
             rawCapture.truncate(0)
 
I added threads in the middle which allows for more processing of the captured frames, but not to grab frames faster.
Cheers

User avatar
HermannSW
Posts: 2033
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: Opencv + Python 60fps 1280x720, how to? :/

Mon Mar 23, 2020 9:25 am

Python is slow, and OpenCV as well.
While v2 camera can capture mode6 1280x720 with 40-90fps, it would really surprise me if you will come near that range at all.

With raspivid 60fps is no problem at all, with 0 frameskips:
https://github.com/Hermann-SW/userland/ ... ptsanalyze

Code: Select all

🍓 raspivid -md 6 -w 1280 -h 720 -o tst.h264 -pts tst.pts -fps 60 -t 4000
🍓 ./ptsanalyze tst.pts 0
creating tstamps.csv
229 frames were captured at 60fps
frame delta time[us] distribution
      1 16596
      1 16602
      2 16603
      2 16604
      1 16607
      2 16608

Code: Select all

      8 16609
     30 16610
     61 16611
     64 16612
     34 16613
      9 16614
      3 16615
      2 16616
      1 16619
      3 16620
      1 16622

Code: Select all

      1 16629
after skip frame indices (middle column)
0 frame skips (0%)
🍓 
⇨https://stamm-wilbrandt.de/en/Raspberry_camera.html

https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/github_repo_i420toh264
https://github.com/Hermann-SW/fork-raspiraw
https://twitter.com/HermannSW

wachrno
Posts: 3
Joined: Mon Mar 23, 2020 6:14 am

Re: Opencv + Python 60fps 1280x720, how to? :/

Tue Mar 24, 2020 4:24 am

Thanks for the answer! :)
I see, then the best way would be to actually save everything to disk using raspivid and then have a python program running that would load those images and process them? I was hoping to be able to grab images and do some simple processing at that framerate... So it might not be possible to do that with a pure python + opencv approach?

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 8118
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Opencv + Python 60fps 1280x720, how to? :/

Tue Mar 24, 2020 11:25 am

As HermannSW has said, Python is generally fairly slow unless things are optimised in exactly the right way.
1280x720 is 921600 pixels per frame, so 55.2MPixels/sec if 60fps. Any processing will be either as RGB (24bits/pixel, although more typically 32bits/pixel with 8bits wasted) or YUV (12 bits/pixel).

Even if we ignore any overheads of producing the frames or SDRAM accesses, then for a processor at 1.5GHz you've got 27 instructions per pixel to keep up with realtime. SIMD (eg NEON) and multiple threads help, but it only takes a few memcpy or conversion operations to end up with insufficient time to do the work you want.
I know from experience that people do silly things in OpenCV because they don't think about efficiency (I recall one being take a 16bpp image buffer with data in bits 13-0, import to a matrix, convert to floats, multiply by 8.0, convert back to 16bpp unsigned int buffer. Yes, really!)
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

User avatar
HermannSW
Posts: 2033
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: Opencv + Python 60fps 1280x720, how to? :/

Tue Mar 24, 2020 2:07 pm

wachrno wrote:
Tue Mar 24, 2020 4:24 am
... I was hoping to be able to grab images and do some simple processing at that framerate...
Depends on "what simple processing", eg. edge-detection is not simple.

Because OpenCV was too slow I did develop my own framework based on raspividyuv:
https://github.com/Hermann-SW2/userland ... i420toh264

"simple" processing that I was able to get working was identifying a dark airplane before daylight sky and mark it with a whie 2x2 white square (for debugging) -- real application will be to always center camera with stepper pan/tilt camera system while recording. I used 640x480 because higher framerates are possible with raspividyuv, and the number of pixels to process is smaller:
Image


I plan to use that framework for highspeed robot line following based only on camera frame processing:
viewtopic.php?f=37&t=267999
Image


From robot I used 3 years ago, that is a b/w converted frame, this is how robot sees "present" in front of robot, as well as near and far future:
https://forum.arduino.cc/index.php?topi ... msg3482895
Image


Regarding topic (use of OpenCV), 3yo I did canny edge detection on only 320x208 frames.
OpenCV on Pi Zero was not able to process more than 50fps because time per frame was 20ms:
viewtopic.php?f=43&t=189661#p1250429
⇨https://stamm-wilbrandt.de/en/Raspberry_camera.html

https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/github_repo_i420toh264
https://github.com/Hermann-SW/fork-raspiraw
https://twitter.com/HermannSW

wachrno
Posts: 3
Joined: Mon Mar 23, 2020 6:14 am

Re: Opencv + Python 60fps 1280x720, how to? :/

Wed Mar 25, 2020 5:53 am

Thanks for both replies! I still need to format a bit my mind to hardware constraints... definitely... That is really easily digestible information 6by9...

One of the things I'm planning to do is to only process around 50% of the 1280x720. So this might help a bit, but I would still need to be able to grab the images as fast as possible and then only process part. Is there a way of specifying a ROI of the image directly on the sensor? To increase the performance?
I'm also looking into YUV and just using the brightness component.

That's super cool HermannSW, will explore it to see how I could apply it. Thanks!

My initial goal would be to do some object detection, probably starting with a blob or some well defined shape and see how much I could push it... Once the object was detected once, I could also reduce quite a bit the search area for the next frame... But this might already be a bit too much.

Return to “Camera board”