OpenCV feature tracking example

[jbeale]

(New thread, as I didn't want to clutter an old thread in case there are more posts on this topic)

EDIT: If you aren't familiar with OpenCV on the RPi, below notes show installing OpenCV 2.4 (but not the OpenCV 3.0.0 I'm using)
http://www.pyimagesearch.com/2015/02/23 ... i-2-and-b/
http://www.pyimagesearch.com/2015/03/30 ... nd-python/

Thank to silvanmelchior, btidey and the other RPi Cam Web Interface project contributors yet again for the ever-improving software. I found another use for it as a collector of test cases for an OpenCV experiment. Demo on YouTube: https://www.youtube.com/watch?v=KCevhaR75_s

I'm pretty impressed with OpenCV and Python. After one weekend of playing with examples, I was able to get something that does more than a stock motion-detector webcam does. At least in some cases, it can distinguish cars and pedestrians from tree branches moving in the wind. It starts by detecting any motion, but then it measures how far detected objects travel across the screen. A branch moves only a short distance back and forth, while "interesting" objects move farther. In the video, I apply a velocity and distance-travelled text label to any blob which has moved more than 50 pixels left or right, since it was first detected.

The video input is from the RPi, but I'm actually running OpenCV 3.0.0 and Python 2.7.5 on Windows 7. I *think* you can run it on RPi as well, installing with

Code: Select all

sudo apt-get install libopencv-dev python-opencv

but I haven't tried that yet. Be careful if you are using another version of OpenCV (like 2.x) as some syntax has changed in 3.0.

I am using createBackgroundSubtractorMOG2 and SimpleBlobDetector. The demo source code is at https://github.com/jbeale1/OpenCV
If you want to try this yourself, edit BlobTrack1.py to change my hardcoded input filename to read your own video:

Code: Select all

cap = cv2.VideoCapture('2015-06-28-CarTest.mp4')  # compilation of test cases

In theory, this could all run live on the RPi, for sufficently low values of resolution and frame rate, but I haven't tested that. This was a one-weekend hack starting from near-ignorance of OpenCV. There are all kinds of obvious improvements but surprisingly it mostly works, even without checking if a given blob matches the same object from one frame to the next.

[jbeale]

As a followup, if you use apt-get to install OpenCV, you get version 2.4.1 and my code is written for 3.0.0

You can compile OpenCV 3.0.0 from source on the R-Pi following one of these instructions:
http://robertcastle.com/2014/02/install ... pberry-pi/
http://raspberrypi.stackexchange.com/qu ... berry-pi-b

I had to install cmake-curses-gui before I could use 'ccmake ../'

Code: Select all

sudo apt-get install cmake-curses-gui

Compiling took about 9 hours on a B+. It took 64 minutes on a Pi2 with 'make -j 4'

After installing OpenCV 3.0.0, the code does run unmodified on a RPi B+ (except for saving out a separate .mp4 file)
It takes about 2 seconds per frame (1296x972 .mp4 input, downscaling in OpenCV to 640x480).
Swapping the same SD card between a B+ and an RPi2, using as input 13 second long, 10 MB 1296 x 972 25 fps .mp4 file, I see
B+: 567.6 seconds, RP2: 128.7 seconds, so the RPi2 has a 4.4x speedup over the B+ in this case.

If I take a 320x240 resolution .mp4 file as input, the code runs at 8.5 fps on a RP2, and detects the same objects, so that is more practical.

[jbeale]

Addendum: I did get a h264 format mp4 file output, by specifying the codec number instead of -1 (which popped up a codec selector window under Win7, but not on the pi):

Code: Select all

# codec = cv2.VideoWriter_fourcc(*'X264')  # not currently working
codec = 0x00000021                         # this works though
video = cv2.VideoWriter(voname, codec, 25, (xproc,yproc))  # open output file

and combining output with input video, using

Code: Select all

MP4Box -cat file1.mp4 -cat file2.mp4 -new mergedFile.mp4

I generated this entirely on the Pi: https://www.youtube.com/watch?v=CCFa2ja ... e=youtu.be

[jbeale]

Here is a slightly different approach, using GaussianBlur, absdiff, and findContours (hopefully more CPU-efficient than the Mixture-Of-Gaussian and blob detection approach). The hard part is automatically finding background images that do not contain moving elements; and also if there is more than one new object in frame, matching up the same contour to the same object across frames. So far is working pretty well for cars, but more variable for pedestrians. https://www.youtube.com/watch?v=unMbtizfeUY

My code is here: https://github.com/jbeale1/OpenCV/blob/ ... motion3.py
it is an extension of Adrian Rosebrock's example at
http://www.pyimagesearch.com/2015/05/25 ... nd-opencv/

Here is a plot of measured X velocity (pixels per frame, not any real-world unit) vs X position, for a bunch of cars passing by on the road. Most of these cars are moving at a fairly constant speed in mph, but you can see cars towards the left hand side have a lower apparent velocity as seen by the camera, because they are farther away. The velocity drops off at the edges because I am tracking the center of the box, and as the car moves partially offscreen the center of the (visible) box moves towards the trailing edge as it gets smaller. Positive velocity means the car travels left to right. With some curve fitting to compensate for the perspective effect, I could use this to get a reasonable estimate of actual car speed. The detector is not perfect so there are some wrong points in places. One car slowed to a stop to park (Vx = 0) around x=135.



https://plus.google.com/photos/10992823 ... 0342205185

[jussi24]

Hi jbeale,

this is an interesting matter. Maybe openCV with the pi could be an entrance to the amazing features of OpenCV itself (still lying in the dark of unknowing).
Do I understand right that solving a problem, let's say detecting cars, is always a special solution for a special environment? In your case the cars are detected by their movement not by shape or some characteristic features, right?
Because of the limited power of the PIs I wonder if a PI could just serve as a webcam (raspimjpeg, that we both follow very interested ) and processing is made on a more powerful platform. (How) is it possible to connect an CV-algorithm to a distant IP-video source?

Regards

[jbeale]

... the cars are detected by their movement not by shape or some characteristic features, right?

That is right, my program detects anything that moves and is large enough; not just cars. OpenCV can recognize shapes and features, for example it can recognize faces, but that is slower.
https://www.raspberrypi.org/facial-reco ... era-board/
viewtopic.php?f=38&t=85755

Because of the limited power of the PIs I wonder if a PI could just serve as a webcam- (How) is it possible to connect an CV-algorithm to a distant IP-video source?

I am doing my OpenCV testing on a Win7 laptop because it is faster than the Pi. I just have the Pi recording .mp4 video (using the RPi Cam Web Interface software), transferring the files to the laptop, and processing there. You could also do real-time streaming like an IP camera, try to google "raspberry pi video streaming".

[Eclectic_Energy]

Hi jbeale

Wow this is really cool.
Do you think this code can be adapted to work with a live feed from the picam?

[jbeale]

I put two round spots 1 m apart on my car, and used OpenCV to track the distance between them, as a function of X position in the frame. (code here: https://github.com/jbeale1/OpenCV/blob/ ... cleFind.py )


That enabled me to calculate the x-axis scale in pixels per meter, across the screen (at least for the specific location of the car). Then I could get a better estimate of car speed, although it is less accurate on the left-hand side of the image (x < 600) where perspective effects are a stronger function of side-to-side position of the car on the road. Plot below shows one day's worth of traffic; each car is one line. Positive speed means travel left -> right. The curlicue crossing y=0 is not an error; the mail truck made a U-turn.

[pageauc]

I liked your video
Looked at your code but decided to do my own for just x,y real time motion tracking (well slower than real time but still good enough for what I need for robotics. Frame rate is fast enough to track slower moving objects.

Here is my demo program posting with links to my github code
viewtopic.php?p=790082#p790082

Here is my YouTube demo video https://youtu.be/09JS7twPBsQ

Here is python picamera code https://github.com/pageauc/motion-track ... n-track.py

The code is a demo (written in a few hours) but I intend to rewrite the code and integrate with my python multiprocessing and threaded robot program. The current version uses face detection tracking, but I found it had trouble with the back of heads and detecting faces in low light or profile unless you cycled through several cascades.

I decided to try generic motion tracking without the need for cascade files. This works better in low light and does not need a face to work. Might incorporate both so it can try face detection tracking or cascades under certain conditions. I can use the pi to detect light level to help decide when to try face detection. viewtopic.php?p=774618#p774618

Here is a link to some stuff on my deep learning using RPI B1 and B2 viewtopic.php?p=786040#p786040. Still has trouble recognizing everyday objects and it takes a few seconds to analyse an image.

BTW I think you did a very impressive demo.

[gleega]

Thanks for posting this.

How does one best determine the optimal "min-area" for this to work reliably if one knows the average distance from camera to subject? Is there a mathematical way to derive this? It seems this does not work as well for objects that are moving in front of the camera at a relatively close distance (ie; 5-10 feet). Any input welcomed.

[jbeale]

One thing to be aware of, some of these algorithms do not detect anything until all edges of the object are completely inside the frame. In other words, if you want to detect a person walking by, nothing is detected if an arm or a leg or a head is even partially outside the field of view, as would often happen if the subject is close to the camera.

[gleega]

Good to know. Do you know which of the algorithms do this (ignore any object not wholly obtained within the frame)? Need to work around this somehow.

[pageauc]

Vehicle Speed Camera using OpenCV Motion Tracking. Program code Updated
Just a note in case you missed my forum post. I have updated my vehicle speed camera opencv, python, picamera app. Now works better. See details here. viewtopic.php?p=813535#p813535
YouTube video is here https://youtu.be/eRi50BbJUro
Github code is here https://github.com/pageauc/motion-track ... peed-track
Regards Claude ...

[pageauc]

Good to know. Do you know which of the algorithms do this (ignore any object not wholly obtained within the frame)? Need to work around this somehow.

I have cars in my speed camera that are partly out of frame. The motion method just looks for pixel differences between two images so if an object is moving you will normally get a leading and trailing edge. These can be contoured even if the main moving object is partially out of frame since the trailing edge pixel differences will still be detected. You will see this more clearly if you run the program with gui window on. The difference window will show changes due to motion. You could draw boxes around all contours to see this situation better.

One way to ensure a complete object is in frame might be to track the leading and trailing edges since they will move at the same rate. If you only see one edge then you can act accordingly. You can also check the contour ratio of h/w. If the contour changes dramatically and is near an edge then moving object might be going out of frame. A trailing or leading edge for objects moving left/right would be thin vertically. It comes down to filtering the data for the movement your want to track. This can be size, location, speed, dimensions, Etc.

[henk.snor]

I am really interested in this topic and the different codes. I tried to incorporate this in a situation in which I have a raspberry pi and camera module streaming to a linux desktop. I am already able to stream (h264 format) from the rpi to the desktop and see the video in a videoplayer (vlc) but I am still not able to proces the data with opencv like in the codes in this topic. I tried to modify the code using the h264 example
Post by jbeale » Wed Jul 01, 2015 8:21 pm
Addendum: I did get a h264 format mp4 file output

however I am not able to use h264 as an input. Does anyone how to do this?

[jbeale]

For the stuff I was doing, it was not streaming in real-time. I first converted the .h264 to .mp4 and then used that as input to OpenCV, for example

Code: Select all

MP4Box -fps 24 -add myvid.h264 myvid.mp4

[henk.snor]

Thank you for you response! Is it possible to do this with a live stream? Or is that impossible due to the way opencv analyzes data? I know that for a jpg stream opencv is able to do it.

[stderr]

Thank you for you response! Is it possible to do this with a live stream? Or is that impossible due to the way opencv analyzes data? I know that for a jpg stream opencv is able to do it.

You can do opencv on a series of still shots. So, for example, you could do your feature tracking on images taken every second or whatever. That will really reduce your processing load, which is a good idea because you want to be able to do whatever processing is needed at the resolution you need to find out if anything important is going on.

If you are tricky, you could monitor these tracked images for anything interesting and then save at full frame rate everything that happened from the point when the interesting material started til it ended. Then you could go back later and process the interesting things over a span of time. You would, of course, be alerted to the interesting things in real time, you'd just get more data for later. I'm thinking of a security camera where you want to know you are being intruded on and then later you want the full data for hopefully getting a face to attach to the threat.

[jennie23]

Hi Mr JBeale,

Sorry to ask, how can i get the recorded avi video? I am trying to run motion3.py, with this command:

'python motion3.py'

i have the streaming, but seems like there is no video recorded down. What to do to get the video output?

[jbeale]

This post is two years old and I haven't touched it since then. I assume you are using this code:
https://github.com/jbeale1/OpenCV/blob/ ... motion3.py

I do recall getting video output recorded but please note this was on a windows PC, not a R-Pi. The AVI video container format is associated with windows, on the RPi you more often see MP4 container formats. The recording is done by an OpenCV object. If you're using a different version of openCV than I am the syntax may have changed? Do you get any error messages? Did you set the correct location for the recording in the code? Otherwise, I don't know how to help.

[jennie23]

Hi...

i added in these lines:

fourcc = cv2.cv.CV_FOURCC(*'XVID')
videofilename = "./zvideo_%d.avi" %d
video = cv2.VideoWriter(videofilename, fourcc, 25, (xsp,ysp))

and the recording works...Thanks for your effort making the motion detection algorithm..your code is the one i found the most precisely in detecting. =)

[jbeale]

Thanks for that update and I'm glad it's working for you! I have been doing other things meanwhile, but now I'm curious to revisit this code.