Yggdrasil
Posts: 138
Joined: Sun Aug 26, 2012 8:45 pm

Blob extractor for motion detection

Mon May 05, 2014 11:53 pm

Hello,

I'm currently waiting on the raspivid update which provides the motion estimation vectors :) (http://www.raspberrypi.org/vectors-from ... stimation/ ).

Just to get an idea about the quality of this data, I've put together a blob detection algorithm (which will run on the RPi fast enough) and the juggling image frames of the blog post.
The graphical representation of the output looks like this:
Blob Map_screenshot_2.png
Blob Map_screenshot_2.png (5.24 KiB) Viewed 9317 times
Blob Map_screenshot_1.png
Blob Map_screenshot_1.png (25.78 KiB) Viewed 9317 times
Blob Map_screenshot_3.png
Blob Map_screenshot_3.png (14.38 KiB) Viewed 9317 times
Image 1 and 2 shows the founded blobs (with a size>X) for a fixed thresh. Image 3 shows the OpenCv-Window and an image of a modified algorithm which could be interesting for the parallel search after multiple thresh values. Image 4 (next post) shows the capability to reduce the blob detection on coarse images. (Without using any memory copy operations.)

I hope, I could develop a simple motion detection with this approach, but two questions are still open:
1. Exists a good set of parameters, which filtering the input data well?
2. Can blobs of different frames could be combined, to find traces of movements? I.e. use midpoints of splines as spline interpolation points and check the bending. (The usage of sets of blobs reduce the data quality, but compresses the data for faster analyzing).
Moreover, the detection can be used to pre-filter the image and limit expensive operations on subimages.

If you want try it out and i.e. find good parameters for a blob detection take a look on https://github.com/YggdrasiI/RPIMotionDetection
(Note: I used an OpenCV-Lib with QT-Support. Comment out the button creation lines, I you do not want compile OpenCV with QT dependencies, see http://docs.opencv.org/modules/highgui/ ... eatebutton )

Edit, May 2014: Binary release for testing purposes. Requires RPi + RPi Camera.
Some notes: http://www.raspberrypi.org/forums/viewt ... 17#p556717

Regards
Yggdrasil/Olaf Schulz
Last edited by Yggdrasil on Wed May 28, 2014 7:56 pm, edited 3 times in total.

Yggdrasil
Posts: 138
Joined: Sun Aug 26, 2012 8:45 pm

Re: Blob extractor for motion detection

Mon May 05, 2014 11:54 pm

Last attachment.
Blob Map_screenshot_4.png
Blob Map_screenshot_4.png (5.98 KiB) Viewed 9316 times

Yggdrasil
Posts: 138
Joined: Sun Aug 26, 2012 8:45 pm

Re: Blob extractor for motion detection

Sun May 18, 2014 10:34 pm

I've uploaded a new version and add a new program in apps/raspicam. This version of raspicam uses the opengl backend to print out
A) the normal camera output
B1) the motion vector amplitudes (comment out eval_ids call in main.cpp )
B2) A subset of all detected areas/blobs.
C) Green rectangles for all areas which was detected over several frames.

The attached images shows a sample screenshot of the framebuffer. The blob detection is decoupled from the main thread, but I'm not satisfied with the OpenGL calls, which blocks the main thread. (I currently do not know how to solve this problem.)
Edit: Well, the OpenGL-calls are already outsourced into an other thread, see raspitex_init().

Example command to start the app:
./raspivid -o /dev/null -x /dev/null -t 0 --preview '0,0,800,600' \
--gl --glwin '0,0,800,600' --glscene motion


Ethanol100 encourage in an other thread to develop a framework for the inline motion vectors. Please do not expect my code comply this requirement…

Best regards
Yggdrasil
Attachments
detection_v2.png
detection_v2.png (61.05 KiB) Viewed 9033 times
Last edited by Yggdrasil on Thu May 22, 2014 9:03 am, edited 1 time in total.

steve_gulick
Posts: 31
Joined: Wed Jul 18, 2012 12:06 pm
Contact: Website

Re: Blob extractor for motion detection

Mon May 19, 2014 11:47 pm

Thank you for publishing your code. It compiled and ran fine. Looks very interesting!
Steve

Yggdrasil
Posts: 138
Joined: Sun Aug 26, 2012 8:45 pm

Re: Blob extractor for motion detection

Tue May 20, 2014 7:02 pm

Thanks for your feedback, Steve. :)

The last two days I try to add a tiny pong application, but a frustrating OpenGLES issue prevent me to move forward :-(
I've open a thread in the graphics section (http://www.raspberrypi.org/forums/viewt ... 07&start=0 ) and hope that someone knows the solution with my current shader problem.

Yggdrasil
Posts: 138
Joined: Sun Aug 26, 2012 8:45 pm

Re: Blob extractor for motion detection

Sun May 25, 2014 4:50 pm

As second test application I've implement a tiny pong game :)
The flying raspberry can be influenced by movings nearby the left or right border.
The game runs in HD resolution with approx. 30fps. Look at the start script if you want use a lower resolution.

Compiling (without OpenCV) and start:

Code: Select all

mkdir bulid
cd build
cmake -DWITH_OCV=0 ..
make
cd apps/pong
./start
pong1.jpg
pong1.jpg (18.51 KiB) Viewed 8684 times
pong2.jpg
pong2.jpg (17.48 KiB) Viewed 8684 times
There are many space left for optimization, i.e
- fancy video output filters and winning screen, (Edit: Now, some filters was added. The video output change after some time if the option --pongCycleShaders is set.)
- smother animations due redrawing nearby the ball,
- more accurate motion detections & filtering,

but I think it's also funny in the current state. (And I do not plan to optimize it because I plan to use it only as test app.)

Regards Yggdrasil

Yggdrasil
Posts: 138
Joined: Sun Aug 26, 2012 8:45 pm

Re: Blob extractor for motion detection

Tue May 27, 2014 12:17 am

Currently I'm working on improvements of the blob tracking class.
To improve the quality of the input motion vectors, I switched from 1-norm to 2-norm, now. If somebody is interested in an efficient evaluation of sqrt(a*a + b*b) on integer values take a look into https://github.com/YggdrasiI/RPIMotionD ... am/norm2.c

There is still a little improvement possible because all inline motion vectors are even(?!).
In algorithm 3 I used two look-up maps for f(x)=x^2 and g(x)=sqrt(x), But if 4 | a*a + b*b
is always true this look-up maps can reduced on the halve size.
Moreover, the assembler variant doesn't works correct for -O3 flag. (Fixed)

Regards YggdrasiI

Yggdrasil
Posts: 138
Joined: Sun Aug 26, 2012 8:45 pm

Re: Blob extractor for motion detection

Tue May 27, 2014 8:35 pm

If somebody is interested but fear the compiling… I've uploaded a binary release on Github.

Edit:
• Please the --hflip option from the startup skript if the image was flipped. Also note the arguments low, med, and high for the startup resolution. The resolution will not respect your monitor resolution.
• Users wrote that the applications randomly crashs. Syncing your filesystem before you start the apps to be on the secure side.
• The ball in the pong app moves after two movements was detected. The ball can be reflected nearby the left or right border. Your movement influence the vertical movement of the ball, but find out this on your own!
Don't forget: The application recognize movements. Motionless hands will be ignored.
Last edited by Yggdrasil on Wed May 28, 2014 4:13 pm, edited 2 times in total.

User avatar
DougieLawson
Posts: 36571
Joined: Sun Jun 16, 2013 11:19 pm
Location: Basingstoke, UK
Contact: Website Twitter

Re: Blob extractor for motion detection

Tue May 27, 2014 9:01 pm

Hmmm, that was interesting, it happily started playing pong (despite there not being enough light) but then the whole system became unresponsive (couldn't ssh, couldn't login with kbd) so I hit it with a jumper on the P6 header.
Note: Having anything humorous in your signature is completely banned on this forum. Wear a tin-foil hat and you'll get a ban.

Any DMs sent on Twitter will be answered next month.

This is a doctor free zone.

Yggdrasil
Posts: 138
Joined: Sun Aug 26, 2012 8:45 pm

Re: Blob extractor for motion detection

Tue May 27, 2014 9:58 pm

DougieLawson wrote:Hmmm, that was interesting, it happily started playing pong (despite there not being enough light) but then the whole system became unresponsive (couldn't ssh, couldn't login with kbd) so I hit it with a jumper on the P6 header.
Hi DougieLawson, thanks for your comment.

I assume a power issue.
Which resolution did you use during the game? GPU+Usb peripherals+Camera consumes a lot of power.
Even my 2A power supply made trouble. If I use HD resolution and starts the game, my monitor sporadicly went black for a second (lost hdmi signal?!).
It could be run stable if you reduce the monitor resolution (i.e 'tvservice -e "DMT 9"') and
starts the game in 800x600 via './pong.start low'

Which value did you used for the GPU memory split?



Regards Yggdrasil

User avatar
DougieLawson
Posts: 36571
Joined: Sun Jun 16, 2013 11:19 pm
Location: Basingstoke, UK
Contact: Website Twitter

Re: Blob extractor for motion detection

Tue May 27, 2014 11:02 pm

It's not a power issue. That RPi runs from a CPC power supply, it's normally rock solid and usually runs 24*7.
Memory split is gpu_mem=128M, nothing else, no overclocking, nothing.

I didn't do any set-up, I just started your pong script.
Note: Having anything humorous in your signature is completely banned on this forum. Wear a tin-foil hat and you'll get a ban.

Any DMs sent on Twitter will be answered next month.

This is a doctor free zone.

Yggdrasil
Posts: 138
Joined: Sun Aug 26, 2012 8:45 pm

Re: Blob extractor for motion detection

Wed May 28, 2014 7:43 am

Ok, if power issues aren't the reason I has currently no idea why it crashes. The file-system hopefully wasn't corrupted on your SD card….

I will remove my changes from the config.txt will and test again if it's still running stable on my RPi.
Feedback from other testers are appreciate, too.

BerryPicker
Posts: 177
Joined: Tue Oct 16, 2012 3:03 pm
Location: The East of England

Re: Blob extractor for motion detection

Wed May 28, 2014 11:15 am

Yggdrasil wrote:Feedback from other testers are appreciate, too.
Thank you for providing the tar.gz file and for sharing your work. You anticipated my need to avoid compiling, and the result is an instant fun study experiment.

At the console I made a folder called MotionVectors in my home directory, changed directory to it, copied the tar.gz to it, and unzipped the contents there. I then changed directory to RPiMotionDetection_v1.1 and ran your demos; using CTRL C to exit.

1st demo was run with the command ./motion.start. The picture occupies the top left of the screen, it is inverted, it has an overall blue wash, and is stretched virtically (compressed horizontally). Left alone (~42 fps) with nothing moving the picture 'blinks' at a rate as if affected by background nuclear radiation! When movement is present, shadows are blobbed as readily as moving objects. My hands seem to form blobs more readily than rigid moving objects. I'm surprised to see such a range of colours in the little pixels; I would expect to see the little pixels to be of similar colour when associated with different parts of the same moving object. Perhaps this is why the big green blobs form less readily than I expected. Occasionally the program terminates by itself reporting Unhandled fault:alignment exception.

2nd demo was run with the command ./pong.start. The picture is full screen, also inverted, and zoomed in to just part of the camera's vision. Large sized numerals (the score?) appear as watermarks on the picture (partly off screen). The rasperry momentarily also goes off screen at the bottom of the picture (top of the scene).

Edit. By making changes to the files motion.start and pong.start l have managed to put the pictures the right way up, and to remove zoom and distortion. Pong now looks right and play is even better!

My wish for motion.start development is that the blobs detected have a colour depending upon their direction and speed of motion as in the OpenCV Optical Flow example. http://opencv-python-tutroals.readthedo ... icalfb.jpg

Yggdrasil
Posts: 138
Joined: Sun Aug 26, 2012 8:45 pm

Re: Blob extractor for motion detection

Wed May 28, 2014 4:00 pm

BerryPicker wrote:1st demo was run with the command ./motion.start. The picture occupies the top left of the screen, it is inverted...
Oh, I messed up with the rotating and flipping arguments of raspivid. The --hflip argument in the script turned it upside down. --vflip should be the right one. (If somebody want to know how such an obviously issue can occur: I live in Australia My camera is mounted with 90 degrees rotation. ;) )
it has an overall blue wash, and is stretched virtically (compressed horizontally). Left alone (~42 fps) with nothing moving the picture 'blinks' at a rate as if affected by background nuclear radiation! When movement is present, shadows are blobbed as readily as moving objects. My hands seem to form blobs more readily than rigid moving objects. I'm surprised to see such a range of colours in the little pixels; I would expect to see the little pixels to be of similar colour when associated with different parts of the same moving object. Perhaps this is why the big green blobs form less readily than I expected. Occasionally the program terminates by itself reporting
A good hint. I should give you a short description of the usage and graphical output. :)
I've set up the parameter of this binary to display the two motions with the longest duration. If you move your hands and both are marked with green rectangles, you can move the rest of the body, but still following the
hands.
(This stabilization uses a tiny trick which I can not describe in one sentence.)

The video is overlay-ed with three information:
1. The h264 video encoder delivers motion estimation vectors (IMV). The norm of this vectors will be used to
form areas with the same movement speed. The set of this areas will be filtered and printed out as flickering
colours on the screen. For a better distinction of areas this values (area ids) was mapped into the rgb space. This data switched with each video frame. (Well you could also displayed the IMV data but this currently require recompilation....)

2. Each frame contains many errors. To eliminate this errors a tracker object collect the areas of several
frames. If the tracker decide that the trace of a movement was stable over N frames it will mark the trace as active. The big green rectangles marks the last known position of a movement. (It's the bounding box of an area from (1)).

3. To prove that the green areas not simply marking objects with a very restricted live time, I've added the
drawing of a movement history. Currently, it represents the midpoints of the bounding boxes for each frame.
As you can see, this polygonal line isn't smooth. Thus is not optimal for the detection of movement directions. I'm planning to switch from bounding box midpoints to the barycenters of detected areas. I assume this would stabilize the data.

If you wondering why hand shaking isn't recognized immediately: This is just the detection filter of the tracker. The history line shows where the tracking begins.
My wish for motion.start development is that the blobs detected have a colour depending upon their direction and speed of motion as in the OpenCV Optical Flow example. http://opencv-python-tutroals.readthedo ... icalfb.jpg
Yes, such graphical representation (and detection) would be much more nicer. A high quality estimation of the detected motions is still missing in my application. Probably this detection can be a direct result of an direct analysis of the IMV data of 1-3 frames. Thats differs a little bit from my approach.
It's not on my road map at the moment, but can be an side-effect output ;)
Currently I'm targeting simple motion detection. I hope this can be integrated into a motion control of XBMC
or will be used for some games.
Unhandled fault:alignment exception.
I fear the hunting of this bug :| Hopfully I can reproduce this sometime.
Last edited by Yggdrasil on Wed May 28, 2014 7:33 pm, edited 1 time in total.

BerryPicker
Posts: 177
Joined: Tue Oct 16, 2012 3:03 pm
Location: The East of England

Re: Blob extractor for motion detection

Wed May 28, 2014 7:22 pm

Yggdrasil wrote:I should give you a short description of the usage and graphical output. :)
Thank you for these explanations. They help in understanding what's happening under the bonnet and making sense of what is seen. A fun learning experience!

User avatar
DougieLawson
Posts: 36571
Joined: Sun Jun 16, 2013 11:19 pm
Location: Basingstoke, UK
Contact: Website Twitter

Re: Blob extractor for motion detection

Wed May 28, 2014 7:29 pm

Yggdrasil wrote:
DougieLawson wrote:Hmmm, that was interesting, it happily started playing pong (despite there not being enough light) but then the whole system became unresponsive (couldn't ssh, couldn't login with kbd) so I hit it with a jumper on the P6 header.
Hi DougieLawson, thanks for your comment.

I assume a power issue.
At some point in the near future I'm going to swap the power supply with my other RPi (which doesn't have a camera). Today I was trying to capture five minutes of H264 video with raspivid and the RPi got into a solid hang.

I'll give your pong stuff another go after I try that swap.
Note: Having anything humorous in your signature is completely banned on this forum. Wear a tin-foil hat and you'll get a ban.

Any DMs sent on Twitter will be answered next month.

This is a doctor free zone.

Yggdrasil
Posts: 138
Joined: Sun Aug 26, 2012 8:45 pm

Alignmend exception crash

Wed Jun 04, 2014 12:31 pm

Hi,

I've found the reason for the spontaneous crashes:

Code: Select all

[613296.539867] Alignment trap: not handling instruction e1931f9f at [<0003f728>]
[613296.539925] Unhandled fault: alignment exception (0x011) at 0x0000001f
[/s]
I found something about this problem here: http://www.raspberrypi.org/forums/viewt ... 63#p147763
It's possible to disable a compiler optimization, but this would slows down the program. :roll:
A better solution should be a change on the affected variable/command. Does anyone know how I can find the variable/command which are in relation with the crash?

BerryPicker
Posts: 177
Joined: Tue Oct 16, 2012 3:03 pm
Location: The East of England

Re: Blob extractor for motion detection

Mon Aug 25, 2014 11:09 am

I've been experimenting with changes to the executable 'motion.start' and notice that if I slow the framerate, e.g. by including '--framerate 7', the displayed picture shows with 2 sets of motion vectors (macroblocks) visible. I had only expected to see just 1 set of motion vectors. I mention this just in case it may be useful in further development.

Return to “Camera board”