caerandir
Posts: 131
Joined: Tue Dec 18, 2012 11:26 am
Location: Bonn, Germany

PoC: Poor man's high framerate full HD motion detection

Wed Nov 13, 2013 9:28 pm

Hi all,

I see a lot of posts regarding motion detection, but all suffer from the fact that the tiny CPU of the Pi needs to do a lot of stuff which actually drops down the framerate to something of the order 2 fps or requires to scale down the images considerably. My idea was that the powerful GPU is specifically designed to do all the stuff required for motion detection, i.e. finding out what has changed between two frames in order to do the video compression.

My first idea was to look into the OpenCV libraries to see if one could use the GPU features - like having the camera put an image in buffer#1, next image in buffer#2 and then let the GPU do the math. I think it is possible, but unfortunately I cannot invest enough time to dig into it (perhaps someone can pick up that path?). So i "nudged" jamesh into implementing variable bitrate (VBR) into raspivid (see here: http://www.raspberrypi.org/phpBB3/viewt ... 43&t=59963) - let him be praised!

And now comes the poor man's motion detection! It works like this:
  • Let raspivid capture a full HD video with VBR
  • Monitor the file size: If there is nothing happening in the image, the changes in filesize will be small - if motion comes into the image, filesize will change much more, since bitrate must go up to account for the image changes.
  • If the filesize change exceeds some threshold, trigger the desired action.
Actually, this principle is that simple I was able to implement a proof of concept into a few lines of bash script:

Code: Select all

#!/bin/bash
#
# Poor man's high framerate motion detection by caerandir (Proof of concept version)
#
# Adjust Threshold to your conditions:
Threshold=100
#
# Start recording video...
raspivid -o test.mpeg -t 1000000 -b 0 -qp 25 -e &
sleep 1
#
# Now run in circles and monitor file size change
LastChange=0
for (( ; ; ))
do
	# Get filesize
	LSoutput=$(/bin/ls -s ~/test.mpeg)
	FileSizeChange=($LSoutput)
	# Calculate difference
	Difference=$(($FileSizeChange-$LastChange))
	LastChange=$FileSizeChange
	echo $Difference
	if [ $Difference -gt $Threshold ]; then
		echo "Motion detected!"
		# Fill in here whatever you want to happen
	fi
	sleep 1
done
# Stop the thing with Ctrl+C
You will need to tweak Threshold to your conditions, since the file size change strongly depends on the general scene (although I do not really understand why...). And: Use good lighting - if the image gets noisy, the difference between no motion and a lot of motion is not too well discernable.

The advantage of this method is of course that you have a full HD video of the moment the motion actually happened, with all the detail and a framerate that will not let you miss the actual thing happening. In the section of the script that is triggered by motion you could possibly upload the video to a webserver or send an alarm e-mail.

However, the script is not more than a proof of concept - one would need to improve this by recording chunks of video and delete old stuff in order to not fill up the filesystem. Also, SD card use is quite heavy - perhaps one should attach a harddisk or something that does not degrade with permanent use. Or one could try and use a RAM disk or a FIFO buffer. And: Some automatic baselining would help, since already changes in the lighting cause the Threshold to "wander".

I hope this idea triggers some creativity ;-) Still I think the proper way would be to use OpenCV directly...

Hope you like this :-)

Caerandir

User avatar
jbeale
Posts: 3578
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: PoC: Poor man's high framerate full HD motion detection

Wed Nov 13, 2013 9:55 pm

This is a great idea! I was just looking for something like this today.

The logical thing is to write the .h264 file into ramdisk (like /run/shm/ ) and copy the file (or portion of the file?) onto SD, etc. only when the change threshold is detected. The ramdisk has limited size (25 MB ?) so you will need to record fairly short videos, before you have to erase and restart.

I think raspivid uses a few seconds for hardware setup and autoexposure setting every time it is stopped and restarted. So you will miss at least a few seconds in between each video file.

The ideal thing would be for raspivid (or similar application) to have a "ping-pong" file output feature. That is write (for example) 10 seconds of video to file A, close file A, and immediately open file B and continue writing, without resetting the camera device. After another 10 seconds, close file B and start (over)writing file A again, and so on. The idea is that if there was anything interesting in file A, it would have been already copied over and saved elsewhere before it gets erased.

Maybe it is straightforward to implement such a thing in the existing raspivid code (?)

poing
Posts: 1131
Joined: Thu Mar 08, 2012 3:32 pm

Re: PoC: Poor man's high framerate full HD motion detection

Wed Nov 13, 2013 10:05 pm

Great post!

I understood that you can both save to a file and read from it at the same time on Linux. Can't you delete while recording as well? Then you could delete frames from the file in /run/shm while recording and keep it at a fixed size.

I think the method in the OP could also be used to find optimum focus when using other (DSLR) lenses, since the file size will max out when the image is in focus.

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5398
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: PoC: Poor man's high framerate full HD motion detection

Wed Nov 13, 2013 10:07 pm

The idea is sound. I'd also suggest modifying raspivid, and keeping a queue of the last N frames in RAM (theres plenty of RAM for several seconds of video)
Keep track of which frames returned are IDR and which are P frames.
Measure the sizes of the P frames (ignore the IDR).
If a P frame is above a size threshold (perhaps M times the average P frame size) then trigger record mode.
If a P frame is below a size thresold then end record mode.

A separate thread writes frames from the queue to sdcard while record mode is enabled. Remember to only start writing from an IDR frame.

You might want to only trigger record mode (and end of record mode) if a sequence of P frames are large - a single frame may be larger after an AGC adjustment.

SnowLeopard
Posts: 106
Joined: Sun Aug 18, 2013 6:10 am

Re: PoC: Poor man's high framerate full HD motion detection

Wed Nov 13, 2013 10:18 pm

You need a simplified version of Pipe Viewer (http://www.ivarch.com/programs/pv.shtml) which would trigger a script or signal a running process when the measured bitrate changes. Also, while nothing is moving, it would dump the video and only when the bitrate changes would it save the video.

User avatar
jbeale
Posts: 3578
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: PoC: Poor man's high framerate full HD motion detection

Thu Nov 14, 2013 2:17 am

For what it's worth, I tried this quick test shell script, recording two 5-second files into the /run/shm ramdisk. I was recording my stopwatch and looked at the last frame of file A and first frame of file B (first using 'avconv -i A.h264 A%04d.jpg' to convert into separate JPEG files for convenient viewing). There is exactly 1 second of real-time gap between the end of A and start of B files, and the first frames are dark as autoexposure hasn't settled. This just tells you what you already know- it's better to build this into an application that maintains the camera process active, rather than try to hack it together as a shell script.

Code: Select all

A="/run/shm/A.h264"
B="/run/shm/B.h264"

sudo raspivid -o $A -t 5000 -qp 30 -rot 180 -fps 12
LSoutput=$(/bin/ls -s $A)
FileSize=($LSoutput)
echo $FileSize
sudo raspivid -o $B -t 5000 -qp 30 -rot 180 -fps 12
LSoutput=$(/bin/ls -s $B)
FileSize=($LSoutput)
echo $FileSize

wibble82
Posts: 66
Joined: Sun Jan 27, 2013 5:06 pm
Contact: Website

Re: PoC: Poor man's high framerate full HD motion detection

Thu Nov 14, 2013 9:20 am

I did have an idea on how to do full framerate HD motion detection very effectively, and was tempted to try it out now that we have the direct to gpu path from mmal.

Basically simple motion detection falls into 2 steps
- first, calculate a movement value per pixel that indicates how likely the pixel is to have a moving object in. typically this is done through some persistant metric comparing the pixel colour to previous pixel colours, though some algorithms get pretty complex.
- second, condense the pixel grid into a single value that indicates the likelyhood of movement existing in this frame

The first step is pretty easy to implement on the gpu. It'd probably have something to do with a rolling average to calculate the background colour, and a constant comparison of that to the current colour. As I said, there's a fair few techniques that vary in complexity, but they can generally be implemented with a simple shader that operates on just a few images per frame.

The next step is a little more complex to do on the gpu, but can be achieved using a technique called pyramidal image collapsing. This basically involves writing a shader that collapses an image into one of half the size, where each new pixel is some value based on the 4 pixels it represents in the quadrant in the larger image. This can be repeated multiple times to get down to a very small texture that can easily be analysed on the cpu. A simple implementation would be a pyramidal collapse that calculated a binary 1 or 0 in a pixel that indicated whether motion had been detected in the larger quadrant. The next collapse would build another set of 1 or 0 that corresponded to the quadrant in the previous image, or the 16 pixels it represents in the original. Again, this can be cleverer (and if you're willing to represent a floating point pixel value by storing it across the 4 rgba components, very accurate).

I was thinking of implementing this as a test for my gpu camera api, but I've not finished the api yet. Food for thought though :)

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 24925
Joined: Sat Jul 30, 2011 7:41 pm

Re: PoC: Poor man's high framerate full HD motion detection

Thu Nov 14, 2013 9:52 am

With regard to chopping the stream up in to different files, I have an outstanding PR that allows the setting of inline headers in the stream. This means that you can stream, then change files without stopping the stream, and the stream will still work. In effect the header that specifies width and height plus some other bits, that is usually only applied at the start of the stream, is sent on (I think) each i-frame. It's only a few bytes to doesn't affect size too much.

I've got a item on my list to enable to chopping up of video in to multiple files at point of recording.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I own the world’s worst thesaurus. Not only is it awful, it’s awful."

mikerr
Posts: 2804
Joined: Thu Jan 12, 2012 12:46 pm
Location: UK
Contact: Website

Re: PoC: Poor man's high framerate full HD motion detection

Thu Nov 14, 2013 9:57 am

To avoid gaps you could use the OP's script to get the motion activity times (start/stop),
then extract that section from the video ( no need to stop raspivid).

[edit] crossed over with james' post do do it at source.

User avatar
jbeale
Posts: 3578
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: PoC: Poor man's high framerate full HD motion detection

Thu Nov 14, 2013 3:00 pm

jamesh wrote:I've got a item on my list to enable to chopping up of video in to multiple files at point of recording.
That would be a very useful feature! Besides motion detection, also handy for people doing car dashboard cameras, for example. It is convenient to have sets of smaller files without gaps in-between, and having to reset the auto-exposure settings.
wibble82 wrote:I did have an idea on how to do full framerate HD motion detection very effectively, and was tempted to try it out now that we have the direct to gpu path from mmal.
Please do feel free to try this out, when you get a chance! :-) You might also consider stopping the pyramid before reaching a single pixel, leaving for example a 4x4 matrix. This allows detecting motion in just selected segments of the full frame (avoid area of plants blowing in the wind, etc.) If it ends up too slow, you could also try working only on "Y" channel, instead of YUV or RGB, or binning and/or subsampling first. Processing even just a small window within the full frame could still be quite useful.

towolf
Posts: 421
Joined: Fri Jan 18, 2013 2:11 pm

Re: PoC: Poor man's high framerate full HD motion detection

Thu Nov 14, 2013 6:05 pm

Looking at the motion vectors in the H264 is more promising than just bit allocation. Just more image noise after an ISO switch would inflate the bitrate.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 24925
Joined: Sat Jul 30, 2011 7:41 pm

Re: PoC: Poor man's high framerate full HD motion detection

Thu Nov 14, 2013 8:54 pm

Indeed it would, but on average this would definitely work as a poor man's motion detector, but with more false positives than the rich mans version.

News on the stream splitting - have managed to split the stream to segmented files. Unfortunately everything after the first file fails to play...will get the streams analysed tomorrow to see what's up.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I own the world’s worst thesaurus. Not only is it awful, it’s awful."

BorisS
Posts: 15
Joined: Fri Oct 11, 2013 7:08 pm

Re: PoC: Poor man's high framerate full HD motion detection

Fri Nov 15, 2013 8:33 am

Hi James,
could you also introduce an additional filename naming scheme together with this feature? Something like %mINTERGER that would work as the %d pattern in raspistill, except that segment/picture filename number is computed modulo given integer. This would allow to toggle (modulo 2) filenames or to have a circular flight-recorder mode out of the box.

lagurus
Posts: 46
Joined: Wed Aug 07, 2013 8:02 am

Re: PoC: Poor man's high framerate full HD motion detection

Fri Nov 15, 2013 9:12 am

jamesh wrote: News on the stream splitting - have managed to split the stream to segmented files. Unfortunately everything after the first file fails to play...will get the streams analysed tomorrow to see what's up.
I had the same issue. It was needed to add on each file start something like "header" which comes from camera with flags MMAL_BUFFER_HEADER_FLAG_CONFIG (they are in fact two - first 20bytes and second 8bytes - in my case) .
Whenever want to create new file, is needed to add these 28bytes on beginning and certainly start with key frame.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 24925
Joined: Sat Jul 30, 2011 7:41 pm

Re: PoC: Poor man's high framerate full HD motion detection

Fri Nov 15, 2013 9:36 am

lagurus wrote:
jamesh wrote: News on the stream splitting - have managed to split the stream to segmented files. Unfortunately everything after the first file fails to play...will get the streams analysed tomorrow to see what's up.
I had the same issue. It was needed to add on each file start something like "header" which comes from camera with flags MMAL_BUFFER_HEADER_FLAG_CONFIG (they are in fact two - first 20bytes and second 8bytes - in my case) .
Whenever want to create new file, is needed to add these 28bytes on beginning and certainly start with key frame.
Hmm, I was expecting that to be done automatically since I have inline headers turned on.

ref: Modulo numbers - good idea. I'll have a think about the best way of doing that.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I own the world’s worst thesaurus. Not only is it awful, it’s awful."

Twinkletoes
Posts: 210
Joined: Fri May 25, 2012 9:44 pm

Re: PoC: Poor man's high framerate full HD motion detection

Fri Nov 15, 2013 10:18 am

This is a clever idea and does work in practice - I had a friend who looked for correlation in stock prices by MP3 encoding the history of two prices as the left and right channels. A smaller file indicated better correlation. This was actively used at a UK bank.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 24925
Joined: Sat Jul 30, 2011 7:41 pm

Re: PoC: Poor man's high framerate full HD motion detection

Fri Nov 15, 2013 10:38 am

jamesh wrote:
lagurus wrote:
jamesh wrote: News on the stream splitting - have managed to split the stream to segmented files. Unfortunately everything after the first file fails to play...will get the streams analysed tomorrow to see what's up.
I had the same issue. It was needed to add on each file start something like "header" which comes from camera with flags MMAL_BUFFER_HEADER_FLAG_CONFIG (they are in fact two - first 20bytes and second 8bytes - in my case) .
Whenever want to create new file, is needed to add these 28bytes on beginning and certainly start with key frame.
Hmm, I was expecting that to be done automatically since I have inline headers turned on.

ref: Modulo numbers - good idea. I'll have a think about the best way of doing that.
OK, got this working now (needed to switch files on receipt of the header, not a new I-frame, since the header precedes the I-Frame), including the modulo stuff, so can now produce multiple file seamless video capture. I'll push it later today - have to commentate on the Cambridge Fun Run first! It's children in need - contributions welcome!
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I own the world’s worst thesaurus. Not only is it awful, it’s awful."

caerandir
Posts: 131
Joined: Tue Dec 18, 2012 11:26 am
Location: Bonn, Germany

Poor man's high framerate full HD motion detection v2

Sat Nov 16, 2013 11:02 pm

Hi all,

jamesh made a few adjustments to raspivid (see here: http://www.raspberrypi.org/phpBB3/viewt ... 43&t=60969) that allowed me to make my motion detection idea more realistic, and so here I post my now fairly extended shell script along with some explanations. But first: Many thanks to jamesh to making this possible!

The general idea stays the same: Use VBR to detect image changes due to the change in required bitrate. However, some adjustments of this idea were required: I realized that the file size of a recorded VBR video is not dominated by the recorded differences, but by the encoded full frames to which the differences are applied. As a result, a file containing motion may even be smaller than the tranquil video, if the scene that resulted from the motion is easier to encode. As an example, imagine recording your room: It contains furniture, pictures on the wall etc. that account for quite some bytes in the video. Now imagine that you suddenly put a piece of white paper in front of the lens: The image is now much less crowded and encoding is much easier, bitrate drops and filesize also. So, the file gets smaller although we all agree that quite some motion was recorded. I altered my script to account for deviations from the baseline in both directions. This now works pretty well, although there is one caveat: It may theoretically happen that the complexity of the tranquil scene and that during motion may result in very similar file sizes - motion will be missed in this case. That brings me back to my original idea: One should in principal only analyze image changes, not images by themselves. However, the current possibilities do not allow for that.

So my script now does the following:
  • Use raspivid to record chunks of one second long videos (thanks to the new segmentation feature in raspivid) to the ramdisk
  • Use a given number of videos to establish a baseline, i.e. the average filesize of those videos - this allows the baseline to adjust to slow changes in the image, like the sunlight wandering over time
  • Compare the most recent video to this baseline
  • If the relative difference exceeds a given threshold, either smaller or larger file, trigger motion handling
Currently two modes are available:
  • Surveillance mode: The baseline is established when no motion is detected. As soon as motion is detected, the baseline will be fixed. Only when no motion is detected any more, baseline will be adjusted again. The idea is, that you have a fairly static scene, like your entrence door, and if this changes, you want to record everything until the scene is back to it's original setup. If anything happens during motion that permanently alters the scene (like your door was opened but not shut again), recording will continue until your filesystem is full.
  • Baseline mode: Even during motion the baseline will be adjusted continuously. So even if the scene permanently changes during motion, at some point recording will stop. However, this may result in recording stopping while still activity is ongoing.
If motion is detected, the script starts recording the video onto a given filesystem in the format YY-MM-DD_hh-mm-ss_motion.h264 just by copying and appending the video chunks from the ramdisk.

I think the rest is pretty self explainatory, so here is the improved version - enjoy :-)

Code: Select all

#!/bin/bash
#
# Poor man's high framerate motion detection v2 - by caerandir
# 
# CONFIGURATION:
#
# Threshold: Filesize change (+/-) relative to the baseline that pulls the trigger (in %)

Threshold=10

# FilesToKeep: Number of files to derive the baseline from

FilesToKeep=10

# SurveillanceMode: 1: Do not adjust baseline during motion 0: Basline mode - always adjust baseline

SurveillanceMode=1

# CaptureDirectory: The directory on a non volatile memory to store scenes of motion

CaptureDirectory=/home/pi/motion

# END CONFIGURATION
#
# Start recording video into ramdisk
Basedir=/dev/shm
# Files in Loop: Numer of files for baseline plus one file for analysis plus one for current recording
FilesInLoop=$(($FilesToKeep+2))
# Adjust the raspivid to your needs. Keep: -qp, -sg, -wr and -t 0.
raspivid -o $Basedir/stream%02d.h264 -fps 30 -g 1 -sg 1000 -wr $FilesInLoop -t 0 -sa -100 -qp 25 -e &
# Wait for things to settle in
sleep 1.5s
#
# Now run in circles and monitor file size change. First cycle is for establishing the baseline.
echo "Calibrating..."
FirstCycleDone=0
RecordingFile=""
PostMotionCountdown=0
for (( ; ; ))
do
	# Get the video chunks sorted by date (most recent first)
	FilelistByTime=($(ls -c $Basedir/stream*.h264))
	FirstFile=${FilelistByTime[0]}
	if [ $FirstCycleDone -eq 1 ]; then
		# Baseline files are recorded
		if [ $FirstFile != $LastFirstFile ]; then
			# Another video chunk is finished - process it
			LastFirstFile=$FirstFile
			SizeOfLastCompletedVideo=$(stat -c%s ${FilelistByTime[1]})
			if [ $PostMotionCountdown -eq 0 ]; then
				# Continuously adjust baseline: Sum up all recorded video chunks and devide by number of chunks (i.e. average file size)
				FilesizeTotal=0
				for (( FileNumber=1; FileNumber<=$FilesInLoop; FileNumber++ ))
				do
					ProcessFile="$Basedir/stream$(printf '%02d' $FileNumber).h264"
					# Exclude the video chunk that is currently recorded, since it is smaller as the rest
					# Exclude the video chunk that is to be analyzed
					if [ $ProcessFile != $FirstFile -a $ProcessFile != ${FilelistByTime[1]} ]; then
						FileSize=$(stat -c%s $ProcessFile)
						FilesizeTotal=$(($FilesizeTotal + $FileSize))
					fi
				done
				Baseline=$(($FilesizeTotal / $FilesToKeep))
			else
				PostMotionCountdown=$(($PostMotionCountdown - 1))
			fi
			# Now compare the most recent completed video chunk to the baseline
			RelativeDelta=$(($SizeOfLastCompletedVideo * 100 / $Baseline - 100))
			if [ $(($RelativeDelta < 0)) -eq 1 ]; then
				# No abs() function in bash?
				RelativeDelta=$((-1 * $RelativeDelta))
			fi
			# Comment out following line for a more silent mode
			# echo "${FilelistByTime[1]} delta: $RelativeDelta% (Size: $SizeOfLastCompletedVideo - Baseline: $Baseline)"
			if [ $(($RelativeDelta > $Threshold)) -eq 1 ]; then
				# Motion detected - start actions
				if [ -z $RecordingFile ]; then
					echo "Motion detected!"
					# Start motion recording
					RecordingFile="$CaptureDirectory/$(date +%y-%m-%d_%H-%M-%S)_motion.h264"
					echo "Creating $RecordingFile"
					cp ${FilelistByTime[2]} $RecordingFile
					#
					# Fill in actions to start motion handling in here
					#
				else
					echo "Ongoing motion..."
					#
					# Fill in any actions during motion here
					#
				fi
				# Continue recording
				cat ${FilelistByTime[1]} >> $RecordingFile
				if [ $SurveillanceMode -eq 1 ]; then
					# Suppress baseline adjustment until motion files are overwritten by silence
					PostMotionCountdown=$FilesInLoop
				fi
			else
				if [ $RecordingFile ]; then
					echo "Motion ended."
					# Finalize recording
					cat ${FilelistByTime[1]} >> $RecordingFile
					#
					# Fill in any actions to finalize motion handling here 
					#
				fi
				RecordingFile=""
			fi
		else
			sleep 0.1s
		fi
	else
		# Wait until one cycle of video chunks was recorded in order to have good statistics for the baseline
		if [ $FirstFile == $Basedir/stream$(printf "%02d" $FilesInLoop).h264 ] ; then
			FirstCycleDone=1
			LastFirstFile=$FirstFile
			echo "Calibration finished, waiting for motion..."
		else
			sleep 0.25s
		fi
	fi
done
# Stop the thing with Ctrl+C

User avatar
jbeale
Posts: 3578
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: PoC: Poor man's high framerate full HD motion detection

Sun Nov 17, 2013 12:57 am

Nice! I'll try out your script. I posted my own motion-detecting script in JamesH's "new raspivid" thread http://www.raspberrypi.org/phpBB3/viewt ... 69#p454894 but yours is several steps more advanced.

This strategy is certainly a different kettle of fish than the other option, this uses nearly no CPU, vs 20-40% CPU for motion-mmal at moderate resolution.

caerandir
Posts: 131
Joined: Tue Dec 18, 2012 11:26 am
Location: Bonn, Germany

Re: PoC: Poor man's high framerate full HD motion detection

Sun Nov 17, 2013 1:03 pm

Hi folks,

here some real world adjustments. Playing around and tweaking parameters I found this to work very well and being very sensitive even to small movements:

Change the raspivid-line in the script to

Code: Select all

raspivid -o $Basedir/stream%02d.h264 -fps 30 -g 30 -sg 500 -wr $FilesInLoop -t 0 -sa -100 -qp 10 -e &
Allows for a Threshold of 1% (!) in indoor scenes, 3-4% outdoor with some movement due to gusty winds.

Another thing: I tested on a 512 MB Raspberry with Pi NoIR (hence the -sa -100 to get b/w images). When moving to my "normal" camera 256 MB Raspberry, the ramdisk was too small to accommodate 12 chunks of the above type. Just going to half framerate and down to 10 chunks solved the problem:

Code: Select all

raspivid -o $Basedir/stream%02d.h264 -fps 15 -g 15 -sg 500 -wr $FilesInLoop -t 0 -qp 10 -e &
You may wonder why now -sg 500 - this is because it seems that raspivid starts to look for the next full frame after that period. This sometimes results in videos of 2 seconds when using -sg 1000 because the full frame was just missed by the algorithm. -sg 500 starts looking after half a second, but the first full frame comes after one second, so chunks stay one second long.

If you want to tweak parameters yourself: Further down in the script there is a line commented out that reports real time information while the script is running. Uncomment and look how things develop.

caerandir
Posts: 131
Joined: Tue Dec 18, 2012 11:26 am
Location: Bonn, Germany

Re: PoC: Poor man's high framerate full HD motion detection

Sun Nov 17, 2013 5:15 pm

Some more experience: Now it is dark in Germany and I tried around with -ex night. This works awsome:

Code: Select all

raspivid -ex night -o $Basedir/stream%02d.h264 -fps 15 -g 5 -sg 500 -wr $FilesInLoop -t 0 -qp 10 -e &
Only thing: Somehow the GPU needs quite some time to settle down: The file sizes vary a lot during the first 30-40 seconds before the baseline gets stable. Same I realized in normal exposure when the image got noisy (during dawn). Obviously the codec does quite some long-term processing. In this situations the SurveillanceMode did not work too well, but BaselineMode is good.

BorisS
Posts: 15
Joined: Fri Oct 11, 2013 7:08 pm

Re: PoC: Poor man's high framerate full HD motion detection

Sun Nov 17, 2013 8:51 pm

Hi,
below there is yet another bash script to record video (and optional stills) on movement detection. In contrast to the poor man's solution it is not based on file size but on the difference of frames. For this purpose, ffmpeg is used to extract one frame from each segment and the compare tool (ImageMagick) to compute the root mean square difference that is used as metric.

Quick tests show it works quite well. Unfortunately, ffmpeg and compare calls are quite slow, thus sample period is a little bit higher. But this may change in future when James implement the feature that stills can be dropped for free (best would be down-scaled bmp that can be processed with low computational complexity).

Enjoy,
Boris

Code: Select all

#!/bin/bash

# record cam output (video and optionally still) in case movement is detected
# requires compare, ffmpeg
# BorisS Nov, 17th 2013

# number of segment periods between reference and current frame
# choose higher numbers for detecting slowly moving objects    
MOV_DET_DIFF_PERIODS=7
# relative rmse between pics that trigger recording
MOV_DET_THRESHOLD=0.04
# save comparison pics for quick overview/timelapse purpose
SAVE_ALSO_PIC=true
# record number of periods before movement is detected
SAVE_NUM_PERIODS_BEFORE_MOV_DET=1
# non volatile storage (sd, nas, ...)
SAVE_DIR='/home/pi/DiskStation1/home/picam/test/'
# mountpoint for ramdisk
RAM_DIR='/ram/'



########################################################################################################################
# helper functions
########################################################################################################################

get_mov_metric () {
  # extract first frame (jpg) from video segment
  ffmpeg -i vid_seg_$VidSegReadyCounter.h264 -vframes 1 -an pic_$PicReadyCounter.jpg >/dev/null 2>&1
  if [ -f pic_$PicReference.jpg ]
  then
    # metric is relative RMSE between reference and current pic 
    all_metric=$(compare -metric RMSE pic_$PicReadyCounter.jpg pic_$PicReference.jpg null: 2>&1)
    rel_metric=$(echo "$all_metric" | sed  -n 's/[^(]*(//p' | sed 's/)//')  
  else
    # ramp up - no reference pic yet
    if  [ $PicReference -ne $PicReadyCounter ]
    then
      PicReference=$(($PicReference + 1))
    fi
    rel_metric=0      
  fi
}


advance_reference_if_required () {
  # pic reference counter is incremented only if it is at the end of the circular window
  if [ $PicReference -eq $PicReadyCounter ]
  then
    PicReference=$(( 1+$PicReference ))
  fi
  PicReference=$(( $PicReference % $CircSizePic ))
}


derive_counters () {
  # wrap counters: pic counters are 0-based; video counter is 1-based! 
  VidSegReadyCounter=$(( 1+( ($VidSegReadyCounter - 1 + $CircSizeVid) % $CircSizeVid) ))
  VidSegUpcomCounter=$(( 1+$VidSegReadyCounter ))
  VidSegUpcomCounter=$(( 1+( ($VidSegUpcomCounter - 1) % $CircSizeVid) ))
  PicReadyCounter=$(( ($PicReadyCounter + $CircSizePic) % $CircSizePic ))
  PicPrevCounter=$(( ($PicReadyCounter-1+$CircSizePic) % $CircSizePic ))
  PicReference=$(( $PicReference % $CircSizePic ))
  advance_reference_if_required
}


incr_counters () {
  # advance pics (except reference) and video counters by one
  VidSegReadyCounter=$(($VidSegReadyCounter + 1))
  PicReadyCounter=$(($PicReadyCounter + 1))
  derive_counters    
}

save_output () {
  if [ -f vid_seg_$VidSegReadyCounter.h264 ]
  then
    timestamp=$(date +%Y_%m_%d_%H_%M_%S -r vid_seg_$VidSegReadyCounter.h264)
    cp vid_seg_$VidSegReadyCounter.h264 $SAVE_DIR"vid_seg_"$timestamp.h264
    if $SAVE_ALSO_PIC
    then
      timestamp=$(date +%Y_%m_%d_%H_%M_%S -r pic_$PicReadyCounter.jpg)
      cp pic_$PicReadyCounter.jpg $SAVE_DIR"pic_"$timestamp.jpg  
    fi
  fi
}

save_output_from_previous_periods () {
  TmpVidSegReadyCounter=$VidSegReadyCounter
  TmpPicReadyCounter=$PicReadyCounter
  TmpPicReference=$PicReference
  for ((i=1; $i <= $SAVE_NUM_PERIODS_BEFORE_MOV_DET; i++))
  do
    VidSegReadyCounter=$(( $VidSegReadyCounter - $i ))
    PicReadyCounter=$(( $PicReadyCounter - $i ))
    derive_counters
    save_output
  done
  VidSegReadyCounter=$TmpVidSegReadyCounter
  PicReadyCounter=$TmpPicReadyCounter
  PicReference=$TmpPicReference
  derive_counters
}

rec_state_and_save () {
  if [ $(echo "$rel_metric > $MOV_DET_THRESHOLD" | bc) -eq 1 ]
  then
    if [ "$RecState" = "off" ]
    then
      save_output_from_previous_periods 
      echo "off->on"
    else
      echo "on"
    fi
    save_output
    RecState="on"
  else 
    if [ "$RecState" = "on" ]
    then
      # rebase PicReference at "on" -> "off" transitions to avoid recordings after object disappeared
      PicReference=$PicReadyCounter
      echo "on->off"
    else
      echo "off"
    fi
    RecState="off"
  fi
}
 
########################################################################################################################
# create RAM dir if necessary
########################################################################################################################

sudo mkdir -p $RAM_DIR
sudo chmod 777 $RAM_DIR

# check if already mounted, if not do it (size may need adaptation depending on required buffer)
mountpoint -q $RAM_DIR || sudo mount -t tmpfs -o size=64m tmpfs $RAM_DIR
cd $RAM_DIR || exit 1

# delete output from previous run
rm pic_*
rm vid_seg_*


########################################################################################################################
# init
########################################################################################################################

# Current ready segment + upcoming segment + saving periods before 
CircSizeVid=$(( 2+SAVE_NUM_PERIODS_BEFORE_MOV_DET ))
# Current pic + period
CircSizePic=$(( 1+MOV_DET_DIFF_PERIODS )) 
# if old pics have to be saved as well, circular buffer is maximum of both buffers
if $SAVE_ALSO_PIC
then
  CircSizePic=$(( $CircSizeVid>$CircSizePic?$CircSizeVid:$CircSizePic ))
fi

# init counters
PicReadyCounter=0
PicReference=0
VidSegReadyCounter=$CircSizeVid
derive_counters

# init states and other stuff
InitPeriod=2
rel_metric=0
RecState="off"

# timestamps are used to keep track of background generated segments, init stamps this way
for ((i=1; $i <= $CircSizeVid; i++))
do
  touch vid_seg_$i.h264
  sleep 1
done


########################################################################################################################
# start to record video segments in background
########################################################################################################################

raspivid -n -g 30 -fps 30 -w 640 -h 480 -sg 10000 -wr $CircSizeVid -t 0 -qp 30 -o vid_seg_%d.h264 &
RaspiVidPID=$!
echo "RaspiVidPID = $RaspiVidPID"


########################################################################################################################
# main loop: check if movement detected and save cam output if applicable
########################################################################################################################

while [ 1 ]
do
  # skip segments if processing too slow
  while [ vid_seg_$VidSegUpcomCounter.h264 -nt vid_seg_$VidSegReadyCounter.h264 ]
  do
    echo "skipping segment"
    cp pic_$PicPrevCounter.jpg pic_$PicReadyCounter.jpg
    incr_counters
  done
  
  # wait until next segment is recorded
  while [ vid_seg_$VidSegReadyCounter.h264 -nt vid_seg_$VidSegUpcomCounter.h264 ]
  do
    # echo "waiting for next video segment"
    sleep 1
  done
  
  # image processing with ramp-up bypass
  if [ $InitPeriod -gt 0 ]
  then
    echo "Init period: $InitPeriod"
    InitPeriod=$((InitPeriod-1))
    PicReference=$PicReadyCounter
  else
    # echo "processing ..."
    get_mov_metric
  fi
  echo "ref:$PicReference cur:$PicReadyCounter metric=$rel_metric"
  rec_state_and_save
  incr_counters
done


User avatar
jbeale
Posts: 3578
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: PoC: Poor man's high framerate full HD motion detection

Tue Nov 19, 2013 8:24 pm

BorisS wrote:Hi,
below there is yet another bash script to record video (and optional stills) on movement detection. In contrast to the poor man's solution it is not based on file size but on the difference of frames. For this purpose, ffmpeg is used to extract one frame from each segment and the compare tool (ImageMagick) to compute the root mean square difference that is used as metric.

Quick tests show it works quite well. Unfortunately, ffmpeg and compare calls are quite slow, thus sample period is a little bit higher. But this may change in future when James implement the feature that stills can be dropped for free (best would be down-scaled bmp that can be processed with low computational complexity).
Good idea, thanks for that contribution! As you say, extracting a scaled-down frame would be best. Otherwise you still might be able to sub-sample to reduce the complexity. Also, in some cases you want to detect changes in only a smaller area of the full frame, so you wouldn't need to test all the pixels.

Redsandro
Posts: 27
Joined: Mon Nov 25, 2013 7:19 pm
Location: The Netherlands
Contact: Website

Re: PoC: Poor man's high framerate full HD motion detection

Mon Nov 25, 2013 7:31 pm

This is an interesting topic. I'm gonna watch this for a while.

I have this Pi + camera thing for a while now but I only got around to try it out this weekend. I am kind of surprised that certain things still take up a lot of time (clockcycles) even though the Pi was praised for it's GPU. I was kinda hoping that someone who is a lot smarter than I am would have made more hardware accelerated camera module awesomeness by now. :mrgreen:

I'd like to record h264 (on motion) and at the same time send (small(er)) pictures over the web for remote view. A little security camera project. This is not possible given the current tools, am I wrong?

Most video cameras can do it (take picture while filming) so I am thinking it's a software limitation.

User avatar
jbeale
Posts: 3578
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: PoC: Poor man's high framerate full HD motion detection

Tue Nov 26, 2013 7:00 am

Redsandro wrote:I'd like to record h264 (on motion) and at the same time send (small(er)) pictures over the web for remote view. A little security camera project. This is not possible given the current tools, am I wrong?

Most video cameras can do it (take picture while filming) so I am thinking it's a software limitation.
It's possible now to record H.264 files with almost no CPU effort as the encoding is done by GPU. You could also, at the same time extract still frames from the recorded h264 file, optionally downsize it, and send that via web / email / whatever, you'd just need to have the CPU do it. A few lines of shell script can do this, but it would not be super fast (not a very high frame rate). Eventually, it should be possible to grab JPEGs while also recording video all done from the GPU side, but that is not yet implemented.

You might also keep an eye on this thread, the "pure Python camera interface" where 'waveform80' mentioned the video + stills you want as a planned feature: http://www.raspberrypi.org/phpBB3/viewt ... 50#p453927

Return to “Camera board”