User avatar
jbeale
Posts: 3578
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

video recording with motion detection, in Python

Sun Sep 28, 2014 7:35 am

I don't know when or how much additional work I'll get done on this, so I'll put up what I have at the moment.
This is a follow-on from a post starting on the picamera thread at http://www.raspberrypi.org/forums/viewt ... 50#p618492

My first attempt to use motion detection with "before" buffer + after recording was not satisfactory, as the circular buffer would sometimes not work, and it would always add extra variable delay depending (I think) on the H264 frame structure (I/P frame) at the moment.

My second attempt was to use motion detection while recording continuously, with still JPEG images also saved at moments of detected motion. I hoped that could be done without interrupting the .h264 file going to disk, but I found it did interrupt the video (created an unrecorded gap of a second or more).

So the latest code simply records segmented .h264 video files of specified length (currently 3600 seconds) and also does motion detection. The videos have a burned-in timestamp. At the end of each a file segment there are some number of seconds where the video timestamp doesn't update and the motion-detection doesn't happen, while the picamera library is (I assume) waiting for an I-frame from the encoder to close the file cleanly.

I have done very little testing, but so far the motion-detect part appears adequate for my use. It works (I think) like the well-known 'motion' code although I haven't examined that program fully.

I use the MMAL video splitter and rescale feature through picamera to get a low-res YUV image, and use the Y component only. I create an average "background" image and look at the difference of each new frame from that background. I also create a rolling-average standard-deviation image to track the expected variability of each pixel (for example, tree leaves or branches blowing in the wind). So detecting a motion event requires a certain number of pixels to have a change that exceeds the recent standard deviation of that pixel by a certain amount, and separately the peak deviation over the whole image has to exceed a threshold. The latter condition avoids triggering on slight overall changes in brightness. At least with "night" exposure mode, the brightness of the image flickers noticibly up and down in some lighting conditions: it looks like an AGC algorithm in the image chain is slightly underdamped and tends to oscillate. My motion-detect is running around 6 fps now although it could go faster at lower YUV resolution. Note, the YUV bitmap resolution used for motion calculations, and the saved video h264 resolution are independent, thanks to the MMAL splitter and rescaler working in the background.

At the moment my code uses only half of a 16:9 frame vertically so the video has a superwide aspect ratio near 4:1, simply because my test location had overhanging wavy branches in the top half of the frame that I didn't want to deal with. That is of course configurable. I use picamera for the camera interface and Numerical Python (numpy) for the computations. I also use Python Imaging Library (PIL) but only for debug output, not needed in normal use.

The other motion-detecting programs I know of all have some delay between detecting motion and recording image or video. Here, I avoid that problem by simply recording all the video all the time, and just logging detected motion in a text file. So for now it requires manual review; but in theory you could automatically extract stills from the video on another computer.

It's not very user-friendly at the moment, and no guarantees it's useful- but if you're brave, code is at https://github.com/jbeale1/PiCam1/blob/ ... RecSeq1.py

UPDATE: just realized I was causing some of the frame-drop problems myself by using f.flush() to force writing a log file, when I should not have. So I may be able to get high-res stills on motion frames saved out in real time, while maintaining the continuous video.

User avatar
jbeale
Posts: 3578
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: video recording with motion detection, in Python

Fri Oct 10, 2014 3:22 am

Progress. The system has now run for a day and the video recording and motion detection system appears to work as designed. This gives you 1080p video (at 8 fps) and 1920x1080 still frames when motion is detected. Note that it is designed as a two-computer solution, and the remote server should be something faster than a Pi unless you have very infrequent motion events. Due to a two-stage pipeline with 30 second long segments, the stills don't show up in the ./events directory on the remote server for a few minutes after the fact, but there are only 2 frames of motion lag in terms of detecting motion and the code could in principal be adjusted to look backwards in time from the motion event as well. For now it is working well enough for me, in a way that none of the other 'motion' solutions on the Pi I'm aware of, yet have.

It is "alpha" code, only just working on my one system. If anyone else manages to get it set up I'd be curious to hear how it works for you. The motion parameters can be adjusted in the code, it's tuned for my particular case for now. It needs MP4Box (apt-get install gpac) and remember you have to compile your own libav https://github.com/libav/libav from source to get a usable avconv, (mine says avconv version v12_dev0-136-g2a5ac99 ), the stock apt-get Debian versions of avconv do not do frame-accurate seek on mp4.

https://github.com/jbeale1/PiCam1

User avatar
jbeale
Posts: 3578
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: video recording with motion detection, in Python

Mon Oct 13, 2014 4:49 pm

Here's a snippet from a motion log file generated by the PiMotion program alongside a .h264 video file. The first column shows the number of pixels showing motion (based on the scaled-down image of 32x16 matrix of 512 pixels), and the third column "AvgValue" shows the average amount of brightness change of those pixels (multiplied by a scale factor). In this case, you can see an object passed by between Time = 06:58:01.883 and 06:58:03.438. The second column shows the time offset from the start of the corresponding .h264 file, so you can extract any given frame using 'avconv'. The fourth column "delayTime" is debug info, it shows the amount of time (sec) left over after each motion-detect process loop, before it is time to start again.

Code: Select all

Pixls, Time(s), AvgValue, delayTime, Date_Time
0,    9.125,       0.0,    0.043,   141013_06:58:00.955
0,    9.375,       0.0,    0.034,   141013_06:58:01.226
0,    9.625,       0.0,    0.026,   141013_06:58:01.482
0,    9.750,       0.0,    0.190,   141013_06:58:01.622
8,    10.000,   3740.9,    0.187,   141013_06:58:01.883
45,   10.250,   4360.3,    0.178,   141013_06:58:02.129
49,   10.500,   4701.3,    0.175,   141013_06:58:02.419
81,   10.750,   4187.0,    0.175,   141013_06:58:02.670
146,  11.000,   4176.7,    0.138,   141013_06:58:02.924
111,  11.250,   2001.1,    0.129,   141013_06:58:03.195
47,   11.500,    974.6,    0.132,   141013_06:58:03.438
0,    11.750,      0.0,    0.137,   141013_06:58:03.685
0,    12.000,      0.0,    0.096,   141013_06:58:03.941
0,    12.250,      0.0,    0.093,   141013_06:58:04.195
0,    12.500,      0.0,    0.095,   141013_06:58:04.455
Below are the six frames (selected by > 20 pixels of motion) extracted from the .h264 file. There is an offset-by-one problem (the object has already left by the last frame) so I should shift each index offset earlier by 0.25 seconds, I haven't fixed that yet. It would also need to skip back into the previous .h264 file, in the case where motion was flagged on the very first frame of a segment.

Image

At a guess this vehicle is 5 meters long, and it looks to travel its length in about 0.5 seconds. So it is going about 10 m/sec (36 km/h, 22 mph).

User avatar
Mettauk
Posts: 238
Joined: Mon Dec 10, 2012 12:40 pm
Location: Zarg

Re: video recording with motion detection, in Python

Mon Oct 13, 2014 7:40 pm

This sounds great as a solution with far better results than ispy etc and it will have other uses HD related.

I'm bogged down in pan tilt motion tracking at the moment but will try this out very soon. So good luck with it if your still working on it that is!
As humans we have been the same for a very very long time, technology changes how we do... not who we are as people.

User avatar
jbeale
Posts: 3578
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: video recording with motion detection, in Python

Mon Oct 13, 2014 9:07 pm

Thanks! It is working pretty well for me at the moment, but I hope to add some more features when time permits. Since it is written in python and a few shell scripts, it should be easier for others to modify / improve than some of the other options. In daylight hours you get good detail. For things moving around 25 mph, this gives better results than anything else I'm aware of, since you get 1920x1080 stills, and basically no lag.

If you were to use the camera's still-image mode you will get more detail, but with a significant lag time penalty. Raw pre-H264-compression video frame grabs have somewhat more detail, but in my experience cause some skipped frames in the video. So even your continuous video record might miss something important, depending exactly when you trigger. In my current config recording only video and extracting stills later, I think you will not miss anything. As long as the network stays up and nothing crashes, of course!

User avatar
jbeale
Posts: 3578
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: video recording with motion detection, in Python

Wed Oct 15, 2014 10:03 pm

I'm adding a feature, reporting the (x,y) center of the detected moving object in the log file. To debug this, I have been exporting the map of changed pixels on each frame. It works OK for smaller objects like a person walking by. For things that fill more of the frame like a passing car, it causes the autoexposure to shift; then all the background pixels suddenly look different and are detected as in motion. I'm not sure if I can alternately do run autoexposure and then "lock the settings", or I have to go fully manual to specify ISO/shutter/AWB-gain and then implement my own (slow) AGC loop in the background. Or if there's a way to control the rate of autoexposure adaption (from the picamera interface).

If this works, animated GIFs (from gifmaker.me/ ) showing motion mask output. People detected OK, cars cause issues.
Walking-Mask.gif
pedestrian walking - motion pixels
Walking-Mask.gif (3.1 KiB) Viewed 17854 times
Cars-Mask.gif
cars driving - motion pixels
Cars-Mask.gif (3.32 KiB) Viewed 17853 times

User avatar
jbeale
Posts: 3578
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

adapting to exposure changes

Thu Oct 16, 2014 6:12 pm

When the camera exposure settings change, the entire image gets brighter or darker, so all the pixels appear to be motion pixels. However, for the best image of whatever is passing by, you don't want to stop the camera from adjusting. So as a workaround in computing my motion map, I am taking the average value of the pixels in a wide stripe around the 4 edges of the frame, and applying a linear scale factor to pixel brightness in a new frame such that its average value matches the stored background. This is not a complete fix, because (1) the camera has some "gamma" scaling, it isn't linear so the brightest and darkest regions do not scale by the same factor as a result of an exposure change and (2) the new object may overlap the edge, so you aren't getting 100% background in your averaging. That effect, as well as changing shadows with brightness (sun appearing from behind clouds) can actually make the "fixed" version worse than the original. So I am calculating motion maps with both new and old methods on each frame, then choosing whichever one gives fewer detected motion pixels.
Car2.gif
passing car with less background clutter
Car2.gif (998 Bytes) Viewed 17798 times
Not perfect, but an improvement from before: now the averaged (x,y) coordinates of the motion pixels in each frame are much closer the actual object position. You can see the "Xpos" column values decreasingly smoothly from 86 to 5.9 as the car moves from right to left. The "countPx:" values at right show the number of motion pixels detected with the original image and the scaled image, so you can see had we been choosing the original method for those frames, there would be a lot of (wrong) background pixels detected as motion. The smaller value does not match the first "Pxls" column because the latter is calculated after the (dilation, 2x erosion) step which cleans up some isolated noise pixels.

Code: Select all

Pxls, sec, AvVal, Delay, Xpos, Ypos, ScaleFac, Date_Time
151, 2.500, 9010, 0.005, 86.2, 12.2, 1.000, 141016_10:14:41.927
395, 2.625, 9378, 0.005, 78.2, 10.3, 1.000, 141016_10:14:42.053
517, 2.750, 9900, 0.160, 73.3,  9.7, 1.000, 141016_10:14:42.235
547, 3.000, 7971, 0.152, 55.9,  9.1, 0.893, 141016_10:14:42.450  * countPx: 2604, 671
445, 3.250, 9490, 0.125, 38.7,  8.3, 0.939, 141016_10:14:42.708  * countPx: 2289, 543
474, 3.500, 6195, 0.123, 24.7,  8.4, 0.977, 141016_10:14:42.968  * countPx: 2052, 778
220, 3.750, 9855, 0.122, 13.2,  6.7, 0.965, 141016_10:14:43.226  * countPx: 1911, 327
 71, 4.125, 8867, 0.071,  5.9,  6.4, 0.926, 141016_10:14:43.614  * countPx: 1784, 116
  0, 4.375,    0, 0.075,  0.0,  0.0, 0.916, 141016_10:14:43.872  * countPx: 1665, 4

User avatar
RaTTuS
Posts: 10531
Joined: Tue Nov 29, 2011 11:12 am
Location: North West UK
Contact: Twitter YouTube

Re: video recording with motion detection, in Python

Mon Oct 20, 2014 8:51 am

very interesting I must have a play with this
How To ask Questions :- http://www.catb.org/esr/faqs/smart-questions.html
WARNING - some parts of this post may be erroneous YMMV

1QC43qbL5FySu2Pi51vGqKqxy3UiJgukSX
Covfefe

User avatar
jbeale
Posts: 3578
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: video recording with motion detection, in Python

Mon Oct 20, 2014 2:15 pm

If you do try it out, let me know how you get on. The most recent motion-detect code is https://github.com/jbeale1/PiCam1/blob/master/stest2.py which is currently working more-or-less OK in my testing; some over- and under- detection of motion pixels, but I think that will never be 100%. On many object it works well enough to track the motion center over many frames and estimate an apparent velocity, so I can later extract data as shown below (generated by https://github.com/jbeale1/PiCam1/blob/ ... tchjpeg.py )

Sec: duration of motion event in seconds
xSize: average horizontal width of bounding box of motion pixels
xVel: apparent horizontal velocity of motion center (pixels per second)
fitPrm: goodness of a linear velocity fit (higher is worse)
frms: number of frames of motion detected

You can bin events into categories based on xSize and xVel with pretty good accuracy. Events with xSize above 40 and velocity magnitude around 50 are very likely to be cars, passing to the right (+xVel) or left (-xVel). Events with xSize below 20 are likely pedestrian or bicycle. Numbers for xVel below 10 are probably not an object simply passing by from one side to the other; might be a car leaving a driveway, or two or more objects at once. Note here that xVel is apparent velocity and not true velocity; the motion detection is not 100% consistent and besides the camera points about 45 degrees across the street so there is some perspective effect.

Code: Select all

   Sec,  xSize,   xVel, fitPrm, frms,    date_time          actual event by inspection
-----------------------------------------------------------------
   2.0,   19.5,   55.5,    3.5,    6,  141019_15:30:03.139  pedestrian ->
   2.6,   30.0,  -41.7,  182.6,   10,  141019_15:32:43.885  me, in yard near camera <-
   1.9,   55.2,  -62.3,   63.1,    6,  141019_15:34:07.057  pickup truck <-
   4.8,   25.1,   -1.3,    3.4,   20,  141019_15:35:50.446  car backing from driveway
   3.9,   20.2,   -2.7,    1.0,    4,  141019_15:35:55.891  same car, still backing
   1.9,   47.8,  -47.9,  119.7,    6,  141019_15:37:45.259  car <-
   1.9,   58.5,  -61.3,  125.3,    6,  141019_15:42:12.647  SUV <-
   1.9,   59.1,  -54.8,  135.0,    7,  141019_15:46:41.071  car <-
   4.6,   25.5,   -7.1, 2230.4,   14,  141019_15:53:41.174  rollerblader + bicycle <-
   2.8,    9.9,  -31.4,   26.8,   10,  141019_15:53:47.262  kid on bicycle <-
   1.2,   52.8,  -94.6,  114.6,    5,  141019_16:04:14.145  car <-
   2.6,   40.3,  -26.5,  354.3,    9,  141019_16:15:08.794  two bicycles <-
   2.1,   61.2,   55.7,   44.1,    6,  141019_16:16:04.328  pickup truck ->
   2.1,   63.4,  -45.3,   93.7,    7,  141019_16:21:19.722  pickup truck <-
   1.2,   74.8,   96.3,  101.5,    4,  141019_16:21:25.167  pickup truck ->
Last edited by jbeale on Tue Dec 09, 2014 7:24 pm, edited 1 time in total.

User avatar
jbeale
Posts: 3578
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: video recording with motion detection, in Python

Tue Oct 28, 2014 4:21 pm

Fixed a bug (so far, anyway) which was causing crashes around sunrise and sunset.

Background: My program records video in chunks of 30 seconds (that is 240 frames at 8 fps, which is 4 GOPs; 1 GOP = 60 frames). During the day, each .h264 file is about 35 MB (about 1.2 MByte/sec = 9.3 Mbit/sec) and during the night, 10 MB. At those bitrates the system works OK. What I did not realize until now was that the bitrate briefly spikes right at sunrise and sunset to twice the daytime value, when the image becomes noisy in dim light but the nighttime-level noise reduction has not yet taken over (see plot below).

Right at sunrise and sunset, the H.264 stream coming in from the camera reaches 20+ Mbit/sec; that exceeds the ~19 Mbit/sec maximum data transfer rate ('scp') going out out to the server. Consequently, the Pi's ramdisc buffer quickly reaches 100%, and the program crashes. The fix (so far, at least) was to increase the ramdisc size but more importantly, set a H.264 video bitrate of 12 Mbit/sec instead of accepting whatever the default is (allegedly 17 Mbit/s, but it seems that is more of a guideline than a rule).

Image
File size dips below average at the sunrise/sunset peaks are artifacts from the system crashing and restarting; dips during the day are from manual restarts.

User avatar
jbeale
Posts: 3578
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Bitrate limit doesn't always work (picamera)

Wed Oct 29, 2014 5:19 pm

Ok, limiting bitrate does not work the way I expected. I used the "bitrate" parameter to specify 12000000 bps but it seems that is not always honored. Yesterday, one 30-second file around sunset reached 70 MB in size, which means the average bitrate for those 30 seconds is 19 Mbps. The strange thing is there is a plateau around 45 MB during sunrise, which works out to 12 Mbps, but there is also a much higher spike. It seems like the bitrate limit is sometimes honored by the encoder, and sometimes not. See also: http://www.raspberrypi.org/forums/viewt ... 75#p633182

Image

User avatar
jbeale
Posts: 3578
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: video recording with motion detection, in Python

Mon Dec 08, 2014 8:11 pm

Update in case of interest. This program has been mostly OK recording video 24/7 for the past month. The problems have been:

1) camera connector assembly got wet in rain (outdoor location). Program still running, but camera does not produce images.
Solution: disassemble, clean up, reassemble into more weather-resistant housing.

2) remote host archiving video & stills (Chrubuntu on Acer C720 chromebook) has slow memory leak, freezes up after running about a week.
Workaround: manual reboot every week.
UPDATE: removed lock screw to enable SeaBIOS, and installed normal Debian 13.10 ( 3.11.0-26-generic x64). Fixed; memory use stable.

3) Afternoon direct sunlight in background with shaded foreground: sunny highlights blown out, with default auto-exposure settings.
Solution: camera.exposure_compensation = -7 // make image darker. Default is 0
Most images look somewhat dark and the darkest tones get crushed to 0, but significantly more usable information when sunny.
UPDATE: made exp_comp. adaptive between -8 and 0, aiming for a fixed number of pixels in upper 1/3 of histogram. This works OK.

4) Video bitrate at sunset & sunrise goes well over setpoint and can exceed network bandwidth available
Solution: reduce requested bitrate to 9 Mbps. No significant loss of visual detail noted, in this application.
Last edited by jbeale on Thu Jan 01, 2015 5:01 pm, edited 2 times in total.

joseph_curwen
Posts: 3
Joined: Sat Dec 06, 2014 7:11 pm

Re: video recording with motion detection, in Python

Tue Dec 09, 2014 10:23 am

Thanks a lot for your work

i just had a quick look at your project, and a lot of things are of interest for me.

Hope to have some time to experiment.

spikedrba
Posts: 75
Joined: Fri Feb 28, 2014 2:19 am

Re: video recording with motion detection, in Python

Tue Jan 27, 2015 11:54 pm

Hello John,

replying here from the RPi Cam Web Interface thread. Thanks a ton for working on PiCam, it looks exactly like what I needed (mostly concerned about accuracy of detection).

I'm still a noob with the pi and and even a bigger noob around photography, hopefully you can bear with me for a little. Looking at the thread you mention 8fps and I'm not sure that's gonna work out to pick up licence plates. I have to use an 8mm lens from m12 in order to get a big enough picture of the licence plate, but that means the car at ~20mph stays in the frame a fraction of a second, short enough that I doubt I would get a good shot of it at that fps.

Is there a reason why you picked 8? could that be increased without hitting some performance cap?

thanks,

Spike

JayGee59
Posts: 4
Joined: Tue Sep 12, 2017 10:37 pm

Re: video recording with motion detection, in Python

Tue Sep 12, 2017 10:54 pm

Go to this site: https://hackernoon.com/raspberrypi-home ... 01bd0373c9

Cursor down to the area where he talks about setting the background to avoid anomalies of motion
in your video captures. He also has the code displayed for that.

Return to “Camera board”