pageauc
Posts: 223
Joined: Fri Jan 04, 2013 10:52 pm

pi-timolo (pi, timelapse, motion, lowlight)

Sun Dec 21, 2014 2:32 pm

pi-timolo Released on github
https://github.com/pageauc/pi-timolo
Here are some sample YouTube Videos FYI https://www.youtube.com/playlist?list=P ... 2T0pgfaSG0

2-Aug-2015 Version 2.8 Release
Updated gdrive to version 2.8
Updated makedailymovie.sh to use avconv instead of mencoder
Updated setup-timolo.sh to remove mencoder, gpac and add libav-tools
Updated setup-timolo.sh to allow upgrading if pi-timolo.py does not exist
Deleted some files h2mp4.sh makemovie.py is now makemovie.sh
Updated Readme.md

12-Jun-2015 Version 2.6 Release
IMPORTANT - A replacement for syncing files with google drive is included with this release
- fixed bug that caused a hang when video mode was True
- Added a new quick time lapse after motion detected feature.
- cleaned up console display
- misc bug fixes
- Added gdrive binary for easy syncing of files with internet based google drive see https://github.com/odeke-em/drive/releases amd binary and https://github.com/odeke-em/drive for additional implementation details

Please note
If you are looking for a good web based interactive RPI camera interface I would highly recommend applications per links below. They are relatively easy to install and work well. But I still prefer pi-timolo for remote headless camera situations so I have both installed but you can only run one or the other at the same time.
RPI_Cam_Web_Interface
Details can be found on github https://github.com/silvanmelchior/RPi_Cam_Web_Interface
and here http://elinux.org/RPi-Cam-Web-Interface
This interface can also be setup to use gdrive syncing. Contact me if you need details.
I set this up with a 3 second buffer so video is recorded prior to motion triggering. This interface can also be configured to use file syncing with your google drive but is not part of the setup. Contact me if you need details.
install is pretty simple

Code: Select all

git clone https://github.com/silvanmelchior/RPi_Cam_Web_Interface.git
cd RPi_Cam_Web_Interface
chmod u+x RPi_Cam_Web_Interface_Installer.sh
./RPi_Cam_Web_Interface_Installer.sh install
This runs on Raspbian so you can run other programs as well. I install pi-timolo as well as this web interface. Since the camera cannot be easily shared they cannot run concurrently but it is easy to stop camera on the web interface when needed. If we go away for a long time I switch over to pi-timolo and turn off the RPi-Cam-Web-Interface until I get back. I should write a script to automate this. Anyway I have this interface as well as pi-timolo and switch as required.
motionPIE
This is a stand alone GUI web interface package that is installed as an SD card image file. Instructions can be found here
https://github.com/ccrisan/motionpie/wiki/Installation
Download image files here
https://github.com/ccrisan/motionPie/releases
Note there are separate RPI B and B2 images so download the appropriate one. Just unzip the image and burn to a new SD card, pop into RPI, boot and access Web interface using the IP address on a Web Browser. This can be handy since you can just pop in the motionPIE SD card when required and boot using another raspbian SD if you need the RPI for something else. But this is not as convenient as the Web Interface above running on raspbian.
motionPIE is a little slower at streaming video but works OK (laggy). It is designed to only run video security so you are limited. It has an admin and user mode that can be useful for protecting configuration settings. It Has lots of configuration settings. I configured mine to save files to a share on my network NAS and that was very easy.
Both these web interface packages are pretty much plug and play but if you are interested in implementing your own project with python then my repo might be useful. It also gives you the ability to see and change the Python code if you want to customize it. There is a pi-motion-lite on my github repo that might be of interest to those that want to create their own Python picamera project

6-May-2015 Version 2.5 Release
This version fixes a bug that caused a hang when video mode was True
A new feature adds quick time lapse after motion detected
The console display is cleaned up plus a number of other bug fixes
Please note the older config.py is not compatible with this release
since new variables have been added
Note:
This release does not include grive
As of 20-Apr-2015 grive no longer works until it can be modified.
I have included instructions for installing GDriveFS. Here
viewtopic.php?p=753018&sid=d937a9e30526 ... 6b#p753018

Program Features
- Time lapse and motion detection can be run together or separately
- configuration variables are saved in a config.py file. This allows
pi-timolo.py to be updated without having to redo variable values.
This also allows swapping in different saved configurations.
- Auto detects day, night and twilight changes without using a clock.
- Image date/time and name can be put directly on images. Position and color
can be specified.
- Night motion and images use low light long exposure
- Include a makemove.py script to help create timelapse avi files from images.
- Includes mvleavelast.sh script to move all but last file (since may still be active)
This allows files to be moved to a folder mount point on a remote share if required.
- Includes a setup-timolo.sh script to install/update pi-timolo dependencies.
- Includes a skeleton init.d script pi-timolo.sh if you want to start the program on boot
- Allows logging of summary data to a file or detailed verbose data when enabled
- Include setup-gdrivefs.sh to mount your google drive to a RPI folder. This script
includes instructions.
use rsync to sync files between local and google mount folders

Quick Setup
SSH (putty) or Desktop Terminal login to RPI and perform the following

Code: Select all

cd ~
mkdir pi-timolo (or a folder name of your choice)
cd ./pi-timolo
wget https://raw.github.com/pageauc/pi-timolo/master/pi-timolo.tar
tar -pxvf pi-timolo.tar
sudo ./setup-timolo.sh
Initialize pi-timolo.py files, motion and test motion.

Code: Select all

sudo ./pi-timolo.py
Verify motion then ctrl-c to exit pi-timolo.py
Edit config.py file using nano editor to change any desired settings per comments.
ctrl-x to Save

Code: Select all

nano config.py
test edit changes. ctrl-c to exit pi-timolo.py

Code: Select all

sudo ./pi-timolo.py
If want to mount your web based google drive to a local RPI folder
execute the setup-gdrivefs.sh script and follow instructions

Code: Select all

sudo ./setup-gdrivefs.sh
If you want to run pi-timolo.sh headless on boot
See details instructions in Readme.txt for setting up run init.d, crontab, rsync Etc.

Merry Christmas to All
http://youtu.be/w6_WZz-oYgg
Last edited by pageauc on Wed Sep 02, 2015 7:16 pm, edited 16 times in total.
GitHub - https://github.com/pageauc
YouTube - https://www.youtube.com/user/pageaucp

pageauc
Posts: 223
Joined: Fri Jan 04, 2013 10:52 pm

Re: pi-timolo (pi, timelapse, motion, lowlight) project rele

Fri Jan 16, 2015 2:32 pm

Note regarding pi-timolo version 2.2
--------------------------------------------------
This version fixes a bug that caused motion capture to go into a continuous loop after forceTimer triggered.
Also this version greatly simplifies the previous convoluted checkIfDay() logic and is much simpler.
I still need to do a bit of code clean up but otherwise seems to work good.

Note the sunriseThreshold and nightDayTimer variables are no
longer used. The sunset threshold IS used and sets the day pixel average threshold and may need to be
tuned slightly higher or lower. Currently I have full daylight setting at 90 and lower lighting
conditions (under an awning) during the day set to approx 55 (your mileage may vary).

Also I would like to see if there is a way to auto tune the motion sensitivity and/or threshold as well as the sunsetThreshold (aka twilightThreshold) to be auto tuned as well. This would make it much easier.

Also I have found that if you do editing on a pc (I use notepad++ but set file menu item Edit, eol conversion to unix format) and transfer files (I use Filezilla with SFTP option) to the RPI, the python scripts that use shebang #! to point to an interpreter may fail due to not being in unix format. I use the dos2unix command to fix this problem. The dos2unix command will need to be installed

Code: Select all

sudo apt-get install dos2unix
If you use pi-timolo let me know.

Claude ...
...
GitHub - https://github.com/pageauc
YouTube - https://www.youtube.com/user/pageaucp

josehebu
Posts: 1
Joined: Mon Dec 29, 2014 3:51 am

Re: pi-timolo (pi, timelapse, motion, lowlight)

Sun Jan 18, 2015 4:27 am

First of all, i want to thank you for development this program, it is helping me a lot. I was working with this program but suddenly it stopped working. I wish that you could help me with this error, i don't know what happens.
Attachments
2015-01-17-224447_657x476_scrot.png
2015-01-17-224447_657x476_scrot.png (40.59 KiB) Viewed 11890 times

pageauc
Posts: 223
Joined: Fri Jan 04, 2013 10:52 pm

Re: pi-timolo (pi, timelapse, motion, lowlight)

Sun Jan 18, 2015 3:20 pm

If the install was working for you previously and your firmware and libraries are correct then the issue might be hardware related. I would do some basic trouble shooting per http://www.raspberrypi.org/learning/pyt ... mera-setup
Stop before the step CAMERA PROGRAMMING: CAPTURE WHEN ACTIVATED

http://picamera.readthedocs.org/en/late ... array.html
The picamera code that is generating the error should be the following function in the picamera python library code.

Code: Select all

def bytes_to_rgb(data, resolution):
    """
    Converts a bytes objects containing RGB/BGR data to a `numpy`_ array.
    """
    width, height = resolution
    fwidth, fheight = raw_resolution(resolution)
    if len(data) != (fwidth * fheight * 3):
        raise PiCameraValueError(
            'Incorrect buffer length for resolution %dx%d' % (width, height))
    # Crop to the actual resolution
    return np.frombuffer(data, dtype=np.uint8).\
            reshape((fheight, fwidth, 3))[:height, :width, :]
This error indicates that the raw stream buffer does not match the camera resolution specified. I have seen this error with older RPI firmware. I managed to get this fixed by running

Code: Select all

sudo rpi-update 

I assume you have power cycled the RPI just in case there was a memory leak/fault or some other program that was being run might have caused a memory or similar issue.
I have run the python picamera using opencv and other memory hungry programs like deep learning viewtopic.php?p=786040#p786040 without issue so your problem points to some sort of problem that is truncating the memory stream or the camera not generating the required stream output length causing the buffer size to not match the specified width/height resolution per the code above.

Let me know what the basic trouble shooting results are and proceed from there.
Last edited by pageauc on Wed Jul 08, 2015 4:33 pm, edited 3 times in total.
GitHub - https://github.com/pageauc
YouTube - https://www.youtube.com/user/pageaucp

pageauc
Posts: 223
Joined: Fri Jan 04, 2013 10:52 pm

Re: pi-timolo (pi, timelapse, motion, lowlight)

Sun Jan 18, 2015 4:43 pm

One other possibility is that you may have to increase the time.sleep in the pi-timolo.py getStreamImage() function. I set it to a minimum 0.1 sec. I believe this allows the camera hardware time to initialize. You could try increasing this value to something like 0.5 sec
The normal value would be around 2 seconds but since I want to minimize the motion detect loop time this is set to a low value. It might be that some hardware is slower and not capturing a complete stream. Give it a try to see if this helps.
see picamera recipes for examples
http://picamera.readthedocs.org/en/latest/recipes1.html

Code: Select all

def getStreamImage(isDay):
    # Capture an image stream to memory based on daymode
    with picamera.PiCamera() as camera:
        time.sleep(0.1)
        camera.resolution = (testWidth, testHeight)
        with picamera.array.PiRGBArray(camera) as stream:
Increase the time.sleep() value in pi-timolo.py def getStreamImage() function to see if this corrects the problem.
Claude ...
GitHub - https://github.com/pageauc
YouTube - https://www.youtube.com/user/pageaucp

pageauc
Posts: 223
Joined: Fri Jan 04, 2013 10:52 pm

Re: pi-timolo (pi, timelapse, motion, lowlight)

Sun Jan 18, 2015 8:45 pm

email reply from Josh on the results of his testing so looks like hardware related. I suggested he try reseating the cable at the camera and RPI end. I have found the RPI camera module to be pretty durable and survived some pretty rough handling on occasion.

reply from Josehb
-------------------------
I did the steps until "CAMERA PROGRAMMING: CAPTURE WHEN ACTIVATED", but when i put "raspistill -o image.jpg" on terminal it shows me: "mmal: Received unexpected camera control callback event, 0x4f525245" and stops here. I have been looking for a solution for this error, but this looks like a camera module problem, or even that the ribbon which connects the raspberry with the module has been damaged. Tomorrow, i will borrow another camera module and try to make it run.
I thank you for the time that you take to help me. i will write you to tell you if it solved.
GitHub - https://github.com/pageauc
YouTube - https://www.youtube.com/user/pageaucp

deitz645
Posts: 2
Joined: Wed Apr 01, 2015 3:22 pm

Re: pi-timolo (pi, timelapse, motion, lowlight)

Thu Apr 02, 2015 4:14 pm

Thanks for a great tool. I was able to get it up and running on my B+ Pi without too much trouble. Now that it's running, I was wondering if the time lag between sensing motion and actually creating the jpg normal or is there a setting change to speed it up? As you can see below there's a bout a 46 second lag.

Another note is I should have created a generic Google account for this because the initial grive sync copied everything I had on Google Drive to my Pi, even though I moved my Google Drive stuff to a sub folder.

20150402_12:06:09 checkForMotion - Found Motion - sensitivity=300 Exceeded ...

20150402_12:06:11 getNightCamSettings - dayPixAve=49 ratio=0.456 ISO=364 shut=2642222 2.642 sec
20150402_12:06:55 takeNightImage - Size=640x480 dayPixAve=49 ISO=364 shut=2.642 sec /home/pi/pi-timolo/motion/mo-finn_cam1-1008.jpg
20150402_12:06:55 writeDataToImage - Added Text[1008 20150402_12:06:55] on /home/pi/pi-timolo/motion/mo-finn_cam1-1008.jpg
20150402_12:06:55 postImageProcessing - Next Counter=1009 /home/pi/pi-timolo/mo-pi-timolo.dat

20150402_12:06:55 .

pageauc
Posts: 223
Joined: Fri Jan 04, 2013 10:52 pm

Re: pi-timolo (pi, timelapse, motion, lowlight)

Thu Apr 02, 2015 5:55 pm

There is normally a delay due to the motion detection image stream pixel comparison loop. Overclocking can speed if not already done. Also there is a short 0.1 or 0.2 sec time delay when the full size image is taken. This is to allow for the camera to stabilize otherwise the image can be a little wonky. I have done some testing and this seems to be normal for the default motion detect stream image size. It might be possible to reduce the size of the detection stream image to speed up the motion detection loop that compares pixels between the two stream images. If you did then you would also have to change the sensitivity setting since there would be fewer pixels in a smaller stream image. The low light feature at night would still have delays due to the long exposure needed.

I have a request from another user to have an option for taking a fast time lapse sequence for a specified period of time after motion is detected. Although there is a video feature after motion is detected the short fast time lapse after motion detection would be a good feature to have and might help capture more detail once motion is detected. This would not solve the problem of the initial delay between motion detection and taking the first large image but would capture more detail by having an optional fast time lapse sequence after motion feature. I will implement this feature because it would be useful for me in some situations where there is not very much motion activity. We winter in Texas and will shortly be moving back to Canada so it will be several weeks before this is done and tested but I will post update on GitHub and this forum when it is available and maybe do a short YouTube video for those interested.

Regards Claude ..
GitHub - https://github.com/pageauc
YouTube - https://www.youtube.com/user/pageaucp

pageauc
Posts: 223
Joined: Fri Jan 04, 2013 10:52 pm

Re: pi-timolo (pi, timelapse, motion, lowlight)

Thu Apr 02, 2015 10:00 pm

Your comment about grive syncing all google drive data to your RPI can be changed by moving all google drive data that you do not want synced into a google drive sub folder. Only the files in the root of the google drive are sync'd by grive. This is documented in my GitHub readme file. I also learned this the hard way. Once all non RPI data is moved to a google drive sub folder you should be good to go without having to create another account. This works for me.

Regards Claude ....
GitHub - https://github.com/pageauc
YouTube - https://www.youtube.com/user/pageaucp

deitz645
Posts: 2
Joined: Wed Apr 01, 2015 3:22 pm

Re: pi-timolo (pi, timelapse, motion, lowlight)

Mon Apr 06, 2015 11:50 am

Claude,
Thanks for both your replies. I found that the motion sense and capture runs MUCH faster when I'm running it direct or headless than via PUTTY or TIGHTVNC. Now I have to adjust it to make it less sensitive :) - We trained it on out puppy Friday evening and went out for dinner...By the time we we got home there were over 500 images and the image count had rolled oveer and overwritten the beginning images. Which was no problem since I'm in the primordial stages of getting the motion sense set up just right.

I followed your grive / Google Drive sync instructions to the letter. Again it's a minor issue, but on the initial set up you run sudo ./grive -a from the pi-timolo directory and grive does an initial sync from that path and syncs everything in that directory and sub-directories to Google Drive and everything it finds in Google Drive (including any sub-directories) to the Pi. At least that's what I experienced. And I had to do it a few times because I fat fingered a command and kept deleting the hidden .grive and .grive_state files. But that was my own fault. And like you said, once it's set up and the .grive and .grive_state files are in the Motion sub-directory, only those files are synced.

Thanks again for a great Python tool. It's forced me to become a bit more proficient :)

Regards,
Carl

pageauc
Posts: 223
Joined: Fri Jan 04, 2013 10:52 pm

Re: pi-timolo (pi, timelapse, motion, lowlight)

Mon Apr 06, 2015 1:46 pm

I have a makedailymovie.sh bash script as part of the install that I run from a crontab entry. You could make a copy of this and call it makedogmovie.sh.

Code: Select all

cd ~
cd pi-timolo
cp makedailymovie.sh makedogmovie.sh

You would need to modify the code to add an hour parameter to the output file name. Currently it only outputs the date and not the hour but that is easy to add per below.
Use nano to add hour parameter to movie output name in the makedogmovie.sh script
see commented line for required change

Code: Select all

# get current working folder
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
folder_source=$DIR/motion
folder_destination=$DIR/mnt
moviename=$DIR/dailymovie_$(date '+%Y%m%d%h').avi  
# added hour to above line to movie name to make it unique if script run more than once a day
# Note the finished avi is copied to the folder_destination (I use a mounted network share).  The previous avi files in pi-timolo 
# are deleted the next time the script is run.  This avoids uniquely named avi files filling sd storage space 
movielist=$DIR/dailymovie.txt
That way you would not loose anything if that is what you are interested in. I find it easier to view a movie than individual images. This works better if the movie is moved to a mounted network share since the pi has limited storage unless you attach an external USB hard drive. I sometimes attach a USB hard drive directly to the rpi running pi-timolo. I format it as ext4 format when running a very long time lapse but fat can also be used.
Normally I like to use one of my rpi's with XBMC (Kodi) installed since it creates a share for the drive and it makes a low cost NAS drive on the network and you can use the XBMC interface on your TV to view your movies directly on your TV. This makes it convenient and easy to manage. This setup works better if run through a wired network rather than wifi since the movies can take a while to transfer over the network depending on how big they are.

To modify the pi-timolo.py python code to make a movie when the number count reaches 500 (or whatever maximum you set). This can be done with a subprocess.call command. in the postImageProcessing function.

Code: Select all

   
 if numberon:
        counter += 1
        if countermax > 0:
            if (counter > counterstart + countermax):
                if recycle:
                    counter = counterstart
                    # line below was added to run makedogmovie.sh when the file count reaches a maxiumum.
                    # Note this will delay motion detection until movie is made unless you have a RPI B2 and use
                    # multiprocessor library and run as a child process.
                    subprocess.call("/home/pi/pi-timolo/makedogmovie.sh",  shell=True)
                else:
Hope this helps and it would be a good learning experience as well. Let me know if you need any help.
Regards Claude ...
GitHub - https://github.com/pageauc
YouTube - https://www.youtube.com/user/pageaucp

pageauc
Posts: 223
Joined: Fri Jan 04, 2013 10:52 pm

Re: pi-timolo (pi, timelapse, motion, lowlight)

Thu Apr 09, 2015 1:38 pm

Recently I decided to add a second security camera using grive and pi-timolo. The problem I had was that the motion folder is considered the root of the google drive and both security cameras put all files in the google drive root and so were mixed together. After a bit of testing I found that if I simply made a sub folder in the motion folder for the specific camera and changed the config.py to put images in the subfolder then it worked great.. I also had to change makedailymovie.sh to point to the same subfolder. No changes were made to sync.sh that executes grive to sync everything in the motion folder and no grive hidden files were needed in the subfolder.
Now I can have files for each camera in it's own google drive subfolder and this works but beware that grive will sync everything in the google drive to each camera motion folder including all the subfolders. This is a limitation of grive at the moment. I would like to have grive sync only to a specified google drive subfolder but at this time it only works from the root folder as far as I can see. Let me know if you have any questions about this since the explanation may be a little confusing if you have not worked with pi-timolo and grive.

Regards
Claude ....
GitHub - https://github.com/pageauc
YouTube - https://www.youtube.com/user/pageaucp

pageauc
Posts: 223
Joined: Fri Jan 04, 2013 10:52 pm

Re: pi-timolo (pi, timelapse, motion, lowlight)

Sun Apr 19, 2015 12:35 pm

This video recorded the whole winter while we were south. The low light night shots turned out great and of course winter has longer nights so you get more.

Port Elgin Winter Time Lapse using Raspberry PI and python pi-timolo
https://youtu.be/pFkt0ZOrPzg

Port Elgin Winter 2014-2015 Time Lapse
Watch for the moon rises. The first one is near 1-Dec top left and there are a few more later in the video above the houses across the street.

Well, when we got back from wintering in Texas, I checked my raspberry pi computer that I left to record the winter. Approx 47,000+ images were taken at 15 minute intervals throughout the winter. Images are just as they were saved from the camera. I edited out some parts of the video so not all images were used.

The RPi and attached external usb Ext4 formatted hard drive were connected to a UPS uninterruptible power supply in case of power failures. I was surprised and delighted that the setup worked so well over the winter. Images were 1080p but encoded to 720p at 30 FPS for this video. I used an early version of a python program I wrote called pi-timolo This stands for Raspberry PI, Time lapse, Motion detection and Low light, The program is written in python and uses average pixel value and a threshold value to detect start of twilight and then ramps camera ISO and exposure settings to automatically transition smoothly between day/night modes automatically without needing a clock time reading. This algorithm works well and is even better on the new version of pi-timolo. A clock or accurate time is not needed for determining sunrise and sunset, except if you need accurate date/time information optionally displayed directly on the images The pi-timolo python program code and installation instructions are available on my github repo at

https://github.com/pageauc/pi-timolo

The current version of pi-timolo is much improved over the version used for this video. After copying the images to my NAS, I encoded the images in batches of 10,000 using mencoder 64 bit on my Win7 desktop computer. mencoder can be found on sourceforge here

http://mplayerwin.sourceforge.net/downloads.html

Each batch of 10,000 images was put in its own subfolder and I created a text file listing of all the image file names in each subfolder sorted by file name. This list is used by mencoder during encoding. From a DOS window in the folder containing the images I typed the following command

Code: Select all

dir /b /on *.jpg >images.txt
The above command redirects a list of files to encode to an images.txt file that is used in the mencoder encoding command below.

I then created a batch file to run mencoder.exe for each batch of images. I created a w: mapped network drive to my image folder share located on my NAS. This share contains the image subfolders eg Part_1, Part_2 Etc. Here is a typical batch file command that worked for me.

Code: Select all

rem w is a NAS mapped drive to my image file subfolders
w:
rem Part_1 is a subfolder containing a portion of the total images
cd Part_1
rem Change the path to the mencoder.exe file as required.
rem Note the images.txt file listing all the jpg files to encode
rem must be located in each image subfolder.
c:\Users\Pageauc\mencoder64\mplayer-svn-37386-x86_64/mencoder.exe -nosound -ovc lavc -lavcopts codec=mpeg4:aspect=16/9:vbitrate=8000000 -vf scale=1280:720 -o winter_part1_30.avi -mf type=jpeg:fps=30  f://@images.txt
You can modify the above to suit your situation and file locations.

Drop me a line if you have any questions.
Last edited by pageauc on Wed May 06, 2015 11:15 am, edited 1 time in total.
GitHub - https://github.com/pageauc
YouTube - https://www.youtube.com/user/pageaucp

pageauc
Posts: 223
Joined: Fri Jan 04, 2013 10:52 pm

Re: pi-timolo (pi, timelapse, motion, lowlight)

Wed May 06, 2015 4:19 am

I have posted a forum post at viewtopic.php?p=753018#p753018 regarding the recent problem with grive and an alternate solution that uses gdrivefs. I originally tried recompiling grive from several sources but no joy. I then found the problem was with the google docs api so current grive is pooched until give is updated by someone. In the mean time I found a nice solution and wrote instructions on implementing it. My first attempt to install gdrivefs involved cloning the google repository and compiling from source but I had problems with insufficient memory when compiling and other issues. It also required more steps. I then discovered that there was a github repo for gdrivefs with better instructions that worked well on several of my RPI's. I have written a sync-leavelast.sh script to automate syncing my RPI security camera images with my google drive. This is included in pi-timolo ver 2.51 on github. The leavelast is to avoid conflict and hang if the camera is still writing an image.

Here is a YouTube playlist of my RPI camera video's. Some are just done with motion. The Port Elgin winter day/night time lapse was my test for long duration (47000 images) taken with a very early version of pi-timolo https://www.youtube.com/playlist?list=P ... 2T0pgfaSG0
Regards
Claude ....
Last edited by pageauc on Sun May 10, 2015 12:25 pm, edited 2 times in total.
GitHub - https://github.com/pageauc
YouTube - https://www.youtube.com/user/pageaucp

derosia
Posts: 4
Joined: Wed Oct 15, 2014 11:45 pm

Re: pi-timolo (pi, timelapse, motion, lowlight)

Sun May 10, 2015 2:47 am

Edit: I realized after posting this that I an only running my application during the daytime, so I stripped out all the night sensitivity stuff. So this may not be very useful to you guys.

I needed some motion detection so I used the code in your function checkForMotion. It was maxing out my CPU and causing a significant delay in response between movement and detection/response. This may not matter for most people's application but it was quite noticeable in mine. I ended up changing the step in the for loop to step by 2 instead of 1:

Code: Select all

        for w in range(0, preview_width, 2):
            for h in range(0, preview_height, 2):
                # get the diff of the pixel. Conversion to int
                # is required to avoid unsigned short overflow.
                pixDiff = abs(int(data1[h][w][pixColor]) - int(data2[h][w][pixColor]))
                if  pixDiff > threshold:
                    pixChanges += 1
                if pixChanges > sensitivity:
                    break; # break inner loop
            if pixChanges > sensitivity:
                break; #break outer loop.
So now you're checking 1/4 of the pixels for a change but at least they're alternating pixels so the chances of something noticeable sneaking in between them seems really small. Obviously you'll have to adjust your threshold and sensitivity accordingly but it made a big difference in terms of performance for my application.

pageauc
Posts: 223
Joined: Fri Jan 04, 2013 10:52 pm

Re: pi-timolo (pi, timelapse, motion, lowlight)

Sun May 10, 2015 2:51 am

Thanks

I will do some testing and check it out and implement in next release if OK. For performance reasons, the motion stream image size default is 100x75 pixels so 1875 pixels will be checked instead of 7500 (assuming sensitivity not reached) . For a 720p image 1280x720 that represents a detection gap spacing for one pixel of approx 12.8 x 2 pixels horizontal and 9.6 x 2 vertically. That would represent a square on the 720p image of approx 25.6 x 19.2 pixels for one pixel detection capability. This is a bit of a gap for one pixel so not sure about distant objects like people walking. Sometimes I have to change sensitivity lower to catch motion of people on the sidewalk across the street. Very low sensitivity can generate a lot of false positives due to lighting, camera settings and random motion. A multi frame threshold could be used. Eg 5-10 frames that exceed sensitivity level but this would introduce latency into the system.

Have you found any detection issues with distant movement?

Since the loop is checking images continuously it should help performance but not sure about good detection.. The sensitivity will as you say need to be adjusted or I could just take the existing sensitivity value and divide by 4

Will let you know how testing goes.
Claude ...
Last edited by pageauc on Sun May 10, 2015 11:03 am, edited 5 times in total.
GitHub - https://github.com/pageauc
YouTube - https://www.youtube.com/user/pageaucp

User avatar
electronicsguy
Posts: 156
Joined: Wed Jan 21, 2015 11:20 pm
Contact: Website

Re: pi-timolo (pi, timelapse, motion, lowlight)

Sun May 10, 2015 3:08 am

hi,
great work. a question about motion detection: does this implement a circular buffer? If motion is detected, I'd like some past (maybe 30 sec) images also to be stored, so that the final timelapse video is more fluid.
blog: https://electronicsguy.wordpress.com
github: https://github.com/electronicsguy

pageauc
Posts: 223
Joined: Fri Jan 04, 2013 10:52 pm

Re: pi-timolo (pi, timelapse, motion, lowlight)

Sun May 10, 2015 4:05 am

There is no past buffer stream. Motion uses single small images and does a pixel compare between two of these and if motion is found (enough changed pixels) then it takes a full size image capture. You are thinking like a PVR where video stream is saved on a continuous basis so you can go back and replay before the event as well as after. pi-timolo was not designed to do that. I wrote it for myself and made the code open source on github. Anyone is free to modify the code to suit their own needs. At this time it does what I want it to do. I do improvements but I am just a retired guy with other projects and interests.

The new pi-timolo github release 2.51 implements a quick time lapse feature after motion is detected. This can be adjusted by variables for duration and delay timer with zero being as fast as possible. If the text on image is turned off I have found you can get around 17 images in 10 seconds for 720p size images. There is an initial delay switching between camera motion stream mode and taking the first full size day image and saving to disk, so you might be able to get 2 frames per second rather than 1.7 per second for longer periods on SD. The images still have to be saved so that would be where most of the delay is I believe.

Very fast time lapse requires putting camera in video stream mode at maximum fps for a specific resolution for a period of time (limited by memory) then stopping the stream and extracting individual image frames from the stream buffer and saving extracted images to disk after the capture or alternatively grabbing an image from a continuous stream on the fly, truncating the stream, processing the image and then repeating but saving to disk would still be a delay. A gstream to a remote network resource can also speed things up but more complicated. Keeping a history buffer prior to motion detection gets tricky because the camera would need to detect motion on a continuous basis as well as continuously take a full size image stream for history. You could try running pi-timolo with time lapse and motion detect options turned on and saving to the same folder using date/time sequencing. Not the best solution since time lapse mode is always on generating images all the time. A function could be written to delete time lapse images older than 30 seconds (or specified period) and the quick time lapse on motion would capture after motion was detected but this would also slow things down and you would have to make sure the deletion function was suspended to skip over the desired motion sequence.

I wrote pi-timolo for long duration time lapse so not so interested in very fast time lapse sequences or pre motion history except for the slower long duration time lapse (on image every 2 to 15 minutes). I do run some cameras in both modes since camera is mostly busy doing motion detection and taking one time lapse frame every few minutes does not greatly affect motion detection. There is probably other code that can do specialized requirements. Give the pi-timolo 2.51 quick time lapse after motion a try along with normal time lapse mode turned on and both saving to the same folder. It should smooth out the action and get more motion frames captured after motion is detected but will not have previous history except for the continuous time lapse mode running at the same time. You could write a delete older time lapse images than specified time period function. This method has limits as to how fast it can work and obviously generates more images that would need to be deleted. Sorry for rambling on.

Hope this helps
Regards Claude ....
Last edited by pageauc on Wed May 13, 2015 10:02 am, edited 3 times in total.
GitHub - https://github.com/pageauc
YouTube - https://www.youtube.com/user/pageaucp

pageauc
Posts: 223
Joined: Fri Jan 04, 2013 10:52 pm

Re: pi-timolo (pi, timelapse, motion, lowlight)

Sun May 10, 2015 5:06 am

derosia wrote:Edit: I realized after posting this that I an only running my application during the daytime, so I stripped out all the night sensitivity stuff. So this may not be very useful to you guys.

I needed some motion detection so I used the code in your function checkForMotion. It was maxing out my CPU and causing a significant delay in response between movement and detection/response. This may not matter for most people's application but it was quite noticeable in mine. I ended up changing the step in the for loop to step by 2 instead of 1:

Code: Select all

        for w in range(0, preview_width, 2):
            for h in range(0, preview_height, 2):
                # get the diff of the pixel. Conversion to int
                # is required to avoid unsigned short overflow.
                pixDiff = abs(int(data1[h][w][pixColor]) - int(data2[h][w][pixColor]))
                if  pixDiff > threshold:
                    pixChanges += 1
                if pixChanges > sensitivity:
                    break; # break inner loop
            if pixChanges > sensitivity:
                break; #break outer loop.
So now you're checking 1/4 of the pixels for a change but at least they're alternating pixels so the chances of something noticeable sneaking in between them seems really small. Obviously you'll have to adjust your threshold and sensitivity accordingly but it made a big difference in terms of performance for my application.
GitHub - https://github.com/pageauc
YouTube - https://www.youtube.com/user/pageaucp

pageauc
Posts: 223
Joined: Fri Jan 04, 2013 10:52 pm

Re: pi-timolo (pi, timelapse, motion, lowlight)

Sun May 10, 2015 5:17 am

Night long duration takes around 10 or more seconds for a single image. Motion detection uses day mode and does work at night if there is enough light or contrast eg silhouette but if there is movement is gets smeared like head lights or people moving. I have on occasion caught rabbits moving at night and then standing still for greater than the night long duration period and you get a pretty clear image of the rabbit. One stood at the end of the driveway and moved around slowly so quite a nice short sequence.

Your suggestion about skipping pixels is worth investigating. Especially if it reduces cpu usage. There is also a numpy pixel average function for day/night/twilight detection that might be generating cpu as well. I have tried to reduce the frequency of this function in the code as much as possible.
Thanks for your message.
Claude ...
GitHub - https://github.com/pageauc
YouTube - https://www.youtube.com/user/pageaucp

derosia
Posts: 4
Joined: Wed Oct 15, 2014 11:45 pm

Re: pi-timolo (pi, timelapse, motion, lowlight)

Sun May 10, 2015 1:08 pm

Claude,

I just started experimenting and I am using this code in a kiosk type situation so I am mostly concerned with causing the device to do something when someone is relatively close. I'm also running at a sort of strange resolution 1125x748 and wanted to try stepping in increments of 3. In any event at a threshold of 10 and sensitivity of 100 the camera seemed to keep detecting motion at nothing in particular. I doubled the sensitivity until it stopped getting triggered by just the light in my apartment and ended up at 3200 even stepping by 3. But if I'm doing the math correctly (1125x748 = 841,500 pixels) then I'm checking1/9th of them = 93,500. If even 3200 of them are changing that's 3.4% of the sample pixels and is plenty sensitive enough for my particular application. I'm sure most people are looking for a lot more sensitivity so, your mileage may vary.

-James

pageauc
Posts: 223
Joined: Fri Jan 04, 2013 10:52 pm

Re: pi-timolo (pi, timelapse, motion, lowlight)

Sun May 10, 2015 9:00 pm

James.

Thanks for your post. I have run across lighting issues like street lights that change in intensity or room lighting changes slightly eg when a cloud goes over. I have usually been able to resolve these issues false positives by increasing the threshold (how much a pixel has to change before it is counted as changed). Reviewing the images helps. Normally a value of 10 is OK but sometimes it helps to increase the value to 20, 30 or more. The pixel value is based on green. If threshold change is dramatic you may have adjust sensitivity a bit. I am looking at trying to automate this and twilight threshold.

Twilight threshold would be fairly easy to auto adjust since you are looking for extreme pixel averages near 0 or 255. If these values are extreme then the twilight threshold can be increased or decreased to compensate and value saved to disk. That is because the camera in day mode does an auto white balance but as ambient lighting decreases the image gets too dark (below twilight threshold) so the camera needs to be switched to night mode with a long AWB to determine appropriate ISO and Exposure time until full night lighting is reached where the camera ISO and Exposure time are at max.

Regards Claude ....
GitHub - https://github.com/pageauc
YouTube - https://www.youtube.com/user/pageaucp

Applefamily
Posts: 13
Joined: Sat Nov 16, 2013 11:54 pm

Re: pi-timolo (pi, timelapse, motion, lowlight)

Sat Jul 04, 2015 1:17 am

Can I change the save directory to an external Hard drive? I did not see a configuration for this. Will I need to change the Python script?

pageauc
Posts: 223
Joined: Fri Jan 04, 2013 10:52 pm

Re: pi-timolo (pi, timelapse, motion, lowlight)

Sat Jul 04, 2015 2:20 am

No you do not need to change pi-timolo code but config.py will need to point to the drive mount point.or a link to the mount point.
I attached an external usb Hard drive and partitioned and formated drive to ext4 using gparted (you will need to apt-get this).

Code: Select all

sudo apt-get install gparted
This is a gui app so you will need to run from the pi desktop. google for additional info on using gparted.
You can also use a normal Fat32 formatted drive if you like. I then created a mount point for the drive. This can be in the /mnt folder but in my case I decided to put it in the /home/pi folder

Code: Select all

cd ~
mkdir mnt
mkdir mnt/hd_usb1
Now create an entry in the /etc/fstab to auto mount your new drive.

Code: Select all

sudo nano /etc/fstab
use nano to add a line in fstab file to mount your hard drive. In my case the line below. This will be different if you are using a fat32 formatted drive. Google if you need more detail
sample fstab entry to mount ext4 formatted drive Modify as required

Code: Select all

/dev/sda1     /home/pi/mnt/hd_usb1  ext4 rw,defaults 0  0
or
sample fstab entry to mount fat32 formatted drive. Modify as required

Code: Select all

/dev/sda1      /home/pi/mnt/hd_usb1   vfat  auto,user,rw,uid=pi,gid=pi   0  0
mount the drive and check if it mounts OK. fix any issues.

Code: Select all

sudo mount  /dev/sda1
mount
cd ~
cd mnt/hd_usb1
mkdir data1
I then went into pi-timolo folder and created a link to the drive.
eg

Code: Select all

cd ~
cd pi-timolo
ln -s /home/pi/mnt/hd_usb1/data1 folder_name
Test to make sure the link works and you can write files to the hard drive link that points to the HD mount point. Please note you can change the mount point locations and folder names to suit your needs.
Once the mount point is working you can change the pi-timolo config.py to point the the new folder_name link folder.
change the motionDir= and/or timelapseDir= entries to point to the appropriate folder link and folder name
eg

Code: Select all

motionDir=folder_name
This is just a sample guide. If you need more detail just google and you should be able to get more information for what you want.
GitHub - https://github.com/pageauc
YouTube - https://www.youtube.com/user/pageaucp

Applefamily
Posts: 13
Joined: Sat Nov 16, 2013 11:54 pm

Re: pi-timolo (pi, timelapse, motion, lowlight)

Sun Jul 05, 2015 2:13 am

Thank you worked perfectly as you already knew.

more questions
1 does the make movie use the config.py to pick the files? I was in the pi-timolo folder typed sudo ./makemovie.py and presto buzz wizz a movie was formed in this folder.
2 does makemovie use both the motion and timelapsed images? It seemed to only use the timelapsed.

*Update I'm home and found by opening the Makemovie.py with nano that there are variables to change both the source folder and output. I'm building my new movie with images saved to the USB drive and outputting to this folder.
Last edited by Applefamily on Mon Jul 06, 2015 1:58 am, edited 1 time in total.

Return to “Graphics, sound and multimedia”