User avatar
jbeale
Posts: 3518
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: PiKrellCam: motion vector detect + OSD web interface

Thu Nov 19, 2015 8:13 pm

lassiko wrote:Not sure if already answered, but how can i take a still from a shell, what is the command?
I'm not sitting at my Pi, but based on some info at http://billw2.github.io/pikrellcam/pikrellcam.html I believe you may be able to simply do:

Code: Select all

echo "still" > ~/pikrellcam/www/FIFO
UPDATE: Yes- just tested it; this works. So you could trigger a still from any external process, sensor etc.

billw
Posts: 404
Joined: Tue Sep 18, 2012 8:23 pm

Re: PiKrellCam: motion vector detect + OSD web interface

Fri Nov 20, 2015 1:32 am

A git update is ready with changes to the timelapse display, the
/var/run/pikrellcam/state file and the preview clean code.

@jbeale - If it is convenient, try running for a time from a terminal:

Code: Select all

pikrellcam -debug
Ignore the initial "encoder not clear (1) -> skipping mjpeg frame.",
lines, then there will be output for motion records when I've detected
a potential race. I've made some changes so that hopefully clean
previews will work better.

You can get two relevant debug outputs:

Code: Select all

vidname: holdoff not clear -> rename skipped
or
vidname: encoder not clear -> preview save delayed
If you get more dirty previews, see if video is the same as
any of the "vidnames" in debug output you get.

I'm keeping a count of the I420 frames sent to the jpeg encoder and
the jpegs the encoder sends back to pikrellcam.
The "encoder not clear" means that when I want to send a frame that
is clean to the encoder, the encoder has not yet sent back the
jpeg I expected from the previous I420 dirty frame.

The "holdoff not clear" means the /var/run/pikrellcam/mjpeg.jpg
(the clean one we want) was going to be renamed with a dirty jpeg
before it got copied into the final preview jpeg.

User avatar
jbeale
Posts: 3518
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: PiKrellCam: motion vector detect + OSD web interface

Fri Nov 20, 2015 2:08 am

In case of interest, I took a still frame and a video at night with PiKrellCam, using the "night" exposure setting. At the default 24 fps, the video frame exposure is 1/24 sec long, but the still frame exposure is 1/4 second long. That makes a difference on image quality and motion blur, as you can see. The subject is a timer with a second hand that makes one full 360 degree revolution every second. On the still image you can see the hand sweeps out 90 degrees, showing that it is a 1/4 second exposure. The video still sweeps about 15 degrees.
Image
Image

User avatar
jbeale
Posts: 3518
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: PiKrellCam: motion vector detect + OSD web interface

Fri Nov 20, 2015 2:29 am

@billw : new version looks good! Bug seems to have been fixed, at least I couldn't trigger it so far. Debug output looks like below:

Code: Select all

[email protected] ~ $ pikrellcam -debug
debugging...
encoder not clear (1) -> skipping mjpeg frame.
encoder not clear (2) -> skipping mjpeg frame. 
encoder not clear (3) -> skipping mjpeg frame.
  Syncing recv/send counts.
Saving to /home/pi/pikrellcam/media/videos/motion_2015-11-19_18.18.56_0.mp4: 0.500 secs Interleaving
encoder not clear (1) -> skipping mjpeg frame.
UPLOADED /home/pi/pikrellcam/media/stills/motion_2015-11-19_18.18.56_0.jpg
Saving to /home/pi/pikrellcam/media/videos/motion_2015-11-19_18.19.03_1.mp4: 0.500 secs Interleaving
UPLOADED /home/pi/pikrellcam/media/stills/motion_2015-11-19_18.19.03_1.jpg
Saving to /home/pi/pikrellcam/media/videos/motion_2015-11-19_18.19.18_2.mp4: 0.500 secs Interleaving
motion_2015-11-19_18.19.25_3.mp4: holdoff not clear -> rename skipped
UPLOADED /home/pi/pikrellcam/media/stills/motion_2015-11-19_18.19.18_2.jpg
UPLOADED /home/pi/pikrellcam/media/stills/motion_2015-11-19_18.19.25_3.jpg
Saving to /home/pi/pikrellcam/media/videos/motion_2015-11-19_18.19.54_4.mp4: 0.500 secs Interleaving
UPLOADED /home/pi/pikrellcam/media/stills/motion_2015-11-19_18.19.54_4.jpg
Saving to /home/pi/pikrellcam/media/videos/motion_2015-11-19_18.20.16_5.mp4: 0.500 secs Interleaving
UPLOADED /home/pi/pikrellcam/media/stills/motion_2015-11-19_18.20.16_5.jpg
encoder not clear (2) -> skipping mjpeg frame.
Saving to /home/pi/pikrellcam/media/videos/motion_2015-11-19_18.22.12_6.mp4: 0.500 secs Interleaving
UPLOADED /home/pi/pikrellcam/media/stills/motion_2015-11-19_18.22.12_6.jpg
motion_2015-11-19_18.22.36_8.mp4: encoder not clear -> preview save delayed
motion_2015-11-19_18.22.36_8.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.22.36_8.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.22.36_8.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.22.36_8.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.22.36_8.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.22.36_8.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.22.36_8.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.22.36_8.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.22.36_8.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.22.36_8.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.22.36_8.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.22.36_8.mp4: holdoff not clear -> rename skipped
Saving to /home/pi/pikrellcam/media/videos/motion_2015-11-19_18.22.27_7.mp4: 0.500 secs Interleaving
motion_2015-11-19_18.22.36_8.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.22.36_8.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.22.36_8.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.22.36_8.mp4: holdoff not clear -> rename skipped
  Syncing recv/send counts.
encoder not clear (1) -> skipping mjpeg frame.
encoder not clear (2) -> skipping mjpeg frame.
encoder not clear (3) -> skipping mjpeg frame.
  Syncing recv/send counts.
UPLOADED /home/pi/pikrellcam/media/stills/motion_2015-11-19_18.22.27_7.jpg
Saving to /home/pi/pikrellcam/media/videos/motion_2015-11-19_18.22.36_8.mp4: 0.500 secs Interleaving
encoder not clear (1) -> skipping mjpeg frame.
UPLOADED /home/pi/pikrellcam/media/stills/motion_2015-11-19_18.22.36_8.jpg
motion_2015-11-19_18.22.57_10.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.22.57_10.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.22.57_10.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.22.57_10.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.22.57_10.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.22.57_10.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.22.57_10.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.22.57_10.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.22.57_10.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.22.57_10.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.22.57_10.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.22.57_10.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.22.57_10.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.22.57_10.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.22.57_10.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.22.57_10.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.22.57_10.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.22.57_10.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.22.57_10.mp4: holdoff not clear -> rename skipped
encoder not clear (2) -> skipping mjpeg frame.
motion_2015-11-19_18.22.57_10.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.22.57_10.mp4: holdoff not clear -> rename skipped
Saving to /home/pi/pikrellcam/media/videos/motion_2015-11-19_18.22.43_9.mp4: 0.500 secs Interleaving
encoder not clear (3) -> skipping mjpeg frame.
  Syncing recv/send counts.
encoder not clear (1) -> skipping mjpeg frame.
encoder not clear (2) -> skipping mjpeg frame.
encoder not clear (3) -> skipping mjpeg frame.
  Syncing recv/send counts.
encoder not clear (1) -> skipping mjpeg frame.
Saving to /home/pi/pikrellcam/media/videos/motion_2015-11-19_18.22.57_10.mp4: 0.500 secs Interleaving
UPLOADED /home/pi/pikrellcam/media/stills/motion_2015-11-19_18.22.57_10.jpg
UPLOADED /home/pi/pikrellcam/media/stills/motion_2015-11-19_18.22.57_10.jpg
encoder not clear (2) -> skipping mjpeg frame.
Saving to /home/pi/pikrellcam/media/videos/motion_2015-11-19_18.23.11_11.mp4: 0.500 secs Interleaving
UPLOADED /home/pi/pikrellcam/media/stills/motion_2015-11-19_18.23.11_11.jpg
motion_2015-11-19_18.24.12_13.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.24.12_13.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.24.12_13.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.24.12_13.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.24.12_13.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.24.12_13.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.24.12_13.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.24.12_13.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.24.12_13.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.24.12_13.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.24.12_13.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.24.12_13.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.24.12_13.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.24.12_13.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.24.12_13.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.24.12_13.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.24.12_13.mp4: holdoff not clear -> rename skipped
Saving to /home/pi/pikrellcam/media/videos/motion_2015-11-19_18.24.01_12.mp4: 0.500 secs Interleaving
motion_2015-11-19_18.24.12_13.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.24.12_13.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.24.12_13.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.24.12_13.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.24.12_13.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.24.12_13.mp4: holdoff not clear -> rename skipped
motion_2015-11-19_18.24.12_13.mp4: holdoff not clear -> rename skipped
encoder not clear (3) -> skipping mjpeg frame.
  Syncing recv/send counts.
encoder not clear (1) -> skipping mjpeg frame.
encoder not clear (2) -> skipping mjpeg frame.
encoder not clear (3) -> skipping mjpeg frame.
  Syncing recv/send counts.
encoder not clear (1) -> skipping mjpeg frame.
Saving to /home/pi/pikrellcam/media/videos/motion_2015-11-19_18.24.12_13.mp4: 0.500 secs Interleaving
UPLOADED /home/pi/pikrellcam/media/stills/motion_2015-11-19_18.24.12_13.jpg
UPLOADED /home/pi/pikrellcam/media/stills/motion_2015-11-19_18.24.12_13.jpg
Saving to /home/pi/pikrellcam/media/videos/motion_2015-11-19_18.24.30_14.mp4: 0.500 secs Interleaving
UPLOADED /home/pi/pikrellcam/media/stills/motion_2015-11-19_18.24.30_14.jpg

billw
Posts: 404
Joined: Tue Sep 18, 2012 8:23 pm

Re: PiKrellCam: motion vector detect + OSD web interface

Fri Nov 20, 2015 4:18 am

jbeale wrote:@billw : new version looks good! Bug seems to have been fixed, at least I couldn't trigger it so far. Debug output looks like below:
It's interesting that you get a lot "encoder not clear" messages and when I run I almost
never get them. Possibly an issue affected by other loadings on the system and it
definitely tells me where I need to look some more in case there still might be some dirty
previews that slip through. Frequent "encoder not clear" probably means the jpeg encoder
is on the edge of being able to keep up with the buffers being fed to it. If there are times
when you start to get dirty previews again, one experiment to try would be to increase the
mjpeg divider from say 4 to 6.
Anyway, at least I'm glad it's better for you.

User avatar
jbeale
Posts: 3518
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: PiKrellCam: motion vector detect + OSD web interface

Fri Nov 20, 2015 4:55 am

@billw : FWIW my test device here was a Model A+ unit (or maybe just A?); maybe the Model 2 is better in that way. Immediately after each event it transmits the still JPEG via wifi to a remote site, maybe that is the loading factor. At any rate, for now I'd call it good.

I'm looking forward, if/when time permits, to the possibility of getting per-frame motion vector text output :-)
cheers,
John

tomtgrp
Posts: 41
Joined: Thu Jul 09, 2015 8:35 am

Re: PiKrellCam: motion vector detect + OSD web interface

Fri Nov 20, 2015 2:34 pm

billw wrote:
jbeale wrote:@billw : new version looks good! Bug seems to have been fixed, at least I couldn't trigger it so far. Debug output looks like below:
It's interesting that you get a lot "encoder not clear" messages and when I run I almost
never get them. Possibly an issue affected by other loadings on the system and it
definitely tells me where I need to look some more in case there still might be some dirty
previews that slip through. Frequent "encoder not clear" probably means the jpeg encoder
is on the edge of being able to keep up with the buffers being fed to it. If there are times
when you start to get dirty previews again, one experiment to try would be to increase the
mjpeg divider from say 4 to 6.
Anyway, at least I'm glad it's better for you.
Hello,

i also get these in my pi2 (jessie), no overclocking, no other programs runnning.
mjpeg divider 1, 24 fps, 1080p, mjpeg_width 800, mjpeg_quality 12, motion_area_min_side 80
occurence nearly every second (perhaps add a timestamp in front of the debugging messages?)

Setting mjpeg divider to 2 reduces it to 1-2 messages per minute

When starting the rtsp stream the messages occur every second. Load rises to 80% on one cpu,
then after some time the preview image freezes completely (annotate time stops counting up), no further images are generated, so pikrellcam has to be stopped/started.

This sometimes also happened without adding additional load to the pi. You can avoid it by lowering
the fps or increasing mjpeg divider, bot on the other hand a smooth and fast preview is the goal.

Hope this helps.

Br Tom

Code: Select all

gst-rtsp-server, test-launch, pipe:
"(  multifilesrc location=/var/run/pikrellcam/mjpeg.jpg index=1 caps=\"image/jpeg,framerate=25/1\" \
        ! jpegdec ! timeoverlay halignment=right valignment=top \
        ! clockoverlay halignment=left valignment=top time-format=\"%Y/%m/%d %H:%M:%S\" \
        ! textoverlay halignment=center valignment=bottom text=\"Test\" \
        ! queue ! omxh264enc control-rate=1 target-bitrate=1250000 \
        ! video/x-h264,stream-format=byte-stream,profile=high ! h264parse !  rtph264pay name=pay0 pt=96 )"

Code: Select all

debugging...
encoder not clear (1) -> skipping mjpeg frame.
encoder not clear (2) -> skipping mjpeg frame.
encoder not clear (3) -> skipping mjpeg frame.
  Syncing recv/send counts.
encoder not clear (1) -> skipping mjpeg frame.
encoder not clear (2) -> skipping mjpeg frame.
encoder not clear (3) -> skipping mjpeg frame.
  Syncing recv/send counts.
encoder not clear (1) -> skipping mjpeg frame.
encoder not clear (2) -> skipping mjpeg frame.
encoder not clear (3) -> skipping mjpeg frame.
  Syncing recv/send counts.
encoder not clear (1) -> skipping mjpeg frame.
encoder not clear (2) -> skipping mjpeg frame.
encoder not clear (3) -> skipping mjpeg frame.
  Syncing recv/send counts.
encoder not clear (1) -> skipping mjpeg frame.
encoder not clear (2) -> skipping mjpeg frame.
encoder not clear (3) -> skipping mjpeg frame.
  Syncing recv/send counts.
encoder not clear (1) -> skipping mjpeg frame.
encoder not clear (2) -> skipping mjpeg frame.
encoder not clear (3) -> skipping mjpeg frame.
  Syncing recv/send counts.
encoder not clear (1) -> skipping mjpeg frame.
encoder not clear (2) -> skipping mjpeg frame.
encoder not clear (3) -> skipping mjpeg frame.
  Syncing recv/send counts.
encoder not clear (1) -> skipping mjpeg frame.
encoder not clear (2) -> skipping mjpeg frame.
encoder not clear (3) -> skipping mjpeg frame.
  Syncing recv/send counts.
encoder not clear (1) -> skipping mjpeg frame.
encoder not clear (2) -> skipping mjpeg frame.
encoder not clear (3) -> skipping mjpeg frame.
  Syncing recv/send counts.
encoder not clear (1) -> skipping mjpeg frame.
encoder not clear (2) -> skipping mjpeg frame.
encoder not clear (3) -> skipping mjpeg frame.
  Syncing recv/send counts.
encoder not clear (1) -> skipping mjpeg frame.
encoder not clear (2) -> skipping mjpeg frame.
encoder not clear (3) -> skipping mjpeg frame.
  Syncing recv/send counts.
encoder not clear (1) -> skipping mjpeg frame.
encoder not clear (2) -> skipping mjpeg frame.
encoder not clear (3) -> skipping mjpeg frame.
  Syncing recv/send counts.
encoder not clear (1) -> skipping mjpeg frame.
encoder not clear (2) -> skipping mjpeg frame.
encoder not clear (3) -> skipping mjpeg frame.
  Syncing recv/send counts.
encoder not clear (1) -> skipping mjpeg frame.
encoder not clear (2) -> skipping mjpeg frame.
encoder not clear (3) -> skipping mjpeg frame.
  Syncing recv/send counts.
encoder not clear (1) -> skipping mjpeg frame.
encoder not clear (2) -> skipping mjpeg frame.
encoder not clear (3) -> skipping mjpeg frame.
  Syncing recv/send counts.
encoder not clear (1) -> skipping mjpeg frame.
encoder not clear (2) -> skipping mjpeg frame.
encoder not clear (3) -> skipping mjpeg frame.
  Syncing recv/send counts.
encoder not clear (1) -> skipping mjpeg frame.
encoder not clear (2) -> skipping mjpeg frame.
encoder not clear (3) -> skipping mjpeg frame.
  Syncing recv/send counts.
encoder not clear (1) -> skipping mjpeg frame.
encoder not clear (2) -> skipping mjpeg frame.
encoder not clear (3) -> skipping mjpeg frame.
  Syncing recv/send counts.
encoder not clear (1) -> skipping mjpeg frame.
encoder not clear (2) -> skipping mjpeg frame.
encoder not clear (3) -> skipping mjpeg frame.
  Syncing recv/send counts.

User avatar
jbeale
Posts: 3518
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: PiKrellCam: motion vector detect + OSD web interface

Fri Nov 20, 2015 2:57 pm

That's a good point about the system load. FWIW all my systems run with mjpeg_divider 4. If it is set lower I would expect the CPU to at some point run out of time to handle each frame, as you experienced. On a Pi2, I don't know that the code makes direct use of more than one CPU at a time for the processing. If the smoothest preview is the goal, I would expect that streaming H.264 instead of MJPEG would be the answer.

tomtgrp
Posts: 41
Joined: Thu Jul 09, 2015 8:35 am

Re: automatic archiving from crontab

Sat Nov 21, 2015 4:17 pm

jbeale wrote:The below commands work if I execute them directly from the bash prompt as user pi, but they do not work when put into the crontab file. What am I doing wrong?

Code: Select all

# Move yesterday's stills and video to that date's archive folder
0 2 * * * echo "archive_still day `date --date="yesterday" +'%F'`" > /home/pi/pikrellcam/www/FIFO
0 5 * * * echo "archive_video day `date --date="yesterday" +'%F'`" > /home/pi/pikrellcam/www/FIFO
EDIT: looks like I should make it a script and call that; such as a /home/pi/pikrellcam/scripts/do-archive

Code: Select all

#!/bin/bash

# Move yesterday's stills and video to that date's archive folder
echo "archive_still day `date --date="yesterday" +'%F'`" > /home/pi/pikrellcam/www/FIFO
echo "archive_video day `date --date="yesterday" +'%F'`" > /home/pi/pikrellcam/www/FIFO

# delete old files and directories to save storage space
find /home/pi/pikrellcam/media/archive -mtime +7 -exec rm {} \;
find /home/pi/pikrellcam/media/archive -mtime +7 -type d -exec rmdir {} \;

What about this idea? (to be improved, e.g. error checks if _OLDEST is empty)
It keeps as much as possible files present.

Code: Select all

#!/bin/bash

_ARCHIVE_OLDER_DAYS="-2 Days"

# Move _ARCHIVE_OLDER_DAYS stills and video to that date's archive folder
echo "archive_still day `date --date="$_ARCHIVE_OLDER_DAYS" +'%F'`"  > /home/pi/pikrellcam/www/FIFO
echo "archive_video day `date --date="$_ARCHIVE_OLDER_DAYS" +'%F'`"  > /home/pi/pikrellcam/www/FIFO

# if used space is over threshold, delete old files and directories to save storage space until threshold reached
# this keeps as much as possible files present
_USED_SPACE=`df  | grep root | awk '{print $5}' |  sed -e "s/\%//g"`
echo "used space: $_USED_SPACE%"

_USED_SPACE_LIMIT=95    #percent

if ((_USED_SPACE > _USED_SPACE_LIMIT))
then
 echo -e "used space over $_USED_SPACE_LIMIT%, deleting old files..."

 _LOOP=0;

 while ((_USED_SPACE >  _USED_SPACE_LIMIT && _LOOP < 100))
 do
  _OLDEST=`find /home/pi/pikrellcam/media/archive/ -type f -printf '%T+ %p\n' | sort | head -n 1 | cut -d "+" -f 1 | sed -e "s/\-/\//g"`
  echo "oldest files:$_OLDEST"

  echo "delete oldest folder:/home/pi/pikrellcam/media/archive/$_OLDEST"
  rm -rfv "/home/pi/pikrellcam/media/archive/$_OLDEST"
  sync

  _USED_SPACE=`df  | grep root | awk '{print $5}' |  sed -e "s/\%//g"`
  echo "used space now: $_USED_SPACE%"
  _LOOP=$((_LOOP + 1))
 done

 echo "done"
 df
else
 echo -e "used space below $_USED_SPACE_LIMIT%, nothing to do!"
fi


User avatar
jbeale
Posts: 3518
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: automatic archiving from crontab

Sat Nov 21, 2015 5:24 pm

tomtgrp wrote:What about this idea? (to be improved, e.g. error checks if _OLDEST is empty)
It keeps as much as possible files present.
Thanks for that code, I like the idea. I have not upgraded to "Jessie"; still on the older raspbian from April 2015 so I must have a different version of bash, because the first "IF" comparison gave me a syntax error. At any rate, the slightly modified code below works for me:

Code: Select all

#!/bin/bash

_ARCHIVE_OLDER_DAYS="-1 Days"

# Move _ARCHIVE_OLDER_DAYS stills and video to that date's archive folder
echo "archive_still day `date --date="$_ARCHIVE_OLDER_DAYS" +'%F'`"  > /home/pi/pikrellcam/www/FIFO
echo "archive_video day `date --date="$_ARCHIVE_OLDER_DAYS" +'%F'`"  > /home/pi/pikrellcam/www/FIFO

# if used space is over threshold, delete old files and directories to save storage space until threshold reached
# this keeps as much as possible files present
_USED_SPACE=`df  | grep root | awk '{print $5}' |  sed -e "s/\%//g"`
# echo "used space: $_USED_SPACE%"

_USED_SPACE_LIMIT=90    #percent

if [[ ( "$_USED_SPACE" > "$_USED_SPACE_LIMIT" ) ]] ; then
 echo -e "used space over $_USED_SPACE_LIMIT%, deleting old files..."

 _LOOP=0;

 while [[ ( "$_USED_SPACE" > "$_USED_SPACE_LIMIT" )  && ( "$_LOOP" < "100" ) ]] ; do
  _OLDEST=`find /home/pi/pikrellcam/media/archive/ -type f -printf '%T+ %p\n' | sort | head -n 1 | cut -d "+" -f 1 | sed -e "s/\-/\//g"`
  echo "oldest files:$_OLDEST"

  echo "delete oldest folder:/home/pi/pikrellcam/media/archive/$_OLDEST"
  rm -rfv "/home/pi/pikrellcam/media/archive/$_OLDEST"
  sync

  _USED_SPACE=`df  | grep root | awk '{print $5}' |  sed -e "s/\%//g"`
  echo "used space now: $_USED_SPACE%"
  _LOOP=$((_LOOP + 1))
 done

 echo "done"
 df
else
 echo -e "used space below $_USED_SPACE_LIMIT%, nothing to do!"
fi

jit
Posts: 33
Joined: Fri Apr 18, 2014 2:52 pm

Re: PiKrellCam: motion vector detect + OSD web interface

Sat Nov 21, 2015 11:15 pm

@billw - Many thanks for your impressive work on pikrellcam. I've been running a 'motion' based setup for sometime and have only recently come back to the forums to see what's new. This is truly amazing.

One thing that I would like to ask you and others is around what the best setup is to go back and forth into day/night mode. e.g. what are the best camera settings to use. I saw your suggestion on using a user script to switch to the appropriate setup upon start up, but wondered if any functionality had been added in to support running specific 'at' commands at startup?

With regard to the whole day/night issue, I was wondering whether it might be possible to auto detect the current conditions based on the picture and auto adjust the camera settings to compensate? Although I expect that's harder than it sounds.

I've used tmpfs for the storage of videos (concious of the fact that SD cards can fail due to excessive writes). Since tmpfs space is limited, is there a way to specify max size/length of a recording such that it could then be processed by a user script and potentially pushed onto remote storage whilst allowing further recording of the event to take place?

User avatar
jbeale
Posts: 3518
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: PiKrellCam: motion vector detect + OSD web interface

Sun Nov 22, 2015 2:15 am

jit wrote:One thing that I would like to ask you and others is around what the best setup is to go back and forth into day/night mode.
IMHO after some testing I've concluded that the R-Pi camera is not useful for motion detection at night (it's OK if you just want long-exposure still frames). The PiKrellCam software is the best thing out there for motion detection and it does a great job from dawn to dusk, but it doesn't change the basic properties of the camera. (Of course you can make it work by adding lights, but you have to use a fairly unreasonable level of illumination.)

billw
Posts: 404
Joined: Tue Sep 18, 2012 8:23 pm

Re: PiKrellCam: motion vector detect + OSD web interface

Sun Nov 22, 2015 4:23 am

jit wrote:@billw - Many thanks for your impressive work on pikrellcam. I've been running a 'motion' based setup for sometime and have only recently come back to the forums to see what's new. This is truly amazing.
Thanks.
... but wondered if any functionality had been added in to support running specific 'at' commands at startup?
Yes, you can run at commands at start. I have this in my at-commands.conf:

Code: Select all

daily start "@motion load_regions window"
I've just started a Help page which I intend to cover pikrellcam in detail in one place. Right now
you have to read the config files and this forum and usage info is unfortunately really spread
out.
With regard to the whole day/night issue, I was wondering whether it might be possible to auto detect the current conditions based on the picture and auto adjust the camera settings to compensate?
I think jbeale is right on this. You can compensate some with the at-commands discussed in
viewtopic.php?p=793722#p793722
but that just improves things a bit at a cost of slow framerate.
I've used tmpfs for the storage of videos (concious of the fact that SD cards can fail due to excessive writes). Since tmpfs space is limited, is there a way to specify max size/length of a recording such that it could then be processed by a user script and potentially pushed onto remote storage whilst allowing further recording of the event to take place?
A max length has been on my todo, but other things have been coming up and I haven't got
to it yet. But I think the best thing is plug a USB flash drive into the Pi and have pikrellcam
auto mount it on the media directory at startup (scripts/startup). I've been doing that for months
and I don't worry about the SD card a bit. Somehow the possibility of wearing out a USB disk
doesn't bother me.

Right now, to bias pikrellcam towards making shorter length recordings, set the event_gap
time lower so the motion videos have a better chance of ending themselves.

tomtgrp
Posts: 41
Joined: Thu Jul 09, 2015 8:35 am

Re: PiKrellCam: motion vector detect + OSD web interface

Sun Nov 22, 2015 6:46 pm

Problems on small screens:

Hello,

just noticed page rendering problems on small screens viewing the Videos/Thumbs/Stills pages.
You can see it if you make the browser window small in x-direction.
current.jpg
current.jpg (17.09 KiB) Viewed 4599 times
Changed media-archive.php, line 646 from float: right to float: top, now page is visible. Maybe not the best solution, but a fix.
new.jpg
new.jpg (34.76 KiB) Viewed 4599 times

billw
Posts: 404
Joined: Tue Sep 18, 2012 8:23 pm

Re: PiKrellCam: motion vector detect + OSD web interface

Sun Nov 22, 2015 6:56 pm

jbeale, pikrellcam can write motion.xxxx.txt files (not pushed to git yet) with:

Code: Select all

 2.167,  26,  40,  10, -31, 1061,  148
 2.333,  28,  44,   0,  -9,   81,  220
 3.000,  23,  31,  -6,   8,  100,   66
...
Each line is is a composite frame vector for each motion event with:

Code: Select all

time,  x, y, dx, dy,  mag2, mag2_count 
Should there be a header line for each text file: # time, x, y, ... ?

The x,y coordinates are currently in motion frame coords. Should they be
scaled to mjpeg_width or video_width coordinates?

I have the web page able to delete the text files when you delete a video, but
the archiving does not deal with them yet. Should archiving move the txt files
or maybe delete them? If your motion_end does not deal with them and videos
are archived, there will be dangling txt files in the media/videos directory.

And I'm just now thinking I should square root the mag2 so you have a value more
related to the configured magnitude limit.

jit
Posts: 33
Joined: Fri Apr 18, 2014 2:52 pm

Re: PiKrellCam: motion vector detect + OSD web interface

Sun Nov 22, 2015 10:31 pm

jbeale wrote:
jit wrote:One thing that I would like to ask you and others is around what the best setup is to go back and forth into day/night mode.
IMHO after some testing I've concluded that the R-Pi camera is not useful for motion detection at night (it's OK if you just want long-exposure still frames). The PiKrellCam software is the best thing out there for motion detection and it does a great job from dawn to dusk, but it doesn't change the basic properties of the camera. (Of course you can make it work by adding lights, but you have to use a fairly unreasonable level of illumination.)
That is indeed what I'm finding. I have a PIR floodlight setup in conjunction with one pi, that works very well. No need to change any settings at night. However I've just tried to setup another where its just using street lighting, but that is struggling. For the time being I've settled on the following settings (thanks for the pointer @billw, since that appears to be better than nothing. But I think I'll have to install another PIR floodlight.

Code: Select all

daily dusk "@brightness 50"
daily dusk "@video_fps 4"
daily dusk "@mjpeg_divider 1"
daily dusk "@mp4_box_fps 4"
daily dusk "@exposure_mode night"
daily sunrise "@brightness 50"
daily sunrise "@video_fps 24"
daily sunrise "@mjpeg_divider 6"
daily sunrise "@exposure_mode auto"

jit
Posts: 33
Joined: Fri Apr 18, 2014 2:52 pm

Re: PiKrellCam: motion vector detect + OSD web interface

Sun Nov 22, 2015 10:49 pm

You can compensate some with the at-commands discussed in
viewtopic.php?p=793722#p793722
but that just improves things a bit at a cost of slow framerate.
Thanks for the tip. I've also added in use of exposure_mode night, which seems to make a considerable difference at night (at least to the OSD). Btw, I love the fact that you can tinker with the camera settings and see them update live on the OSD.
I've used tmpfs for the storage of videos (concious of the fact that SD cards can fail due to excessive writes). Since tmpfs space is limited, is there a way to specify max size/length of a recording such that it could then be processed by a user script and potentially pushed onto remote storage whilst allowing further recording of the event to take place?
A max length has been on my todo, but other things have been coming up and I haven't got
to it yet. But I think the best thing is plug a USB flash drive into the Pi and have pikrellcam
auto mount it on the media directory at startup (scripts/startup). I've been doing that for months
and I don't worry about the SD card a bit. Somehow the possibility of wearing out a USB disk
doesn't bother me.

Right now, to bias pikrellcam towards making shorter length recordings, set the event_gap
time lower so the motion videos have a better chance of ending themselves.
Thanks, I just checked and looks like I'd already set the event_gap to the minimum of 5. Ironically after having pikrellcam running for a couple of weeks of testing internally with great success, I put it by the window today facing outwards and ran into tmpfs space issues. To be fair there was quite a bit of activity outside. I also underestimated how much tmpfs would be available, the PI B only has 182M free, where as my shiny new PI 2 Model B has 426M free. I think I may have to switch back to writing to SD. Wondering if there's a way around this, although I can't think of one right now. Thinking if its possible to spill onto SD if tmpfs is full... probably not if your in the middle of a recording. Although probably not worth the effort given how cheap USB disks / SD cards are these days.

When the disk space issue occurred, everything pretty much stopped - the OSD showed a preview of that moment in time. During the time that it happened, some uploads were in progress and eventually cleared some space in tmpfs, but unfortunately pikrellcam didn't spring back into action.

Code: Select all

15:30:39 : mjpeg_callback: mjpeg_encoder file write error.  No space left on device[code]
Last edited by jit on Tue Nov 24, 2015 9:12 am, edited 2 times in total.

User avatar
jbeale
Posts: 3518
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: PiKrellCam: motion vector detect + OSD web interface

Mon Nov 23, 2015 2:03 pm

billw wrote:jbeale, pikrellcam can write motion.xxxx.txt files (not pushed to git yet) with:

Code: Select all

 2.167,  26,  40,  10, -31, 1061,  148
 2.333,  28,  44,   0,  -9,   81,  220
 3.000,  23,  31,  -6,   8,  100,   66
...
Each line is is a composite frame vector for each motion event with:

Code: Select all

time,  x, y, dx, dy,  mag2, mag2_count 
Should there be a header line for each text file: # time, x, y, ... ?

The x,y coordinates are currently in motion frame coords. Should they be
scaled to mjpeg_width or video_width coordinates?

I have the web page able to delete the text files when you delete a video, but
the archiving does not deal with them yet. Should archiving move the txt files
or maybe delete them? If your motion_end does not deal with them and videos
are archived, there will be dangling txt files in the media/videos directory.

And I'm just now thinking I should square root the mag2 so you have a value more
related to the configured magnitude limit.
@billw : Fantastic work, I am eager to try this out! Do you think the filename extension should be .csv (comma separated values) to be more informative and separate from more generic text? Yes i like the column header, a few more bytes doesn't hurt. About motion frame coords, I dont know exactly what you are using but I would select whichever one is less likely to change based on user config if possible. I would keep the CSV file together with the video even in the archive. It is not very big. Dashcams with GPS tracklogs do this also. Sometimes meta-data is more useful than the original data, you could search and sort archives based on motion parameters. Some post-process script might be possible to count cars and pedestrians separately, for example.
Yes, good thought about doing square root. Thanks again!

billw
Posts: 404
Joined: Tue Sep 18, 2012 8:23 pm

Re: PiKrellCam: motion vector detect + OSD web interface

Mon Nov 23, 2015 10:04 pm

An upgrade is available.

Changes:
  • 1) Motion Regions panel has new "List" button and improved user feedback.
  • 2) Bug fix: scripts archive-video and archive-still were chmod happy and
    were changing media files to +x. It's just a cosmetic issue, but still ugly.
    If you want to fix, cd to your archive_dir and run:

    Code: Select all

    find . -type f -exec chmod 0644 {} \;
  • 3) tomtgrp's media-archive.php rendering fix.
jbeale- set motion_stats to on in pikrellcam.conf and there will be a
.csv for motion videos. The name is the same as the video with .mp4 replaced
with .csv and the files are deleted or archived along with the video/thumb.

I put in two header lines. First line is each data line format and the
second header line is the motion vector array size for current video settings.
I'm leaving it at that because all the sizes can change with user config
changes and the x,y coordinates start out in motion vector array coords.

But this or any other format change you may want, just let me know and
I'll change it. At this point, .csv files look like:

Code: Select all

# time, x, y, dx, dy, magnitude, count
# width 121 height 68
 2.625,	 88,  20, -12,	 2,  12,  512
 2.792,	 89,  19,  -9,	 1,   9,  369
 3.292,	 76,  18,  21, -15,  26,  550
...
The first data line time is the first motion detect. It will typically
be between pre_capture and pre_capture + 1 time because all recordings start
on a key frame and I request those with 1 second resolution. But the time
can be less than pre_capture if the video started right after the previous
video stopped and pre_capture was not yet accumulated in the circular buffer.

User avatar
jbeale
Posts: 3518
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: PiKrellCam: motion vector detect + OSD web interface

Mon Nov 23, 2015 11:45 pm

billw wrote:An upgrade is available.
Outstanding! Everything I was hoping for, and even the csv columns line up nicely. Looking forward to some data analysis shortly.

Here's an example of that stats output for one video "event" this evening. You can see from the X and DX variables that there are several distinct moving objects. In this case it was 4 cars passing by at night (only headlights & taillights visible). One of them was turning around, so it appears twice going in different directions at start and end. This data is pretty clean; you can see that the velocity changes only slowly, so a simple algorithm should be able to recognize separate moving objects, and extract a still frame from the video for each one.
Image

raw data:

Code: Select all

time, x, y, dx, dy, magnitude, count
# width: 121 height: 68  filename: motion_2015-11-23_19.29.16_9.csv
 2.750,  47,  36, -64,   1,  64,  139
 2.917,  55,  35, -64,   1,  64,  193
 3.250,  79,  35, -66,   1,  66,   93
 3.583, 110,  35, -64,   0,  64,   45
 4.417,  31,  31, -40,   0,  40,   46
 4.750,  47,  35, -34,   0,  34,  122
 4.917,  55,  35, -33,   0,  33,   96
 5.083,  64,  34, -29,   0,  29,  126
 5.250,  71,  35, -29,   0,  29,  184
 5.417,  77,  36, -29,  -1,  29,  214
 5.583,  86,  35, -27,   0,  27,  204
 5.750,  88,  38, -25,   0,  25,  159
 5.917,  96,  39, -24,   0,  24,  110
 7.417,  45,  31, -56,   0,  56, 1482
 7.750,  70,  35, -67,   0,  67, 1063
13.583,  91,  28,  61,  -1,  61,   52
13.750,  71,  29,  62,  -2,  62,   68
13.917,  82,  29,  61,  -6,  61,  102
14.083,  66,  31,  58,   0,  58,  124
14.250,  80,  29,  56,  -1,  56,   41
14.417,  67,  31,  51,   0,  51,   68
14.583,  54,  31,  43,   0,  43,  111
14.750,  43,  31,  39,   0,  39,  131
15.083,  26,  19,  29,   0,  29,   45
17.250,  46,  38,  18,   0,  18,   80
17.417,  39,  35,  17,   0,  17,   49
17.583,  35,  34,  17,   0,  17,   50

jit
Posts: 33
Joined: Fri Apr 18, 2014 2:52 pm

Re: PiKrellCam: motion vector detect + OSD web interface

Tue Nov 24, 2015 9:07 am

I have a minor item of feedback. I was playing around with the UI on my phone and must have accidentally hit the motion button which disabled motion detection. It would be great to have an override flag that could be set to either prevent the display of the button, or simply ignore the button press. Will stop any 'fat finger' issues.

Jasimo
Posts: 51
Joined: Mon Apr 27, 2015 11:50 am

Re: PiKrellCam: motion vector detect + OSD web interface

Tue Nov 24, 2015 1:35 pm

Hi,

playing around with PiKrellcam for a while now and I must say a great piece of code.
One thing about the motion detection. Is there a way that I can see (after the recording is done) where the motion detection exactly was in the picture and which threshold of vector_magnitute and vector_count is reached.
This will help to fine trigger the parameters to avoid false positives in detection.

rgs
Jan

User avatar
jbeale
Posts: 3518
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: PiKrellCam: motion vector detect + OSD web interface

Tue Nov 24, 2015 2:55 pm

Jasimo wrote: Is there a way that I can see (after the recording is done) where the motion detection exactly was in the picture and which threshold of vector_magnitute and vector_count is reached.
This will help to fine trigger the parameters to avoid false positives in detection.
Yes! That is a new feature in the version released yesterday. After updating and stop/restart, edit the file ~/.pikrellcam/pikrellcam.conf and change the line "motion_stats off" to read "motion_stats on". Then stop/restart once more. Now after each recorded event there will be a .csv file next to each .mp4 file in the video directory with the data: time (seconds since video start), (x,y) position, (dx, dy) velocity, velocity magnitude, and motion area pixel count.

Here's a plot of the X and DX data from some cars moving past: viewtopic.php?f=43&t=115583&start=375#p848425

Jasimo
Posts: 51
Joined: Mon Apr 27, 2015 11:50 am

Re: PiKrellCam: motion vector detect + OSD web interface

Tue Nov 24, 2015 3:08 pm

Wow, cool new feature well done.
Is it planned to show the data of the csv file direcly in the recorded movie, something like an overlay? If i look in the csv, for me it is not possible to determine where in the picture the detection really was.

billw
Posts: 404
Joined: Tue Sep 18, 2012 8:23 pm

Re: PiKrellCam: motion vector detect + OSD web interface

Tue Nov 24, 2015 3:55 pm

jbeale wrote: Outstanding! Everything I was hoping for, and even the csv columns line up nicely. Looking forward to some data analysis shortly.

Here's an example of that stats output for one video "event" this evening.
Image
Nice charts! At some point can you outline your steps for generating them?

Return to “Camera board”