PiKrellCam: motion vector detect + OSD web interface

[Gavinmc42]

billw,

Just had idea, fourth servo for daylight/IR filter.
Already have sunrise/sunset so can add tiny servo filter wheel.
Hmm, if don't need tilt, pan and zoom, one servo output should work?
Latching rotary solenoid?
Modify servo to only two positions?

[billw]

And learning billw's code so I can add a max limit.

This issue is how to apply the max limit (what are the rules) - if already well into a video with
lower than max detects and then a max limit is exceeded, should the video be discarded?.

If it is as simple as that then what if pikrellcam records the max vector and then there are variables
for it to be passed to an on_motion_end script. You could just test for limit exceeded and where in
the frame and if you don't like it just delete the video right then. This would not require any changes
to the detect algorithm or analysis of the csv file. You just look at one vector?

[billw]

Is there any way of getting a faster frame rate on the preview? Even at the lowest res it only seems to be around 2 or 3 frames/sec if that. I'm only going to be using it on my LAN or in ad-hoc mode with a tablet, so I don't need to reduce data for WAN viewing.

Does the Pi's video memory allocation affect performance at all? I currently have 64mb allocated.
Do I need the MPEG2 and VC1 licenses installed to make this software function at peak performance?

The preview is updated at video_fps/mjpeg_divider rate and the web page reads that at 150msec
timeout interval. If you get only 2-3 frames/sec could you be hitting bandwidth limits? What is the
size of /run/pikrellcam/jmjpeg.jpg file? If bandwidth is a problem, then get that lower by decreasing
mjpeg_quality. I have settings like mjpeg_width 800 and mjpeg_quality 7 and get about a 33k
/run/pikrellcam/mjpeg.jpg file size.

I have gpu_mem=128 and haven't tested at 64mb so don't know if that affects things. If it does
and the GPU is slowed down then there can be frames dropped which would limit the update
frequency of mjpg.jpg. I have disabled code in pikrellcam to report mjpeg frame rates and should
re-enable that for testing. One thing to check is look at video_last_fps in /run/pikrellcam/state file
and if it goes less than your configured value then it's a sign the GPU is maxed out.

EDIT: I just enabled mjpeg fps debugging. Upgrade and then quit pikrellcam so you can run pikrellcam from a terminal:

Code: Select all

pikrellcam  -debg-fps

Then watch the output as you change Config->Settings->mjpeg_divider. You should see
the programmed rate or programmed rate - 1 as you change the divider. If the rate falls below
that it means frames are being dropped. If you get a good rate and web preview still is updating
slow, then it must be a network bandwidth problem.

[jbeale]

what if pikrellcam records the max vector and then there are variables
for it to be passed to an on_motion_end script. You could just test for limit exceeded and where in
the frame and if you don't like it just delete the video right then. This would not require any changes
to the detect algorithm or analysis of the csv file. You just look at one vector?

I think that setup might work to reject high-speed street traffic, as long as your vector-count is high enough (or your field of view clean enough) so that for example a shaking tree branch or speedy bird or falling raindrops or other foreground object doesn't reject an otherwise valid frame. I think the general problem is when you have a mix of the slow (intended target) items and fast (other stuff) item(s) within the same event.

[billw]

what if pikrellcam records the max vector and then there are variables
for it to be passed to an on_motion_end script. You could just test for limit exceeded and where in
the frame and if you don't like it just delete the video right then. This would not require any changes
to the detect algorithm or analysis of the csv file. You just look at one vector?

I think that setup might work to reject high-speed street traffic, as long as your vector-count is high enough (or your field of view clean enough) so that for example a shaking tree branch or speedy bird or falling raindrops or other foreground object doesn't reject an otherwise valid frame. I think the general problem is when you have a mix of the slow (intended target) items and fast (other stuff) item(s) within the same event.

I agree, which is what makes this difficult to have as a builtin pikrellcam function since there's several
factors which might need to be configured. I could have a set of stats collected: max motion detect
vector, average motion detect vector, number of motion detects, and maybe other yet to be imagined
stats. It can end up being a lot of variables passed to the motion_end_script, but at least the
decisions I don't know how to generalize inside of pikrellcam could be made in the script by the user.

[Gavinmc42]

Large fast moving objects that go all the way across the frame.
These are the cars I don't want.
But I could see there could be a case where that is all someone wants.
Probably a pattern matching algorithm?

Not sure how well that would work, has to be simple and fast.
Would be useful if matches get sorted into different folders to start with.
Yes/no/maybe? big/fast, big/slow, small/fast, small/slow, further sorting.
Another pattern? if motion stops before leaving the frame.
Ideally, for me it gets sorted to birds, dogs, people, cars passing, cars stopping, stalkers;)

snails - tiny/slow
birds - tiny/fast
dogs, people - medium/slow- walkers, medium fast - joggers, bike riders?
cars passing - large/fast
cars stopping - large/slow
stalkers -????

Then once tuned the option to save or delete.
Hmm, looks like window comparators.
Already have area motion windows.
Got Setup, Motion Regions, System, have a fourth option Pattern Match

[jbeale]

Large fast moving objects that go all the way across the frame. These are the cars I don't want.

It would be very powerful to be able to detect based on "distance travelled by object" also to reject branches just moving back and forth in the wind, but I don't think PKC can actually do specific object detection. However, if your cars cross all four detection areas, maybe it's good enough to have speed stats for each area, so you can reject if a large-enough area of max speed is reached in every region. Is that possible?

[Gavinmc42]

Would be useful to put those quad cores to work.
We have a minimum threshold setting now, add a maximum setting turns it into a window comparator.
Add a few more compares and it becomes a sorter.
Talking robotic vision stuff now. fruit sorting etc if color gets added, whoops, off track.

[billw]

Talking robotic vision stuff now. fruit sorting etc if color gets added, whoops, off track.

Yeah, separate project. Someday maybe add a YUV analyzer of the I420 frame that runs alongside the motion vector analyzer. It could be hinted by the motion vector position and bounding box but would still need some CPU time.

It would be very powerful to be able to detect based on "distance travelled by object" also to reject branches just moving back and forth in the wind, but I don't think PKC can actually do specific object detection. However, if your cars cross all four detection areas, maybe it's good enough to have speed stats for each area, so you can reject if a large-enough area of max speed is reached in every region. Is that possible?

PKC knows objects by vector magnitude and count so it only knows there is an object and can't differentiate between objects.

This kind of detection still seems to me a post processing reject or accept of the video. If you want to detect on distance traveled, what if an object moves in slowly, lingers, and then moves the distance detect limit - how to handle the pre-capture?

There could be a motion-state file (/run/pikrellcam/motion-state) that updates with as much data as we want at each motion detect event It could have current vector/count, max vector/count, average vector/count, speed averages, total vector distance traveled (swaying branches cumulative vector distances would be small), etc.

The on_motion_begin command could run a python script that opens the motion-state file and loop/sleeps reading it. There would be a motion sequence number so it would know when a new motion event had occurred and the number could flag the motion end so the script would know when to clean up.

So the script cumulatively watches the progress of the event accumulating data. At the end it can decide to delete or keep the video. This idea is also along the lines of what I've been thinking about regarding motion tracking. The python script can make position decisions during the event and write to the FIFO to control PKC servos for motion tracking. And it could notify other camera installations so they can decide to start recording or move servos.

So instead of hardwiring or implementing an interpreted control language inside PKC, there can be a library of contributed python scripts that would work as is or be starting points for anybody with a special need for motion detection or motion servo tracking.

[Gavinmc42]

The motion preset regions are really good for ignoring tree branches, best thing about Pikrellcam.
As long as no trees are in the region, I don't notice many false triggers.
I do have one bush that needs trimming, getting a haircut this weekend:)

Fast cars zoom though then post capture minimum is 1sec, so I get the next car as well.
Around school run time I can get 5 cars in one video, that's not a 2 second gap people.
I don't worry about precapture, if it was zero it would not bother me.
Pre/post times of zero would make smaller files and one file per car?
Post capture of zero might be better.

Interesting, this morning, going though the videos, file size is a good indicator.
Had one double dump truck that went back and forwards about five times, could nearly always tell because the file size was the largest.
Long slow movement, wife and dogs in the front yard also make large files.
Have not looks at the vector files yet, job for the weekend, possible sorting later?

Might recompile for zero pre/post times and test, that is easy enough to do.
Any fast vectors that go from side to side will be cars so filed straight to fast car folder, 99% sorted:)
If it has some slow vectors as well, filed in "maybe" folder.
Really need pre/post time to be related to vector magnitude, zero for fast, 5-10 secs for slow?
File size then becomes an easy sort, fast cars = short video = small files.

Slow motion tracking would be the interesting stuff, two camera system, normal and slomo servo tracker.
Because it is slow could be post processed from motion vector file?

billw, fruit sorting another project.
More important is the Lego picking up robot
QPUlib might be useful here for YUV processing.
Do not need 8Mp for Lego's

[Gavinmc42]

Another update?
debug fps.

Tried to update from 3.0.4 to 3.0.5 on my V2 test system, did not work.
Could be the SD card so swapped that and put a V1 camera in.
Installing pikrellcam now, don't expect any problems.

Saying that it is one of my oldest Pi's.
Someone asked about reliability of the Pi's on another post.
MTBF of electro caps? actually lifetime of electro cap C6 - 5000hrs?
My old Pi's are way past that, could be why I am having issues recently.

Anyone using old Pi's with the electro caps for security should be aware of this.
Newer models with the sw power supply use tant caps.
Have we pasted the use by date on the old models?
Retiring my old Pi's for security use might not have been a good idea.

[jbeale]

MTBF of electro caps? actually lifetime of electro cap C6 - 5000hrs?

Highly unlikely. That is less than one year! I have two Pi's with uptime over 750 days, that is over 18000 hours and no issues yet.

[Gavinmc42]

jbeale,
Are they model B's or B+?
Silver capacitor near the USB power?
Only one of my old ones is flaky.
Probably also depends on my power supply and cable.

Going to need IR illumination flood lights, just had a suspected stalker car pull up outside just after dark.
Turned lights off, back on again, off again then on and moved off when the wife when outside to look.
Been ridiculous the last month, the old PS eyecams are better at night but lack detail.
Made a lot of complaints and was told we are frivolous and vexatious and unless we have license plate numbers we have no proof. They never stay long enough for the cops to get here.

[Harrie1966]

I have a question I would like to use the Function timelapse but only when there is motion detected. Is this possible and I can customize this turn or a script. I'm very satisfied with this program works very well. only timelapse takes up too much space on an SD card. I hope you can help me.

[billw]

I have a question I would like to use the Function timelapse but only when there is motion detected. Is this possible and I can customize this turn or a script. I'm very satisfied with this program works very well. only timelapse takes up too much space on an SD card. I hope you can help me.

I assume you want a single timelapse to capture over some time like a day? If so you can start and
end the timelapse with at commands each day, and use the on_motion commands to put the timelapse
in and out of hold state. For example, in ~/.pikrellcam/at-commands.conf, start a 1 second timelapse
for the day at dawn with it starting in the hold state, and at sunset end the timelapse:

Code: Select all

daily dawn "@tl_start 1"
daily dawn "@tl_hold on"
daily sunset "@tl_end"

Then in ~/.pikrellcam/pikrellcam.conf set the on motion commands:

Code: Select all

on_motion_begin sh -c "echo tl_hold off" > /home/pi/pikrellcam/www/FIFO
on_motion_end sh -c "echo tl_hold on" > /home/pi/pikrellcam/www/FIFO

But note that the motion videos that get recorded may have skips when the camera changes modes
to take the stills for the timelapse. If you don't want the videos, then the on_motion_end command
should be a script that holds the timelapse and deletes the video.

[LucidEye]

I've been having fun tinkering with this software.

I know this is off on a tangent, but photography is one of my main hobbies and I've been looking for more ways to shoot time lapse movies. I really want to setup a small, self-contained, portable pan/tilt (motion control) time lapse rig with the Pi, but I can't seem to get stepper motors to work on my Pi for some reason... So, out of curiosity, is there any way to make the pan and tilt servos step very slowly from preset to preset while shooting a time lapse... by slowly I mean panning, say, from the far left pan limit to the far right pan limit over a period of several minutes to several hours? I'm assuming after setting the max limits on the pan/tilt servos, the software then "knows" how many pulses it takes to go from one limit to the opposite limit... so I'm guessing there would be some way to tell it to send pulses spaced at a specified time interval in order to do a super-slow pan... and I'm guessing I'd need to use the minimum width pulse in order to make the pan appear as smooth as possible... too fast a pan or tilt movement in the servos would show up as jerkiness in the finished time lapse. Like I said... of on a tangent... that's what happens when I stay up geeking out till 4am

[billw]

So, out of curiosity, is there any way to make the pan and tilt servos step very slowly from preset to preset while shooting a time lapse... by slowly I mean panning, say, from the far left pan limit to the far right pan limit over a period of several minutes to several hours? I'm assuming after setting the max limits on the pan/tilt servos, the software then "knows" how many pulses it takes to go from one limit to the opposite limit... so I'm guessing there would be some way to tell it to send pulses spaced at a specified time interval in order to do a super-slow pan... and I'm guessing I'd need to use the minimum width pulse in order to make the pan appear as smooth as possible... too fast a pan or tilt movement in the servos would show up as jerkiness in the finished time lapse. Like I said... of on a tangent... that's what happens when I stay up geeking out till 4am

There's some ideas in there for commands I could add to do this better, but as a demo of what you
could do now, here's a script that does a 100 frame panning timelapse of period 2 sec:

Code: Select all

#!/bin/bash

SCAN=2
STEP=0

# pan left and give time for servo to reach left limit
echo servo pan_left $SCAN > ~/pikrellcam/www/FIFO
sleep 10

echo tl_start 2 > ~/pikrellcam/www/FIFO
sleep 1
COUNTER=0
while [	 $COUNTER -lt 100 ]
    do
    echo servo pan_right $STEP > ~/pikrellcam/www/FIFO
    let COUNTER=COUNTER+1
    sleep 2
    done
echo tl_end > ~/pikrellcam/www/FIFO

It could use better script control of when the timelapse shot is taken and maybe more than one
shot per step because the minimum servo step is 10usec so the steps in the video are apparent.
But it does work. Until I add a way for a script to get the pan limits, you would have to hardwire this
script to know what your limits are. Change the sleep time to get a pan over minutes or hours.

[LucidEye]

So, after getting everything setup and testing for a couple of days... it seems that pikrellcam is causing some serious bandwidth issues on my network.

All my connections are WiFi, currently I can not connect via ethernet cable because of the location of the router, and I can not move the router because it is acting as a WiFi bridge to a main gateway router. I know that causes a traffic bottleneck, but still... I had been using RPi_Cam_Web_Interface for a long time, and I was always able to get smooth preview video at 15-20fps even in full-sensor resolution... and with no noticeable bandwidth drag-down on any of the other machines on my network... and that was running on a Pi 2.

With pikrellcam running on a Pi3, even if I don't have the web interface open, my network is running so slow I can't even do simple web browsing.... and I know it's the Pi because the second I unplug it, my network runs at full speed again. No matter how low I set the resolution, bitrate, mjpeg_divider, etc... I can still only manage about 3fps on the preview... and trying to watch any recorded videos from the Pi is impossible as they stop and start every 2 seconds while it chokes on trying to buffer the video. I tried downloading one 3 minute video I took... file size was about 45MB... it took 9 minutes to download to a machine on the same network segment! I checked the WiFi signal at the router and I was getting 54Mbps @ 58% signal strength... the Pi is only 20 feet from the router with clear line of sight... With a full-speed high signal connection, I should be getting much better preview streaming and file download rates than this.

Any ideas why it's such a bandwidth hog? It seems that it is sending out a lot of network traffic even when it doesn't need to be.

[jbeale]

You are right that shouldn't be happening. Anything else running on the Pi3?

There is a simple text mode tool called "speedometer" that can show you current bandwidth use. You can SSH into your Pi3 and run it to see what the current TX and Rx rate is.

apt-get install speedometer
https://excess.org/speedometer/

[billw]

You can also graphically monitor with gkrellm the wlan0 traffic, cpu usage, etc of wifi connected
headless Pi boxes. ssh in and:

Code: Select all

sudo apt-get install gkrellmd

Then on your workstation (if the headless is say rpiH):

Code: Select all

sudo apt-get install gkrellm
gkrellm -s rpiH
   #or if rpiH is not in /etc/hosts, use its IP:
gkrellm -s IP-number

If you have multiple Pi machines, gkrellmd on each can be configured to use a different port and
all can be displayed at once on the workstation with multiple: gkrellm -s rpiX -P portnumber

Edit: after installing gkrellmd on the Pi, you need to edit /etc/gkrellmd.conf or create a ~/.gkrellmd.conf
or a /usr/local/etc/gkrellmd.conf to enable remote host connections to gkrellmd with allow-host
lines. Do a: man gkrellmd

[Gavinmc42]

Anyone tried pikrellcam on a Zero?
Got two Zero's on the way with cables.

I think there is away to do networking over serial?
Not sure max baud rate on the Zero, 10Mbaud?
Cannot connect to 3B because of Uart change?

Pi B, B+, 2B with serial to a Zero mean I only have to network the B.
Can have the B talking to 1-6 Zero's.
3 cameras is 180 degree, 6 cameras 360 degree.
Using a 2B with 4 cores for extra motion vector analysis?

180 degree system, 3 cameras, 2 Zero's and a 2B, extra motion vectors stuff in three cores, forth core saving to USB PiDrive?
How hard is it to do multicore coding in C?

[LucidEye]

You are right that shouldn't be happening. Anything else running on the Pi3?

There is a simple text mode tool called "speedometer" that can show you current bandwidth use. You can SSH into your Pi3 and run it to see what the current TX and Rx rate is.

apt-get install speedometer
https://excess.org/speedometer/

As far as I know, nothing else is running on the pi. I'm running Jessie Lite to minimize any extraneous CPU usage by the OS. It is currently doing auto-login to the console at boot with the default user "pi"... but that really shouldn't make any difference.... does it need to login at boot, or does the pikrellcam service start regardless of login?

I'll try the "speeodometer", but even my SSH connections are running with severe lag after the pi3 has booted.

As soon as I can afford to, I am planning on taking my router off of the WiFi bridge and connecting it to the gateway router with a couple of gigabit powerline ethernet adapters... that will automatically cut my radio traffic in half and double the WiFi throughput. Eventually I want to run the pikrellcam cameras via PoE anyway, as archiving daily videos to my NAS seems to be too much for the little Pi to do via WiFi, so I have to relocate my router regardless.

[LucidEye]

Well that's handy

I'll give it a go when time permits and try see what's going on in there.

So, as I understand, all of pikrellcam's processes are running on only one of the 4 CPU cores?
Seems a shame to let all that computing power go to waste
Is there any way to divide the work to other cores... like all video processes on one or 2 cores and GPU, all read/writes to the SD card on another core, all web-interface and network I/O on the remaining core? Divide and conquer? I've also been experiencing crashes whenever the Pi tries to compile a timelapse video... even a short one... the progress meter shows up, and a minute or 2 later the web interface becomes unresponsive, and even the SSH connection gets dropped... have to physically unplug the pi to reboot it I've even tried switching SD cards and I'm using a class 1 32gb card to keep the read/write bottleneck to a minimum and it doesn't seem to be helping.

You can also graphically monitor with gkrellm the wlan0 traffic, cpu usage, etc of wifi connected
headless Pi boxes. ssh in and:

Code: Select all

sudo apt-get install gkrellmd

Then on your workstation (if the headless is say rpiH):

Code: Select all

sudo apt-get install gkrellm
gkrellm -s rpiH
   #or if rpiH is not in /etc/hosts, use its IP:
gkrellm -s IP-number

If you have multiple Pi machines, gkrellmd on each can be configured to use a different port and
all can be displayed at once on the workstation with multiple: gkrellm -s rpiX -P portnumber

[jbeale]

I've also been experiencing crashes whenever the Pi tries to compile a timelapse video... even a short one... the progress meter shows up, and a minute or 2 later the web interface becomes unresponsive, and even the SSH connection gets dropped... have to physically unplug the pi to reboot it

This should not happen, any chance there is a power supply issue? Using a 1.5 A or more power supply with good quality cord? I have experienced wifi slowdown and outage which was related to my power supply, so it is possible. Is this a Pi3 using built-in wifi? I have read that there are issues with the driver for the builtin Pi3 wifi

[billw]

So, as I understand, all of pikrellcam's processes are running on only one of the 4 CPU cores?
Seems a shame to let all that computing power go to waste
Is there any way to divide the work to other cores... like all video processes on one or 2 cores and GPU, all read/writes to the SD card on another core, all web-interface and network I/O on the remaining core? Divide and conquer?

I think all the cores are used nicely. A camera app starts off inherently multithreaded since the
callbacks to process the video data buffers are all threads set up by the MMAL library layer.
The PKC process then can run on one core while the GPU callback threads run on other cores with
all cross thread data accessing coordinated with mutex locks. If you have servos, then there's
another thread PKC starts to handle servo moves. Don't worry about Pi2's or Pi3's, it's only
the B+ that gets a single core workout.