stephencassidy
Posts: 37
Joined: Mon Aug 19, 2013 11:12 am

Re: PiKrellCam: motion vector detect + OSD web interface

Mon Jan 09, 2017 7:44 am

I have also started to experience the same issues as a couple of others on here. On 3 of my 4 camera pi's the video is freezing and can be resumed by stopping and starting pikrellcam from the web interface. Sometimes everything runs for a while again but mostly it works for a few seconds and freezes again. All of my cameras had been working perfectly until a few weeks ago and I am aware of somebody else with the same problems since last week.

I have noticed that if left then pimrellcam will randomly spring into life for a while then stop.

I have monitored the log file where no errors are recorded and monitoring the pikrellcam process with htop show it consuming CPU and then dropping down the list as the video freezes. I captured it happening and posted a video on yotube.

https://youtube.com/watch?v=Lds1m9spfsc

I am at a loss now, I have tried the debugging that billw suggested but I was way out of my depth.

Ste

LucidEye
Posts: 84
Joined: Sun Aug 04, 2013 2:20 pm

Re: PiKrellCam: motion vector detect + OSD web interface

Mon Jan 09, 2017 7:08 pm

stephencassidy wrote:I have also started to experience the same issues as a couple of others on here. On 3 of my 4 camera pi's the video is freezing and can be resumed by stopping and starting pikrellcam from the web interface. Sometimes everything runs for a while again but mostly it works for a few seconds and freezes again. All of my cameras had been working perfectly until a few weeks ago and I am aware of somebody else with the same problems since last week.

I have noticed that if left then pimrellcam will randomly spring into life for a while then stop.

I have monitored the log file where no errors are recorded and monitoring the pikrellcam process with htop show it consuming CPU and then dropping down the list as the video freezes. I captured it happening and posted a video on yotube.

https://youtube.com/watch?v=Lds1m9spfsc

I am at a loss now, I have tried the debugging that billw suggested but I was way out of my depth.

Ste
I have noticed this problem as well on 2 different Pi camera setups I am running... but I determined it was caused by an inadequate power supply. After upgrading to 3amp power supplies and better quality USB cables I have not seen this problem recurring.

Fox Mulder
Posts: 42
Joined: Fri Sep 21, 2012 11:24 am

Re: PiKrellCam: motion vector detect + OSD web interface

Tue Jan 10, 2017 1:55 pm

The problem seems to appear faster. First the image was frozen after a few days but now it freezes just a few hours after the last restart. Restarting pikrellcam or rebooting the whole pi doesn't seem to help at all. Strange thing is that i use the same hardware since a few months and it worked without any problems. I have plenty of space left on the device and everything else works as expected.
Interesting is that pikrellcam works "normally" and reacts to every command, which i can see in the log and console. Just the Video stream seems to be not updating anymore. So i'm wondering if the problem doesn't come from pikrellcam but from some changes to the video core code maybe.

stephencassidy
Posts: 37
Joined: Mon Aug 19, 2013 11:12 am

Re: PiKrellCam: motion vector detect + OSD web interface

Tue Jan 10, 2017 2:54 pm

My Pi's have been running flawlessly too and have only just started to suffer this issue.

The only pi that is working as expected is a pi zero with a USB dongle soldered directly to the USB port and powered using an old Samsung mobile USB plug.

The Pi that is failing in my video clip is a Pi 2 powered using a PC ATX power supply connected to my router via Ethernet so I am not entirely convinced its network or power related.

I was wondering if it could be the SD cards starting to fail but the other pi that is failing is running from USB which I appreciate could also fail but its a bit of a coincidence that 3 out of 4 are all suffering exactly the same issue.

I too am suspecting changes elsewhere but I have no idea where.

Ste

Fox Mulder
Posts: 42
Joined: Fri Sep 21, 2012 11:24 am

Re: PiKrellCam: motion vector detect + OSD web interface

Tue Jan 10, 2017 4:02 pm

pikrellcam froze again after about only 30 minutes.
Now i did a complete gdb trace of all threads like mentioned.
The results are here: http://pastebin.com/KnL25Js9

Ciao,
Rainer

billw
Posts: 415
Joined: Tue Sep 18, 2012 8:23 pm

Re: PiKrellCam: motion vector detect + OSD web interface

Wed Jan 11, 2017 1:53 am

Fox Mulder wrote:pikrellcam froze again after about only 30 minutes.
Now i did a complete gdb trace of all threads like mentioned.
The results are here: http://pastebin.com/KnL25Js9
I suppose wading through those threads was tedious, but thanks for doing
it. The thing is, they all look normal and are waiting where they would
be if gdb is attached to a properly running pikrellcam. So now the thing
to know is if any threads are still able to run normally. At least it
seems the video preview callback thread is not behaving.

When you get a freeze, can you try some things:

1) See if the main pikrellcam thread will respond (read from the FIFO)
by making it do something that does not involve the camera.
Probably the simplest thing to do is see if pikrellcam will delete the log
file. Try clicking the Delete Log button on the web page and if that
doesn't work, try via the FIFO:

echo delete_log > /home/pi/pikrellcam/www/FIFO

And even though you won't see a change on the web page, click the web
page Enable Motion button and cat the /run/pikrellcam/state file.
See if the motion_enable line (first line) changes when you cat the
file after each click of the web page button.
Based on your comments above, these should work.

2) Try to exercise the camera still capture thread. If the above delete
log worked from the web page, take a still from the web page. If it does
not work, click the button again and check the log file for a jpeg encoder
busy line. If you get no still and no log message, try again to take a
still using the FIFO:

echo still > /home/pi/pikrellcam/www/FIFO

This exercises a separate camera thread so would be interesting to know if it works.

3) If pikrellcam has responded to any FIFO commands above, then just to be
sure, even though the preview is frozen, see if you can take a manual video.
Click the manual record and stop web page buttons. The video circular
buffer thread is different from the web page preview image update thread.

Based on your comments above I'm not sure where this will lead. The part about a reboot
not helping makes me think something about the system is not working right, but still knowing
if any threads will respond might be useful.

stephencassidy
Posts: 37
Joined: Mon Aug 19, 2013 11:12 am

Re: PiKrellCam: motion vector detect + OSD web interface

Wed Jan 11, 2017 9:01 am

Hi Billw,

Mine was frozen again this morning so I tried your suggestions.

1) I was able to clear the log file from the web interface
motion_enable line does indeed toggle in the pkrellcam/state file

2) The still capture appeared to work but created a file of zero bytes in size

-rw-r--r-- 1 pi pi 0 Jan 11 08:41 image_2017-01-11_08.41.36_0.jpg

echo still > /home/pi/pikrellcam/www/FIFO results in
08:44:09 : command_process: still
08:44:09 : still capture failed because jpeg encoder is busy.


3) Manual record results in the following in the log file

08:45:41 : command_process: record on
08:45:41 : Video record: /mount/nfs/ssd/camera1/videos/manual_2017-01-11_08.14.04_0.mp4.h264 ...
08:46:09 : command_process: record off
08:46:09 : Video manual record stopped. Header size: 28 h264 file size: 18160685
08:46:09 : execl:[rm /mount/nfs/ssd/camera1/videos/manual_2017-01-11_08.14.04_0.mp4.h264]


I hope that helps

Ste

billw
Posts: 415
Joined: Tue Sep 18, 2012 8:23 pm

Re: PiKrellCam: motion vector detect + OSD web interface

Thu Jan 12, 2017 2:06 am

stephencassidy wrote: 3) Manual record results in the following in the log file

08:45:41 : command_process: record on
08:45:41 : Video record: /mount/nfs/ssd/camera1/videos/manual_2017-01-11_08.14.04_0.mp4.h264 ...
08:46:09 : command_process: record off
08:46:09 : Video manual record stopped. Header size: 28 h264 file size: 18160685
08:46:09 : execl:[rm /mount/nfs/ssd/camera1/videos/manual_2017-01-11_08.14.04_0.mp4.h264]
Did this result in a good manual video corresponding to the time the manual record button was
pressed? Unfortunately from the log data I'm not sure if the video callback thread was actually
being called and I need to know if a good video resulted.

At least at this point I know the main pikrellcam thread is running, the preview path is not and the still
thread also has a problem. So, if the video thread is not running, something having to do with the
camera is freezing and pikrellcam is just sitting there waiting. But if it is running then maybe it's more
of a GPU processing overload locking out paths, hence the chance it can spring back to life after
a while.

stephencassidy
Posts: 37
Joined: Mon Aug 19, 2013 11:12 am

Re: PiKrellCam: motion vector detect + OSD web interface

Thu Jan 12, 2017 8:35 am

I can confirm that no manual video was recorded.

Ste

Fox Mulder
Posts: 42
Joined: Fri Sep 21, 2012 11:24 am

Re: PiKrellCam: motion vector detect + OSD web interface

Fri Jan 13, 2017 3:52 pm

Because pikrellcam seems to work normally and only the video stream is frozen i suspected some other changes in the system. So i tried downgrading the Raspberry Pi special packages in raspbian.
The interesting packages, which are related to the native camera interface are the following (installed) ones:
  • libraspberrypi0
    libraspberrypi-bin
    libraspberrypi-dev
    raspberrypi-bootloader
    raspberrypi-kernel
The latest installed version of all these packages is 20161215 (kernel 4.4.38). Now i downgraded to 20160523 (kernel 4.4.11) to use a quite older version were everything surely worked as expected. And where the video lastly freezes only a few seconds after restart of pikrellcam it now works for over 30 minutes without problems. So i tried a few versions and am now at 20161125 (kernel 4.4.34) which is the last version before 20161215. And it also works for a few hours now without problems. I will let it run for the next few days and hope it doesn't freeze again.

Now the problem is that all these packages not only changes the kernel but also all of the rpi's special libraries which makes finding the culprit not very easy (kernel or some other library). But all these packages can only be installed with the same version because of their dependencies.

I will report in the next days if the downgrade of these packages help fix the problem or not. But at the moment it runs for a longer time than in the last few weeks without freezing. :D

Ciao,
Rainer

Fox Mulder
Posts: 42
Joined: Fri Sep 21, 2012 11:24 am

Re: PiKrellCam: motion vector detect + OSD web interface

Sat Jan 14, 2017 9:22 am

Now pikrellcam is running for over 19 hours and works as expected without freezing. Seems the latest rpi updates broke or changed something with respect to the camera stream. If you want to try it yourself you can download the older libraries for raspbian from https://archive.raspberrypi.org/debian/ ... -firmware/.

Ciao,
Rainer

stephencassidy
Posts: 37
Joined: Mon Aug 19, 2013 11:12 am

Re: PiKrellCam: motion vector detect + OSD web interface

Sat Jan 14, 2017 9:30 am

Thanks for the update Rainer. I have just rolled back to 4.4.34 so hopefully it resolves my issue too.

Ste

andies
Posts: 144
Joined: Mon Nov 11, 2013 8:12 pm
Location: Berlin

Re: PiKrellCam: motion vector detect + OSD web interface

Sat Jan 14, 2017 2:39 pm

stephencassidy wrote:I have just rolled back to 4.4.34.
I am new to this rolling-back stuff. How am I doing this?
raspberry B, Noir camera, Mac Book Air, iPhone, Bezzera

stephencassidy
Posts: 37
Joined: Mon Aug 19, 2013 11:12 am

Re: PiKrellCam: motion vector detect + OSD web interface

Sat Jan 14, 2017 4:32 pm

I followed this page

https://tech.enekochan.com/en/2014/03/0 ... -raspbian/

the command I issued was

sudo rpi-update 6e2ab8e3f4dbf1f286a2c955631481f2a3bee49c

Fox Mulder
Posts: 42
Joined: Fri Sep 21, 2012 11:24 am

Re: PiKrellCam: motion vector detect + OSD web interface

Sat Jan 14, 2017 7:56 pm

Anothrer way to roll back only these libraries is to download the wanted versions form https://archive.raspberrypi.org/debian/ ... -firmware/ and install them with dpkg.

So if you want to downgrade to 20161125 download the following libraries:
libraspberrypi0_1.20161125-1_armhf.deb
libraspberrypi-bin_1.20161125-1_armhf.deb
libraspberrypi-dev_1.20161125-1_armhf.deb
raspberrypi-bootloader_1.20161125-1_armhf.deb
raspberrypi-kernel_1.20161125-1_armhf.deb

And install them:

Code: Select all

sudo dpkg -i libraspberrypi0_1.20161125-1_armhf.deb libraspberrypi-bin_1.20161125-1_armhf.deb libraspberrypi-dev_1.20161125-1_armhf.deb raspberrypi-bootloader_1.20161125-1_armhf.deb raspberrypi-kernel_1.20161125-1_armhf.deb
Ciao,
Rainer

User avatar
Ektus
Posts: 11
Joined: Mon Dec 26, 2016 12:02 pm

Re: PiKrellCam: motion vector detect + OSD web interface

Mon Jan 23, 2017 5:25 pm

Is there a way to set the OSD time stamp to 24-hour format? Locale is set to de_DE, but other than removing the AM/PM info this didn't help.

Regards
Ektus.

billw
Posts: 415
Joined: Tue Sep 18, 2012 8:23 pm

Re: PiKrellCam: motion vector detect + OSD web interface

Mon Jan 23, 2017 7:06 pm

Ektus wrote:Is there a way to set the OSD time stamp to 24-hour format? Locale is set to de_DE, but
There's the annotate_format_string variable in pikrellcam.conf that is processed with strftime. So do
a "man strftime" to see what you can put there. If you edit pikrellcam.conf, stop and restart pikrellcam
or the quick way without restarting is to change the variable using the FIFO. To change to 24 hour
format, change the default %l to %k and delete the %p:

Code: Select all

echo annotate_format_string "%a %b %e, %k:%M.%S" > ~/pikrellcam/www/FIFO

User avatar
Ektus
Posts: 11
Joined: Mon Dec 26, 2016 12:02 pm

Re: PiKrellCam: motion vector detect + OSD web interface

Mon Jan 23, 2017 7:13 pm

billw wrote:
Ektus wrote:Is there a way to set the OSD time stamp to 24-hour format? Locale is set to de_DE, but

Code: Select all

echo annotate_format_string "%a %b %e, %k:%M.%S" > ~/pikrellcam/wwwFIFO
That did the trick, thank you very much!


Regards
Ektus.

User avatar
Ektus
Posts: 11
Joined: Mon Dec 26, 2016 12:02 pm

Re: reliable storage for stills and video

Mon Jan 23, 2017 7:35 pm

Ektus wrote:
Ektus wrote:Another one is recording on the 64GB MicroSD-card, but has been in operation only for a couple of month.
That one has failed sometime around end of December / beginning of January. Coming home after the Christmas break, it's showing EXT4-FS errors with corrupt filesystem and Read-error on swap device. The RPi3 has a monitor connected, so I was able to at least see those errors. There are timestamps, but not human-readable :-(

I'll have to investigate further, if the card is dead or if there is just some file system corruption. It's been running transmission additionally and has been uploading 18GB in December. Oh well, guess I'll have to add a HDD there, too...
The SD-Card (64GB SanDisk Ultra class 10) is quite dead. Read error in superblock, partitioning info seems to be present but I wasn't able to read any useful data.
I've since replaced it with a 1TB 2,5" USB hard drive. The RPi 3 is able to boot from USB, and with the timeout set to 5s the hard drive is fast enough. I don't have a SD-card in that system now, so let's see if the hard drive fares better :-)
It seems to be quite fast, too.
Attachments
bonnie_tosh1GB.7z
(928 Bytes) Downloaded 64 times

User avatar
Ektus
Posts: 11
Joined: Mon Dec 26, 2016 12:02 pm

Re: PiKrellCam: motion vector detect + OSD web interface

Tue Jan 24, 2017 5:02 pm

Fox Mulder wrote:Anothrer way to roll back only these libraries is to download the wanted versions form https://archive.raspberrypi.org/debian/ ... -firmware/ and install them with dpkg.

So if you want to downgrade to 20161125 download the following libraries:
libraspberrypi0_1.20161125-1_armhf.deb
libraspberrypi-bin_1.20161125-1_armhf.deb
libraspberrypi-dev_1.20161125-1_armhf.deb
raspberrypi-bootloader_1.20161125-1_armhf.deb
raspberrypi-kernel_1.20161125-1_armhf.deb

And install them:

Code: Select all

sudo dpkg -i libraspberrypi0_1.20161125-1_armhf.deb libraspberrypi-bin_1.20161125-1_armhf.deb libraspberrypi-dev_1.20161125-1_armhf.deb raspberrypi-bootloader_1.20161125-1_armhf.deb raspberrypi-kernel_1.20161125-1_armhf.deb
My newly installed RPi3 has been suffering from the same problem. The camera image did stop updating around 7:30 this morning. It had been running all night, but there was very few image data due to low lighting. At the time of failure, it was beginning to see daylight. Restarting in the evening hours once back from work, it did freeze almost instantly. So I've done the above downgrade now and will report back if it's stable.

[edit]
the list above is not complete. I had to install the corresponding libraspberrypi-doc, too, to replace the existing newer version. Otherwise, installing other packages would fail.
[/edit]


Regards
Ektus.

User avatar
Ektus
Posts: 11
Joined: Mon Dec 26, 2016 12:02 pm

Re: PiKrellCam: motion vector detect + OSD web interface

Wed Jan 25, 2017 4:11 pm

Ektus wrote: My newly installed RPi3 has been suffering from the same problem. The camera image did stop updating around 7:30 this morning. It had been running all night, but there was very few image data due to low lighting. At the time of failure, it was beginning to see daylight. Restarting in the evening hours once back from work, it did freeze almost instantly. So I've done the above downgrade now and will report back if it's stable.
The downgrade didn't help. It failed again at dusk this morning. Sunrise 07:55, last image 07:44:29. Pikrellcam shows no errors in the logfile and thinks it's still running, but the picture is frozen. "Stop" followed by "Start" in the system tab restores operation.

[edit]
But not for long. It freezes again after a couple minutes, as long as it's daylight.
[/edit]


Regards
Ektus.

Fox Mulder
Posts: 42
Joined: Fri Sep 21, 2012 11:24 am

Re: PiKrellCam: motion vector detect + OSD web interface

Wed Jan 25, 2017 7:28 pm

I have a little question to the author of pikrellcam. What formula do you use to calculate sunrise, sunset, dusk, dawn times which i can use in the at-commands and what location do you use for the calculation?

Because i wrote a little python script and calculated these times with help of the package "astral" and all my values are between 10-30 minutes before your calculated times compared to /var/run/pikrellcam/state. For this i used as location Germany/Berlin and i wonder what location you use because i can't set it anywhere. :?:
Ektus wrote: The downgrade didn't help. It failed again at dusk this morning. Sunrise 07:55, last image 07:44:29. Pikrellcam shows no errors in the logfile and thinks it's still running, but the picture is frozen. "Stop" followed by "Start" in the system tab restores operation.
Strange. My Raspberry is running since the downgrade 24/7 for 13 days now without any more freezes. Maybe the RPi3 has another dependency which has to be downgraded. I use my old RPi1 as webcam and don't have a RPi3 so i can't test it. :?

Maybe try the other mentioned method with the following command:

Code: Select all

sudo rpi-update 6e2ab8e3f4dbf1f286a2c955631481f2a3bee49c
Ciao,
Rainer

billw
Posts: 415
Joined: Tue Sep 18, 2012 8:23 pm

Re: PiKrellCam: motion vector detect + OSD web interface

Wed Jan 25, 2017 8:26 pm

Fox Mulder wrote:I have a little question to the author of pikrellcam. What formula do you use to calculate sunrise, sunset, dusk, dawn times which i can use in the at-commands and what location do you use for the calculation?

Because i wrote a little python script and calculated these times with help of the package "astral" and all my values are between 10-30 minutes before your calculated times compared to /var/run/pikrellcam/state. For this i used as location Germany/Berlin and i wonder what location you use because i can't set it anywhere. :?:
See sunriset.c in the src directory for the functions. It's a public domain file that's been around
since 1992. It's accurate for my location and was good for some other locations I checked.
You have to edit latitude and longitude in pikrellcam.conf for your location and restart pikrellcam.
The default is for a longitude close to me (but of course intentionally not exact) in the US so has
a "W" for west. For Berlin try a "13.3E" east longitude or just 13.3. For west longitudes you can
enter a negative number instead of appending a "W".

Sassel
Posts: 4
Joined: Sun Jan 29, 2017 8:39 am

Re: PiKrellCam: motion vector detect + OSD web interface

Sun Jan 29, 2017 9:20 am

Hello.

At first:
-sorry for my bad english ;) and
-@Bill Thank you very much for this nice peace of software! incredible 8-)

I´m using 3 PI´s (2x RPI-3B, 1x RPI A+), each one with a 2,5A Powersupp, HeatSink and the v1.3 Cam.
Rpi-Firmware uptodate

Unfortunately the pikrellcam´s running unstable (no entry´s in the logs, why) but i can see a pattern.
I can start the Pi´s in the evening (6-8pm) without issues. Then they run through in the morning and record videos.
Often they crash at ~8am. If i restart them with STOP/START, they will running again but stopping after a few minutes again and again. Sometimes they will running longer but max a few hours.
I'm not a developer but maybe, the problem lies at the sunrise/sunset-control...??!

Last day i'll start a Rpi3 at 6pm and he stopps at 8:11am
In the logfile he noticed:

--New Day--
dawn 7:30am
sunrise 8:08am
sunset 4:59pm
dusk 5:37pm

These problems have also posted others here but there was never a feedback / was ignored / or I have not read it :?

Greetings,
Sassel

stephencassidy
Posts: 37
Joined: Mon Aug 19, 2013 11:12 am

Re: PiKrellCam: motion vector detect + OSD web interface

Sun Jan 29, 2017 2:21 pm

Hi Sassel,

Have a read back a few posts. I was having the same issues and for me a kernel rollback fixed the issue.

Ste

Return to “Camera board”