Page 1 of 1

Memory leak? System "hangs" after some time. (webcamserver)

Posted: Mon Mar 04, 2019 11:11 am
by pertm84
Hi

I am quite new to this, but I have managed to set up a raspberry pi using stretch as a webserver for uploading stills to my wunderground account through ftp. UFW, fail2ban, apache etc is all set up and working.

Also, I set the camera to livestream whenever I want. Everything is working, but sometimes the system hangs, and I can't SSH into it. Therefore the only solution is to cut the power and perform a reboot, which makes everything working again.

This is output from top command:

Code: Select all

top - 11:57:08 up 7 days, 22:17,  2 users,  load average: 0.43, 0.37, 0.37
Tasks: 148 total,   1 running,  99 sleeping,   0 stopped,   5 zombie
%Cpu(s): 11.1 us,  1.0 sy,  0.0 ni, 87.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :   896672 total,   153296 free,   276868 used,   466508 buff/cache
KiB Swap:   102396 total,   102396 free,        0 used.   514568 avail Mem 
I created a script and added to cron for Motion to restart every 30 min (since it is unstable) and also a script for uploading tmp.jpg to my wunderground account every 15min. So Motion is writing jpeg to ram every 15 min, but will this be deleted/overwritten each time? Is this filling up my ram and swap?

I also suspect that the log-files may slow things down, as UFW fills up my log-files with unnecessary lines. I have no idea where to start my fail-search, since I have little experience with linux.

Re: Memory leak? System "hangs" after some time. (webcamserver)

Posted: Mon Mar 04, 2019 11:15 am
by jamesh
If you are definitley writing the same file each tim,e then its not using up anything. Same with log files, the size of them should not affect speed.

It might help to get full `top` dumps every 15 minutes or so leading up to the lockup to see if anything is using increasing amounts of memory.

I pressume there is nothing in dmesg giving hints as to why things are going wrong?

Re: Memory leak? System "hangs" after some time. (webcamserver)

Posted: Mon Mar 04, 2019 11:37 am
by pertm84
Thanks for the quick answer! I'm not quite sure what to look for in dmesg, and everytime I had to do a hard-reset, it looks like the log-files has restarted.What should I look for in dmesg?

Also when I tried to type dmesg, I get:
-bash: dmesg: command not found

I know it must be installed, because when I: sudo apt-get install util-linux

Reading package lists... Done
Building dependency tree
Reading state information... Done
util-linux is already the newest version (2.29.2-1+deb9u1).
0 upgraded, 0 newly installed, 0 to remove and 121 not upgraded.

Also I tried: sudo apt-get install util-linux --reinstall

Then I got:

Code: Select all

sudo apt-get install util-linux --reinstall
Reading package lists... Done
Building dependency tree       
Reading state information... Done
0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 121 not upgraded.
Need to get 895 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://raspbian.trivini.no/raspbian stretch/main armhf util-linux armhf 2.29.2-1+deb9u1 [895 kB]
Fetched 895 kB in 2s (307 kB/s)     
apt-listchanges: Can't set locale; make sure $LC_* and $LANG are correct!
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
	LANGUAGE = (unset),
	LC_ALL = (unset),
	LC_CTYPE = "UTF-8",
	LANG = (unset)
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
dpkg: warning: 'tar' not found in PATH or not executable
dpkg: error: 1 expected program not found in PATH or not executable
Note: root's PATH should usually contain /usr/local/sbin, /usr/sbin and /sbin
E: Sub-process /usr/bin/dpkg returned an error code (2)
So I thought I should try to change locale, but it was already setup in raspi-config?

Code: Select all

sudo raspi-config
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
	LANGUAGE = (unset),
	LC_ALL = (unset),
	LC_CTYPE = "UTF-8",
	LANG = (unset)
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
/usr/bin/locale: Cannot set LC_CTYPE to default locale: No such file or directory
/usr/bin/locale: Cannot set LC_ALL to default locale: No such file or directory
Generating locales (this might take a while)...
  en_US.UTF-8...character map file `UTF-8' not found: No such file or directory
default character map file `ANSI_X3.4-1968' not found: No such file or directory
 done
Generation complete.
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
	LANGUAGE = (unset),
	LC_ALL = (unset),
	LC_CTYPE = "UTF-8",
	LANG = (unset)
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
	LANGUAGE = (unset),
	LC_ALL = (unset),
	LC_CTYPE = "UTF-8",
	LANG = (unset)
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
*** update-locale: Error: invalid locale settings:  LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LANGUAGE=en_US.UTF-8

Re: Memory leak? System "hangs" after some time. (webcamserver)

Posted: Mon Mar 04, 2019 12:17 pm
by pertm84
OK, it seems there is a problem with my locales. I am not able to install the US utf8.

I get:

Code: Select all

sudo dpkg-reconfigure locales                             
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
	LANGUAGE = "en_US.UTF-8",
	LC_ALL = "en_US.UTF-8",
	LC_CTYPE = "UTF-8",
	LANG = (unset)
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
/usr/bin/locale: Cannot set LC_CTYPE to default locale: No such file or directory
/usr/bin/locale: Cannot set LC_MESSAGES to default locale: No such file or directory
/usr/bin/locale: Cannot set LC_ALL to default locale: No such file or directory
Generating locales (this might take a while)...
  en_US.UTF-8...character map file `UTF-8' not found: No such file or directory
default character map file `ANSI_X3.4-1968' not found: No such file or directory
 done
Generation complete.
*** update-locale: Error: invalid locale settings:  LC_ALL=en_US.UTF-8 LANGUAGE=en_US.UTF-8 LANG=en_US.UTF-8
Inside the locale-gen file there is no locale?

Code: Select all


#!/bin/sh

set -e

LOCALEGEN=/etc/locale.gen
LOCALES=/usr/share/i18n/locales
USER_LOCALES=/usr/local/share/i18n/locales
if [ -n "$POSIXLY_CORRECT" ]; then
  unset POSIXLY_CORRECT
fi


[ -f $LOCALEGEN ] || exit 0;
[ -s $LOCALEGEN ] || exit 0;

KEEP=
if [ "$1" = '--keep-existing' ]; then
        KEEP=1
fi

if [ -z "$KEEP" ]; then
        # Remove all old locale dir and locale-archive before generating new
        # locale data.
        rm -rf /usr/lib/locale/locale-archive || true
fi

umask 022

is_entry_ok() {
  if [ -n "$locale" -a -n "$charset" ] ; then
    true
  else
    echo "error: Bad entry '$locale $charset'"
    false
  fi
}

echo "Generating locales (this might take a while)..."
while read locale charset; do \
        case $locale in \#*) continue;; "") continue;; esac; \
        is_entry_ok || continue
        if [ "$KEEP" ] && PERL_BADLANG=0 perl -MPOSIX -e \
            'exit 1 unless setlocale(LC_ALL, $ARGV[0])' "$locale"; then
                continue
        fi
        echo -n "  `echo $locale | sed 's/\([^.\@]*\).*/\1/'`"; \
        echo -n ".$charset"; \
        echo -n `echo $locale | sed 's/\([^\@]*\)\(\@.*\)*/\2/'`; \
        echo -n '...'; \
        if [ -f $USER_LOCALES/$locale ] ; then
            input=$USER_LOCALES/$locale
        elif [ -f $LOCALES/$locale ]; then 
            input=$locale
        else
            input=`echo $locale | sed 's/\([^.]*\)[^@]*\(.*\)/\1\2/'`
            if [ -f $USER_LOCALES/$input ]; then
                input=$USER_LOCALES/$input
            fi
        fi
        localedef -i $input -c -f $charset -A /usr/share/locale/locale.alias $locale || :; \
        echo ' done'; \
done < $LOCALEGEN
echo "Generation complete."




Re: Memory leak? System "hangs" after some time. (webcamserver)

Posted: Mon Mar 04, 2019 12:33 pm
by jamesh
Very odd that dmesg uis not found - this is installed by default on Raspbian.

You can check /var/log/ssylog* files as an alternative.

Re: Memory leak? System "hangs" after some time. (webcamserver)

Posted: Mon Mar 04, 2019 1:53 pm
by pertm84
Thanks, I've checked the etc/default/locale:

Code: Select all

#  File generated by update-locale
LANG=en_US.UTF-8
# LANGUAGE=en_US.UTF-8
I've also tried all the tips on the internet:

export LC_ALL=en_US.UTF-8
bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)

In the syslog, I couldn't see anything particular, some error messages from ddocker, but I've uninstalled that now, maybe that could be what caused it. Now I am all hung up in fixing this locale issue :oops: :roll:

Re: Memory leak? System "hangs" after some time. (webcamserver)

Posted: Tue Mar 05, 2019 11:44 am
by pertm84
Ok, so I tried looking further into the issue with my hangups, and noticed this using htop:

Image
url to image in case of img fails https://ibb.co/ctV7s3y

So it looks like I have two instances of Motion running? Or am I interpreting the htop wrong?

Re: Memory leak? System "hangs" after some time. (webcamserver)

Posted: Tue Mar 05, 2019 12:08 pm
by DougieLawson
You're interpreting it wrong. I'd guess that pid 7094 spawned pid 7095 to run some subtasks.

Re: Memory leak? System "hangs" after some time. (webcamserver)

Posted: Tue Mar 05, 2019 12:26 pm
by pertm84
Ok, thanks Dougie. At least I can remove that as a probable cause for the hangups ;)

Re: Memory leak? System "hangs" after some time. (webcamserver)

Posted: Wed Mar 06, 2019 4:14 pm
by pertm84
Ok, so everything worked for 9 days, now I get "Network is unreachable" when I try to SSH into it.
This is driving me nuts, because the raspberry is 515 km away from me. :twisted:

I know the router is working, because I have a wifi smartplug on the same network, which is working.

Re: Memory leak? System "hangs" after some time. (webcamserver)

Posted: Wed Mar 06, 2019 4:25 pm
by RaTTuS
look at a watchdog timer to reboot your RPI if it goes meh

Re: Memory leak? System "hangs" after some time. (webcamserver)

Posted: Thu Mar 07, 2019 7:35 am
by pertm84
RaTTuS wrote:
Wed Mar 06, 2019 4:25 pm
look at a watchdog timer to reboot your RPI if it goes meh
Thanks for the tip! I am about to give up the Pi though, I think I'll just buy an IP-camera and save a whole-lot-of-work. I'll bring home the Pi and do some research on why it is so unstable, maybe I can find another use for it..