pertm84
Posts: 29
Joined: Sat Oct 20, 2018 10:59 am
Location: Norway

Memory leak? System "hangs" after some time. (webcamserver)

Mon Mar 04, 2019 11:11 am

Hi

I am quite new to this, but I have managed to set up a raspberry pi using stretch as a webserver for uploading stills to my wunderground account through ftp. UFW, fail2ban, apache etc is all set up and working.

Also, I set the camera to livestream whenever I want. Everything is working, but sometimes the system hangs, and I can't SSH into it. Therefore the only solution is to cut the power and perform a reboot, which makes everything working again.

This is output from top command:

Code: Select all

top - 11:57:08 up 7 days, 22:17,  2 users,  load average: 0.43, 0.37, 0.37
Tasks: 148 total,   1 running,  99 sleeping,   0 stopped,   5 zombie
%Cpu(s): 11.1 us,  1.0 sy,  0.0 ni, 87.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :   896672 total,   153296 free,   276868 used,   466508 buff/cache
KiB Swap:   102396 total,   102396 free,        0 used.   514568 avail Mem 
I created a script and added to cron for Motion to restart every 30 min (since it is unstable) and also a script for uploading tmp.jpg to my wunderground account every 15min. So Motion is writing jpeg to ram every 15 min, but will this be deleted/overwritten each time? Is this filling up my ram and swap?

I also suspect that the log-files may slow things down, as UFW fills up my log-files with unnecessary lines. I have no idea where to start my fail-search, since I have little experience with linux.
Last edited by pertm84 on Mon Mar 04, 2019 11:26 am, edited 1 time in total.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23083
Joined: Sat Jul 30, 2011 7:41 pm

Re: Memory leak? System "hangs" after some time. (webcamserver)

Mon Mar 04, 2019 11:15 am

If you are definitley writing the same file each tim,e then its not using up anything. Same with log files, the size of them should not affect speed.

It might help to get full `top` dumps every 15 minutes or so leading up to the lockup to see if anything is using increasing amounts of memory.

I pressume there is nothing in dmesg giving hints as to why things are going wrong?
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

pertm84
Posts: 29
Joined: Sat Oct 20, 2018 10:59 am
Location: Norway

Re: Memory leak? System "hangs" after some time. (webcamserver)

Mon Mar 04, 2019 11:37 am

Thanks for the quick answer! I'm not quite sure what to look for in dmesg, and everytime I had to do a hard-reset, it looks like the log-files has restarted.What should I look for in dmesg?

Also when I tried to type dmesg, I get:
-bash: dmesg: command not found

I know it must be installed, because when I: sudo apt-get install util-linux

Reading package lists... Done
Building dependency tree
Reading state information... Done
util-linux is already the newest version (2.29.2-1+deb9u1).
0 upgraded, 0 newly installed, 0 to remove and 121 not upgraded.

Also I tried: sudo apt-get install util-linux --reinstall

Then I got:

Code: Select all

sudo apt-get install util-linux --reinstall
Reading package lists... Done
Building dependency tree       
Reading state information... Done
0 upgraded, 0 newly installed, 1 reinstalled, 0 to remove and 121 not upgraded.
Need to get 895 kB of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://raspbian.trivini.no/raspbian stretch/main armhf util-linux armhf 2.29.2-1+deb9u1 [895 kB]
Fetched 895 kB in 2s (307 kB/s)     
apt-listchanges: Can't set locale; make sure $LC_* and $LANG are correct!
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
	LANGUAGE = (unset),
	LC_ALL = (unset),
	LC_CTYPE = "UTF-8",
	LANG = (unset)
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
dpkg: warning: 'tar' not found in PATH or not executable
dpkg: error: 1 expected program not found in PATH or not executable
Note: root's PATH should usually contain /usr/local/sbin, /usr/sbin and /sbin
E: Sub-process /usr/bin/dpkg returned an error code (2)
So I thought I should try to change locale, but it was already setup in raspi-config?

Code: Select all

sudo raspi-config
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
	LANGUAGE = (unset),
	LC_ALL = (unset),
	LC_CTYPE = "UTF-8",
	LANG = (unset)
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
/usr/bin/locale: Cannot set LC_CTYPE to default locale: No such file or directory
/usr/bin/locale: Cannot set LC_ALL to default locale: No such file or directory
Generating locales (this might take a while)...
  en_US.UTF-8...character map file `UTF-8' not found: No such file or directory
default character map file `ANSI_X3.4-1968' not found: No such file or directory
 done
Generation complete.
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
	LANGUAGE = (unset),
	LC_ALL = (unset),
	LC_CTYPE = "UTF-8",
	LANG = (unset)
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
	LANGUAGE = (unset),
	LC_ALL = (unset),
	LC_CTYPE = "UTF-8",
	LANG = (unset)
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
*** update-locale: Error: invalid locale settings:  LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LANGUAGE=en_US.UTF-8

pertm84
Posts: 29
Joined: Sat Oct 20, 2018 10:59 am
Location: Norway

Re: Memory leak? System "hangs" after some time. (webcamserver)

Mon Mar 04, 2019 12:17 pm

OK, it seems there is a problem with my locales. I am not able to install the US utf8.

I get:

Code: Select all

sudo dpkg-reconfigure locales                             
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
	LANGUAGE = "en_US.UTF-8",
	LC_ALL = "en_US.UTF-8",
	LC_CTYPE = "UTF-8",
	LANG = (unset)
    are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
/usr/bin/locale: Cannot set LC_CTYPE to default locale: No such file or directory
/usr/bin/locale: Cannot set LC_MESSAGES to default locale: No such file or directory
/usr/bin/locale: Cannot set LC_ALL to default locale: No such file or directory
Generating locales (this might take a while)...
  en_US.UTF-8...character map file `UTF-8' not found: No such file or directory
default character map file `ANSI_X3.4-1968' not found: No such file or directory
 done
Generation complete.
*** update-locale: Error: invalid locale settings:  LC_ALL=en_US.UTF-8 LANGUAGE=en_US.UTF-8 LANG=en_US.UTF-8
Inside the locale-gen file there is no locale?

Code: Select all


#!/bin/sh

set -e

LOCALEGEN=/etc/locale.gen
LOCALES=/usr/share/i18n/locales
USER_LOCALES=/usr/local/share/i18n/locales
if [ -n "$POSIXLY_CORRECT" ]; then
  unset POSIXLY_CORRECT
fi


[ -f $LOCALEGEN ] || exit 0;
[ -s $LOCALEGEN ] || exit 0;

KEEP=
if [ "$1" = '--keep-existing' ]; then
        KEEP=1
fi

if [ -z "$KEEP" ]; then
        # Remove all old locale dir and locale-archive before generating new
        # locale data.
        rm -rf /usr/lib/locale/locale-archive || true
fi

umask 022

is_entry_ok() {
  if [ -n "$locale" -a -n "$charset" ] ; then
    true
  else
    echo "error: Bad entry '$locale $charset'"
    false
  fi
}

echo "Generating locales (this might take a while)..."
while read locale charset; do \
        case $locale in \#*) continue;; "") continue;; esac; \
        is_entry_ok || continue
        if [ "$KEEP" ] && PERL_BADLANG=0 perl -MPOSIX -e \
            'exit 1 unless setlocale(LC_ALL, $ARGV[0])' "$locale"; then
                continue
        fi
        echo -n "  `echo $locale | sed 's/\([^.\@]*\).*/\1/'`"; \
        echo -n ".$charset"; \
        echo -n `echo $locale | sed 's/\([^\@]*\)\(\@.*\)*/\2/'`; \
        echo -n '...'; \
        if [ -f $USER_LOCALES/$locale ] ; then
            input=$USER_LOCALES/$locale
        elif [ -f $LOCALES/$locale ]; then 
            input=$locale
        else
            input=`echo $locale | sed 's/\([^.]*\)[^@]*\(.*\)/\1\2/'`
            if [ -f $USER_LOCALES/$input ]; then
                input=$USER_LOCALES/$input
            fi
        fi
        localedef -i $input -c -f $charset -A /usr/share/locale/locale.alias $locale || :; \
        echo ' done'; \
done < $LOCALEGEN
echo "Generation complete."




jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23083
Joined: Sat Jul 30, 2011 7:41 pm

Re: Memory leak? System "hangs" after some time. (webcamserver)

Mon Mar 04, 2019 12:33 pm

Very odd that dmesg uis not found - this is installed by default on Raspbian.

You can check /var/log/ssylog* files as an alternative.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

pertm84
Posts: 29
Joined: Sat Oct 20, 2018 10:59 am
Location: Norway

Re: Memory leak? System "hangs" after some time. (webcamserver)

Mon Mar 04, 2019 1:53 pm

Thanks, I've checked the etc/default/locale:

Code: Select all

#  File generated by update-locale
LANG=en_US.UTF-8
# LANGUAGE=en_US.UTF-8
I've also tried all the tips on the internet:

export LC_ALL=en_US.UTF-8
bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)

In the syslog, I couldn't see anything particular, some error messages from ddocker, but I've uninstalled that now, maybe that could be what caused it. Now I am all hung up in fixing this locale issue :oops: :roll:

pertm84
Posts: 29
Joined: Sat Oct 20, 2018 10:59 am
Location: Norway

Re: Memory leak? System "hangs" after some time. (webcamserver)

Tue Mar 05, 2019 11:44 am

Ok, so I tried looking further into the issue with my hangups, and noticed this using htop:

Image
url to image in case of img fails https://ibb.co/ctV7s3y

So it looks like I have two instances of Motion running? Or am I interpreting the htop wrong?

User avatar
DougieLawson
Posts: 35594
Joined: Sun Jun 16, 2013 11:19 pm
Location: Basingstoke, UK
Contact: Website Twitter

Re: Memory leak? System "hangs" after some time. (webcamserver)

Tue Mar 05, 2019 12:08 pm

You're interpreting it wrong. I'd guess that pid 7094 spawned pid 7095 to run some subtasks.
Note: Having anything humorous in your signature is completely banned on this forum. Wear a tin-foil hat and you'll get a ban.

Any DMs sent on Twitter will be answered next month.

This is a doctor free zone.

pertm84
Posts: 29
Joined: Sat Oct 20, 2018 10:59 am
Location: Norway

Re: Memory leak? System "hangs" after some time. (webcamserver)

Tue Mar 05, 2019 12:26 pm

Ok, thanks Dougie. At least I can remove that as a probable cause for the hangups ;)

pertm84
Posts: 29
Joined: Sat Oct 20, 2018 10:59 am
Location: Norway

Re: Memory leak? System "hangs" after some time. (webcamserver)

Wed Mar 06, 2019 4:14 pm

Ok, so everything worked for 9 days, now I get "Network is unreachable" when I try to SSH into it.
This is driving me nuts, because the raspberry is 515 km away from me. :twisted:

I know the router is working, because I have a wifi smartplug on the same network, which is working.

User avatar
RaTTuS
Posts: 10381
Joined: Tue Nov 29, 2011 11:12 am
Location: North West UK

Re: Memory leak? System "hangs" after some time. (webcamserver)

Wed Mar 06, 2019 4:25 pm

look at a watchdog timer to reboot your RPI if it goes meh
How To ask Questions :- http://www.catb.org/esr/faqs/smart-questions.html
WARNING - some parts of this post may be erroneous YMMV

1QC43qbL5FySu2Pi51vGqKqxy3UiJgukSX
Covfefe

pertm84
Posts: 29
Joined: Sat Oct 20, 2018 10:59 am
Location: Norway

Re: Memory leak? System "hangs" after some time. (webcamserver)

Thu Mar 07, 2019 7:35 am

RaTTuS wrote:
Wed Mar 06, 2019 4:25 pm
look at a watchdog timer to reboot your RPI if it goes meh
Thanks for the tip! I am about to give up the Pi though, I think I'll just buy an IP-camera and save a whole-lot-of-work. I'll bring home the Pi and do some research on why it is so unstable, maybe I can find another use for it..

Return to “Camera board”