georgelza
Posts: 68
Joined: Sun Oct 09, 2016 2:17 pm

Re: Scripts dying every night at 0:00-01:00'ish

Tue Nov 21, 2017 4:32 am

Hi all

Have some data collection scripts, EnviroPhat (on a RPi 3B) and a DHT22 on a 2nd Pi (RPi Zero W),

Both of the Python scripts seem to stop getting data every night/morning around 01:00.

Anyone else seen this, and if yes, how you fix it?

G
Last edited by georgelza on Thu Nov 23, 2017 3:45 am, edited 1 time in total.

User avatar
DougieLawson
Posts: 30470
Joined: Sun Jun 16, 2013 11:19 pm
Location: Basingstoke, UK
Contact: Website

Re: Scripts saying every night at 0:00-01:00'ish

Tue Nov 21, 2017 6:58 am

What's your timezone is it Central European Time? Because in that case 01:00 CET is 00:00 UTC.

What language are you using? Does your script have a memory leak? Are there any messages for the oom (out of memory) killer in /var/log/syslog?
Microprocessor, Raspberry Pi & Arduino Hacker
Mainframe database troubleshooter
MQTT Evangelist
Twitter: @DougieLawson

Since 2012: 1B*5, 2B*2, B+, A+, Zero*2, 3B*3

Please post ALL technical questions on the forum. Do not send private messages.

georgelza
Posts: 68
Joined: Sun Oct 09, 2016 2:17 pm

Re: Scripts dying every night at 0:00-01:00'ish

Tue Nov 21, 2017 7:07 am

GMT+2
Python

One's running on a RPi3B with 1GB and other on RPi Zero W with 500MB, if it was a memory leak they would not be behaving so similar/same time dying habit. the Zero would be allot earlier.

G
Last edited by georgelza on Thu Nov 23, 2017 3:45 am, edited 1 time in total.

User avatar
DougieLawson
Posts: 30470
Joined: Sun Jun 16, 2013 11:19 pm
Location: Basingstoke, UK
Contact: Website

Re: Scripts saying every night at 0:00-01:00'ish

Tue Nov 21, 2017 7:21 am

What's in the logs? /var/log/syslog & /var/log/messages?

Put some print() or logging calls in your code so you can see what it's doing when it dies.
Here's a banal example of how to write stuff from a python program to /var/log/syslog

Code: Select all

import syslog
import random

syslog.syslog('Processing started')
error = (random.random() >= 0.5)
if error:
    syslog.syslog(syslog.LOG_ERR, 'Processing NOT started')
Microprocessor, Raspberry Pi & Arduino Hacker
Mainframe database troubleshooter
MQTT Evangelist
Twitter: @DougieLawson

Since 2012: 1B*5, 2B*2, B+, A+, Zero*2, 3B*3

Please post ALL technical questions on the forum. Do not send private messages.

georgelza
Posts: 68
Joined: Sun Oct 09, 2016 2:17 pm

Re: Scripts dying every night at 0:00-01:00'ish

Tue Nov 21, 2017 7:29 am

/var/log/messages and /var/log/syslog is clean, just random even notifications, no errors.

Will add some error catching in and report back.

G
Last edited by georgelza on Thu Nov 23, 2017 3:45 am, edited 1 time in total.

User avatar
Jednorozec
Posts: 790
Joined: Sun Nov 24, 2013 2:17 pm
Location: Deposit, NY

Re: Scripts saying every night at 0:00-01:00'ish

Tue Nov 21, 2017 10:09 am

Perhaps a transient on your power lines when something turns on at the same time every day.
The most important leg of a three legged stool is the one that's missing.
It's called thinking. Why don't you try it sometime?

georgelza
Posts: 68
Joined: Sun Oct 09, 2016 2:17 pm

Re: Scripts dying every night at 0:00-01:00'ish

Wed Nov 22, 2017 8:25 am

Will look at adding that error catch code tonight, BUT...

Both Raspberry's stop sending data at exactly midnight.

I have a @reboot script thats suppose to start the script/data collection on restarts.

This does not feel like a memory error, code error, it's to exact/aligned.

G
Last edited by georgelza on Thu Nov 23, 2017 3:45 am, edited 1 time in total.

Rhadamanthys
Posts: 9
Joined: Wed Nov 22, 2017 5:40 pm

Re: Scripts saying every night at 0:00-01:00'ish

Wed Nov 22, 2017 6:21 pm

As a former development engineer I was used to hear my programmer blaming the hardware if something went unexpectedly. Usualy we found the software bug after days of testing the hardware.

Here's a trivial code snippet that will raise a divide by zero error the first second of the day:

import time, datetime
while True:
x=1/(int((time.time()))%86400)

I'm sure your code is far more sophisticated, but watch for runtime errors

georgelza
Posts: 68
Joined: Sun Oct 09, 2016 2:17 pm

Re: Scripts dying every night at 0:00-01:00'ish

Wed Nov 22, 2017 6:30 pm

... I've not blamed the hardware, if it was fault, then it would not have been this regular/same time every day.

I'm trying to figure out if there is a automated process something that might initiate the restart or reboot or something.

and well, you are talking to a ex programmer... moved into a difference branch of technology.

G
Last edited by georgelza on Thu Nov 23, 2017 3:45 am, edited 1 time in total.

georgelza
Posts: 68
Joined: Sun Oct 09, 2016 2:17 pm

Re: Scripts dying every night at 0:00-01:00'ish

Thu Nov 23, 2017 3:44 am

Update, added a except block, with a exception catch.

pushing messages to syslog.

nothing,

also changed so that the script is now run using supervisor, where previously it was just a nohup python <script> &

and still on clock 0:12 for both Pi's...

Strange, thinking it's network, but then why does it not simply continue submitting data once the network is back, since it is on the network when I check on them at 05:00, some more error catching needed, somehow.

G

User avatar
DougieLawson
Posts: 30470
Joined: Sun Jun 16, 2013 11:19 pm
Location: Basingstoke, UK
Contact: Website

Re: Scripts dying every night at 0:00-01:00'ish

Thu Nov 23, 2017 7:25 am

Post your code.
Microprocessor, Raspberry Pi & Arduino Hacker
Mainframe database troubleshooter
MQTT Evangelist
Twitter: @DougieLawson

Since 2012: 1B*5, 2B*2, B+, A+, Zero*2, 3B*3

Please post ALL technical questions on the forum. Do not send private messages.

georgelza
Posts: 68
Joined: Sun Oct 09, 2016 2:17 pm

Re: Scripts dying every night at 0:00-01:00'ish

Thu Nov 23, 2017 8:08 am

Code: Select all

#
#	George
#	georgelza@gmail.com
#	
#	Lets try our hand at Python
#


from envirophat import weather
import requests
import time
import csv
import httplib, urllib

import logging
import logging.handlers

# constants
URL = 'https://corlysis.com:8086/write'
READING_DATA_PERIOD_MS = 5000.0
SENDING_PERIOD = 2
MAX_LINES_HISTORY = 1000

db="environmental"
token="a99bcfd4xxxxxxxxxxxxx152f"
csvfile 	= "/home/pi/projects/source/corlysis/temp.csv"
barometric = 0

logger = logging.getLogger('myRead_Barometric_EnviroPhat')
logger.setLevel(logging.INFO)
handler = logging.handlers.SysLogHandler('/dev/log')
formatter = logging.Formatter('Python: { "loggerName":"%(name)s", "message":"%(message)s"}')
handler.formatter = formatter
logger.addHandler(handler)

def main():

    corlysis_params = {"db": DB, "u": "token", "p": TOKEN, "precision": "ms"}

    payload = ""
    counter = 1
    problem_counter = 0

    # infinite loop
    while True:
        unix_time_ms = int(time.time()*1000)

		timeC = time.strftime("%Y") + "/" + time.strftime("%m") + "/" + time.strftime("%d") + " " + time.strftime(
			"%H") + ':' + time.strftime("%M") + ':' + time.strftime('%S')

        try:
            # read sensor data and convert it to line protocol
		    barometric = weather.pressure()
		    barometric = round(barometric, 2)
		    barometric = barometric / 100

        except Exception, e:
            # Build up 'data' to be used as a array to write to Syslogd
            data = ["Exception, Read weather.pressure: ", e]
            # Write message to SysLog
            logger.info(data)
        #End Try

        line = "sensors_data pressure={} {}\n".format(barometric, unix_time_ms)

		data = ["Study EnviroPhat", timeC, barometric ]

		with open(csvfile, "a") as output:
			writer = csv.writer(output, delimiter=",", lineterminator='\n')
			writer.writerow(data)
		# End with

		params = urllib.urlencode({'field3': barometric,'key': 'SFF0xxxxxxx0I4YE'})  # use your API key generated in the thingspeak channels for the value of 'key'

		headers = {"Content-typZZe": "application/x-www-form-urlencoded", "Accept": "text/plain"}
		conn = httplib.HTTPConnection("api.thingspeak.com:80")

		try:
			conn.request("POST", "/update", params, headers)
			response = conn.getresponse()
			print response.status, response.reason
			data = response.read()
			conn.close()

        except Exception, e:
            # Build up 'data' to be used as a array to write to Syslogd
            data = ["Exception, Post to api.thingspeak: ", e]
            # Write message to SysLog
            logger.info(data)
        #End Try


	    payload += line

        if counter % SENDING_PERIOD == 0:
            try:
                # try to send data to cloud
                r = requests.post(URL, params=corlysis_params, data=payload)
                if r.status_code != 204:
                    raise Exception("data not written")
                payload = ""
            except Exception, e:
                problem_counter += 1

                # Build up 'data' to be used as a array to write to Syslogd
                data = ["Exception, Cannot write to InfluxDB: ", e]
                # Write message to SysLog
                logger.info(data)

                if problem_counter == MAX_LINES_HISTORY:
                    problem_counter = 0
                    payload = ""

            # End Try

        counter += 1

        # wait for selected time
        time_diff_ms = int(time.time()*1000) - unix_time_ms
        print(time_diff_ms)
        if time_diff_ms < READING_DATA_PERIOD_MS:
            time.sleep((READING_DATA_PERIOD_MS - time_diff_ms)/1000.0)


if __name__ == "__main__":
    main()
    
Last edited by georgelza on Thu Nov 23, 2017 8:12 am, edited 1 time in total.

User avatar
DougieLawson
Posts: 30470
Joined: Sun Jun 16, 2013 11:19 pm
Location: Basingstoke, UK
Contact: Website

Re: Scripts dying every night at 0:00-01:00'ish

Thu Nov 23, 2017 8:09 am

Please edit your post.

Add [code] at the top





and [/code] at the bottom.
Microprocessor, Raspberry Pi & Arduino Hacker
Mainframe database troubleshooter
MQTT Evangelist
Twitter: @DougieLawson

Since 2012: 1B*5, 2B*2, B+, A+, Zero*2, 3B*3

Please post ALL technical questions on the forum. Do not send private messages.

georgelza
Posts: 68
Joined: Sun Oct 09, 2016 2:17 pm

Re: Scripts dying every night at 0:00-01:00'ish

Thu Nov 23, 2017 8:12 am

Done :)

DougieLawson wrote:
Thu Nov 23, 2017 8:09 am
Please edit your post.

Add [code] at the top





and [/code] at the bottom.

User avatar
DougieLawson
Posts: 30470
Joined: Sun Jun 16, 2013 11:19 pm
Location: Basingstoke, UK
Contact: Website

Re: Scripts dying every night at 0:00-01:00'ish

Thu Nov 23, 2017 8:14 am

conn = httplib.HTTPConnection("api.thingspeak.com:80")
should be in your try: except: block.
Microprocessor, Raspberry Pi & Arduino Hacker
Mainframe database troubleshooter
MQTT Evangelist
Twitter: @DougieLawson

Since 2012: 1B*5, 2B*2, B+, A+, Zero*2, 3B*3

Please post ALL technical questions on the forum. Do not send private messages.

georgelza
Posts: 68
Joined: Sun Oct 09, 2016 2:17 pm

Re: Scripts dying every night at 0:00-01:00'ish

Thu Nov 23, 2017 9:29 am

Will do tonight, lets see if we can pickup a exception, probably where it is happening :)

G

hippy
Posts: 2347
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Scripts dying every night at 0:00-01:00'ish

Thu Nov 23, 2017 11:24 am

georgelza wrote:
Thu Nov 23, 2017 8:08 am

Code: Select all

        time_diff_ms = int(time.time()*1000) - unix_time_ms
        print(time_diff_ms)
        if time_diff_ms < READING_DATA_PERIOD_MS:
            time.sleep((READING_DATA_PERIOD_MS - time_diff_ms)/1000.0)
My instinct is that it's somehow getting stuck there as you roll past midnight. Similar to 'Tnext=Tnow+N' where that may lead to a 'Tnext' which can never be an actual time.

Rhadamanthys
Posts: 9
Joined: Wed Nov 22, 2017 5:40 pm

Re: Scripts dying every night at 0:00-01:00'ish

Thu Nov 23, 2017 12:53 pm

Just an idea: What if your internet provider restarts your connection every night? Or your router is commissioned to do so?
Suggest you set your router to deny internet to your pi and see how your script performs

Comment: You also may want to condense the line:

timeC = time.strftime("%Y") + "/" + time.strftime("%m") + "/" + time.strftime("%d") + " " + time.strftime(
"%H") + ':' + time.strftime("%M") + ':' + time.strftime('%S')

to this:

timeC = time.strftime('%Y/%m/%d %H:%M:%S')

a bit more efficient, it will not solve the nightly death though

georgelza
Posts: 68
Joined: Sun Oct 09, 2016 2:17 pm

Re: Scripts dying every night at 0:00-01:00'ish

Thu Nov 23, 2017 1:36 pm

Hi there

Thanks for the suggestions,

So:

#1 moved the conn = httplib.HTTPConnection("api.thingspeak.com:80") to inside the try: block to see if we can catch a error

#2 my router does actually restart itself every night, at exactly mid night, but don't believe this is the cause I have data points after midnight before the "crash", and also when I get to the Pi's at 05;10 in the morning, then both is working/connected, and the code is such that even if it misses a send now, at the next iteration it creates a new connection and sends, so suppose to be a bit more resilient.

#3 running a pretty secure firewall configuration, Telco is def not reaching in and resetting it :)

To confirm #2 and #3 I've just changed my router reboot to 01:00, lets see if the problem also moves a hour.

G

NotRequired
Posts: 152
Joined: Sat Apr 29, 2017 10:36 am
Location: Denmark

Re: Scripts dying every night at 0:00-01:00'ish

Thu Nov 23, 2017 2:35 pm

georgelza wrote:
Thu Nov 23, 2017 1:36 pm
the code is such that even if it misses a send now, at the next iteration it creates a new connection and sends
If the exception thrown by a failing network connection is not caught then the script will terminate because python does not know how to recover from an uncaught exception - exceptions must be handled by the programmer ;)
Please do not ask questions in private messages, they will not help others.

georgelza
Posts: 68
Joined: Sun Oct 09, 2016 2:17 pm

Re: Scripts dying every night at 0:00-01:00'ish

Thu Nov 23, 2017 2:51 pm

Agree.

But supervisor will restart the script.

G

Rhadamanthys
Posts: 9
Joined: Wed Nov 22, 2017 5:40 pm

Re: Scripts dying every night at 0:00-01:00'ish

Thu Nov 23, 2017 5:27 pm

Your scripts die every midnight on 2 RPis, your router restarts every midnight but you are pretty confident there's no correlation...
Rather than waiting another night to see if the issue moves with restart, what about simply pulling the dsl cable off the router to see how the scripts like the missing internet.
supervisor: I understand it gives up on repeated errors

Rhadamanthys
Posts: 9
Joined: Wed Nov 22, 2017 5:40 pm

Re: Scripts dying every night at 0:00-01:00'ish

Thu Nov 23, 2017 5:51 pm

Just learned a bit about supervisor.
For debugging, what prevents you from running your script in the Python shell?
Gives you error messages right away without any overhead code for error logging

georgelza
Posts: 68
Joined: Sun Oct 09, 2016 2:17 pm

Re: Scripts dying every night at 0:00-01:00'ish

Thu Nov 23, 2017 7:29 pm

Router is restarted at midnight, scripts fail after midnight.

Scripts upload data to https://corlysis.com/, I have data points being uploaded after midnight, every night, after the router have restarted.

As for supervisor, the error count before failure is set at 10 000,

G
Rhadamanthys wrote:
Thu Nov 23, 2017 5:27 pm
Your scripts die every midnight on 2 RPis, your router restarts every midnight but you are pretty confident there's no correlation...
Rather than waiting another night to see if the issue moves with restart, what about simply pulling the dsl cable off the router to see how the scripts like the missing internet.
supervisor: I understand it gives up on repeated errors

georgelza
Posts: 68
Joined: Sun Oct 09, 2016 2:17 pm

Re: Scripts dying every night at 0:00-01:00'ish

Thu Nov 23, 2017 7:31 pm

If you have a look at the script, once started it goes to a while True: loop, where it gets values, uploads values and then sleeps for 60 seconds.

This runs perfectly all day till just after 0:00.

if you look at the code above you'll noticed I've added a fair amount of error catching last 24 hours, hoping to get some meaning error messages in syslog tomorrow, as I'm not going to stay awake till mid night.

G
Rhadamanthys wrote:
Thu Nov 23, 2017 5:51 pm
Just learned a bit about supervisor.
For debugging, what prevents you from running your script in the Python shell?
Gives you error messages right away without any overhead code for error logging

Return to “General discussion”

Who is online

Users browsing this forum: Google [Bot], stubright and 33 guests