Re: Scripts dying every night at 0:00-01:00'ish

[georgelza]

Hi all

Have some data collection scripts, EnviroPhat (on a RPi 3B) and a DHT22 on a 2nd Pi (RPi Zero W),

Both of the Python scripts seem to stop getting data every night/morning around 01:00.

Anyone else seen this, and if yes, how you fix it?

G

[DougieLawson]

What's your timezone is it Central European Time? Because in that case 01:00 CET is 00:00 UTC.

What language are you using? Does your script have a memory leak? Are there any messages for the oom (out of memory) killer in /var/log/syslog?

[georgelza]

GMT+2
Python

One's running on a RPi3B with 1GB and other on RPi Zero W with 500MB, if it was a memory leak they would not be behaving so similar/same time dying habit. the Zero would be allot earlier.

G

[DougieLawson]

What's in the logs? /var/log/syslog & /var/log/messages?

Put some print() or logging calls in your code so you can see what it's doing when it dies.
Here's a banal example of how to write stuff from a python program to /var/log/syslog

Code: Select all

import syslog
import random

syslog.syslog('Processing started')
error = (random.random() >= 0.5)
if error:
    syslog.syslog(syslog.LOG_ERR, 'Processing NOT started')

[georgelza]

/var/log/messages and /var/log/syslog is clean, just random even notifications, no errors.

Will add some error catching in and report back.

G

[Jednorozec]

Perhaps a transient on your power lines when something turns on at the same time every day.

[georgelza]

Will look at adding that error catch code tonight, BUT...

Both Raspberry's stop sending data at exactly midnight.

I have a @reboot script thats suppose to start the script/data collection on restarts.

This does not feel like a memory error, code error, it's to exact/aligned.

G

[Rhadamanthys]

As a former development engineer I was used to hear my programmer blaming the hardware if something went unexpectedly. Usualy we found the software bug after days of testing the hardware.

Here's a trivial code snippet that will raise a divide by zero error the first second of the day:

import time, datetime
while True:
x=1/(int((time.time()))%86400)

I'm sure your code is far more sophisticated, but watch for runtime errors

[georgelza]

... I've not blamed the hardware, if it was fault, then it would not have been this regular/same time every day.

I'm trying to figure out if there is a automated process something that might initiate the restart or reboot or something.

and well, you are talking to a ex programmer... moved into a difference branch of technology.

G

[georgelza]

Update, added a except block, with a exception catch.

pushing messages to syslog.

nothing,

also changed so that the script is now run using supervisor, where previously it was just a nohup python <script> &

and still on clock 0:12 for both Pi's...

Strange, thinking it's network, but then why does it not simply continue submitting data once the network is back, since it is on the network when I check on them at 05:00, some more error catching needed, somehow.

G

[DougieLawson]

Post your code.

[georgelza]

Code: Select all

#
#	George
#	[email protected]
#	
#	Lets try our hand at Python
#


from envirophat import weather
import requests
import time
import csv
import httplib, urllib

import logging
import logging.handlers

# constants
URL = 'https://corlysis.com:8086/write'
READING_DATA_PERIOD_MS = 5000.0
SENDING_PERIOD = 2
MAX_LINES_HISTORY = 1000

db="environmental"
token="a99bcfd4xxxxxxxxxxxxx152f"
csvfile 	= "/home/pi/projects/source/corlysis/temp.csv"
barometric = 0

logger = logging.getLogger('myRead_Barometric_EnviroPhat')
logger.setLevel(logging.INFO)
handler = logging.handlers.SysLogHandler('/dev/log')
formatter = logging.Formatter('Python: { "loggerName":"%(name)s", "message":"%(message)s"}')
handler.formatter = formatter
logger.addHandler(handler)

def main():

    corlysis_params = {"db": DB, "u": "token", "p": TOKEN, "precision": "ms"}

    payload = ""
    counter = 1
    problem_counter = 0

    # infinite loop
    while True:
        unix_time_ms = int(time.time()*1000)

		timeC = time.strftime("%Y") + "/" + time.strftime("%m") + "/" + time.strftime("%d") + " " + time.strftime(
			"%H") + ':' + time.strftime("%M") + ':' + time.strftime('%S')

        try:
            # read sensor data and convert it to line protocol
		    barometric = weather.pressure()
		    barometric = round(barometric, 2)
		    barometric = barometric / 100

        except Exception, e:
            # Build up 'data' to be used as a array to write to Syslogd
            data = ["Exception, Read weather.pressure: ", e]
            # Write message to SysLog
            logger.info(data)
        #End Try

        line = "sensors_data pressure={} {}\n".format(barometric, unix_time_ms)

		data = ["Study EnviroPhat", timeC, barometric ]

		with open(csvfile, "a") as output:
			writer = csv.writer(output, delimiter=",", lineterminator='\n')
			writer.writerow(data)
		# End with

		params = urllib.urlencode({'field3': barometric,'key': 'SFF0xxxxxxx0I4YE'})  # use your API key generated in the thingspeak channels for the value of 'key'

		headers = {"Content-typZZe": "application/x-www-form-urlencoded", "Accept": "text/plain"}
		conn = httplib.HTTPConnection("api.thingspeak.com:80")

		try:
			conn.request("POST", "/update", params, headers)
			response = conn.getresponse()
			print response.status, response.reason
			data = response.read()
			conn.close()

        except Exception, e:
            # Build up 'data' to be used as a array to write to Syslogd
            data = ["Exception, Post to api.thingspeak: ", e]
            # Write message to SysLog
            logger.info(data)
        #End Try


	    payload += line

        if counter % SENDING_PERIOD == 0:
            try:
                # try to send data to cloud
                r = requests.post(URL, params=corlysis_params, data=payload)
                if r.status_code != 204:
                    raise Exception("data not written")
                payload = ""
            except Exception, e:
                problem_counter += 1

                # Build up 'data' to be used as a array to write to Syslogd
                data = ["Exception, Cannot write to InfluxDB: ", e]
                # Write message to SysLog
                logger.info(data)

                if problem_counter == MAX_LINES_HISTORY:
                    problem_counter = 0
                    payload = ""

            # End Try

        counter += 1

        # wait for selected time
        time_diff_ms = int(time.time()*1000) - unix_time_ms
        print(time_diff_ms)
        if time_diff_ms < READING_DATA_PERIOD_MS:
            time.sleep((READING_DATA_PERIOD_MS - time_diff_ms)/1000.0)


if __name__ == "__main__":
    main()
    

[DougieLawson]

Please edit your post.

Add [code] at the top





and [/code] at the bottom.

[georgelza]

Done

Please edit your post.

Add [code] at the top





and [/code] at the bottom.

[DougieLawson]

conn = httplib.HTTPConnection("api.thingspeak.com:80")
should be in your try: except: block.

[georgelza]

Will do tonight, lets see if we can pickup a exception, probably where it is happening

G

[hippy]

Code: Select all

        time_diff_ms = int(time.time()*1000) - unix_time_ms
        print(time_diff_ms)
        if time_diff_ms < READING_DATA_PERIOD_MS:
            time.sleep((READING_DATA_PERIOD_MS - time_diff_ms)/1000.0)

My instinct is that it's somehow getting stuck there as you roll past midnight. Similar to 'Tnext=Tnow+N' where that may lead to a 'Tnext' which can never be an actual time.

[Rhadamanthys]

Just an idea: What if your internet provider restarts your connection every night? Or your router is commissioned to do so?
Suggest you set your router to deny internet to your pi and see how your script performs

Comment: You also may want to condense the line:

timeC = time.strftime("%Y") + "/" + time.strftime("%m") + "/" + time.strftime("%d") + " " + time.strftime(
"%H") + ':' + time.strftime("%M") + ':' + time.strftime('%S')

to this:

timeC = time.strftime('%Y/%m/%d %H:%M:%S')

a bit more efficient, it will not solve the nightly death though

[georgelza]

Hi there

Thanks for the suggestions,

So:

#1 moved the conn = httplib.HTTPConnection("api.thingspeak.com:80") to inside the try: block to see if we can catch a error

#2 my router does actually restart itself every night, at exactly mid night, but don't believe this is the cause I have data points after midnight before the "crash", and also when I get to the Pi's at 05;10 in the morning, then both is working/connected, and the code is such that even if it misses a send now, at the next iteration it creates a new connection and sends, so suppose to be a bit more resilient.

#3 running a pretty secure firewall configuration, Telco is def not reaching in and resetting it

To confirm #2 and #3 I've just changed my router reboot to 01:00, lets see if the problem also moves a hour.

G

[NotRequired]

the code is such that even if it misses a send now, at the next iteration it creates a new connection and sends

If the exception thrown by a failing network connection is not caught then the script will terminate because python does not know how to recover from an uncaught exception - exceptions must be handled by the programmer

[georgelza]

Agree.

But supervisor will restart the script.

G

[Rhadamanthys]

Your scripts die every midnight on 2 RPis, your router restarts every midnight but you are pretty confident there's no correlation...
Rather than waiting another night to see if the issue moves with restart, what about simply pulling the dsl cable off the router to see how the scripts like the missing internet.
supervisor: I understand it gives up on repeated errors

[Rhadamanthys]

Just learned a bit about supervisor.
For debugging, what prevents you from running your script in the Python shell?
Gives you error messages right away without any overhead code for error logging

[georgelza]

Router is restarted at midnight, scripts fail after midnight.

Scripts upload data to https://corlysis.com/, I have data points being uploaded after midnight, every night, after the router have restarted.

As for supervisor, the error count before failure is set at 10 000,

G

Your scripts die every midnight on 2 RPis, your router restarts every midnight but you are pretty confident there's no correlation...
Rather than waiting another night to see if the issue moves with restart, what about simply pulling the dsl cable off the router to see how the scripts like the missing internet.
supervisor: I understand it gives up on repeated errors

[georgelza]

If you have a look at the script, once started it goes to a while True: loop, where it gets values, uploads values and then sleeps for 60 seconds.

This runs perfectly all day till just after 0:00.

if you look at the code above you'll noticed I've added a fair amount of error catching last 24 hours, hoping to get some meaning error messages in syslog tomorrow, as I'm not going to stay awake till mid night.

G

Just learned a bit about supervisor.
For debugging, what prevents you from running your script in the Python shell?
Gives you error messages right away without any overhead code for error logging