Cron / Python not running on schedule - intermittently

[Dranoweb]

I've had a good look around, and don't seem to be able to find a relevant post here.

This is not strictly a Python issue, but it's the nearest category to find the problem - that I could find.
However python is very much involved.

I have a cron schedule running a light and sound show system at a military museum.
Seemingly at random, it will simply fail to fire one or more events.
events that have been tested thoroughly and have fired many times in the past without alteration.

I tested each line manually, and everything works.

Today, at 11:40 I noticed that events have failed to fire, and failed to do so for the following 10 minutes, until I manually initiated one of the commands (gramaphone.py)

After which time, everything seems to be happening normally.
some of my scripts disable cron and restart it, so I checked "sudo service cron status" and it reported that it was indeed running.
All code is done via root cron.

Cron and code samples below:

Root Cron schedule:

Code: Select all

##########CUSTOM CRON FOR GIPPSLAND ARMED FORCES MUSEUM LIGHT AND SOUND SHOW#####
SYNCHRONIZED_LIGHTS_HOME=/home/pi/lightshowpi
#################load shutdown and reboot scripts##########################
@reboot python /home/pi/shutdown.py >/dev/null 2>&1
@reboot python /home/pi/reboot.py >/dev/null 2>&1
#################load the clock############################################
@reboot sh /home/pi/loadclock.sh
################ Load everything else #####################################
00 10 * * * python /home/pi/indiansummer.py >/dev/null 2>&1
04 10 * * * python /home/pi/blue.py >/dev/null 2>&1
07 10 * * * python /home/pi/tuxedo.py >/dev/null 2>&1
10 10 * * * python /home/pi/resetrelays.py
############10 minutes before start########
10 10 * * * sh /home/pi/AA.sh >/dev/null 2>&1
14 10 * * * sh /home/pi/americansWWII.sh >/dev/null 2>&1
18 10 * * * python /home/pi/blues.py >/dev/null 2>&1
22 10 * * * python /home/pi/chatanooga.py >/dev/null 2>&1
25 10 * * * python /home/pi/track16.py >/dev/null 2>&1
30 10 * * * mpg321 -g 100 /home/pi/AAbass.mp3 >/dev/null 2>&1###################################
31 10 * * * python /home/pi/track17.py >/dev/null 2>&1
35 10 * * * python /home/pi/track8.py >/dev/null 2>&1
40 10 * * * python /home/pi/gramaphone.py >/dev/null 2>&1
45 10 * * * mpg321 -g 80 /home/pi/p51flyby.mp3 >/dev/null 2>&1
46 10 * * * sh /home/pi/AA.sh >/dev/null 2>&1
50 10 * * * sh /home/pi/AA.sh >/dev/null 2>&1
55 10 * * * sh /home/pi/AA.sh >/dev/null 2>&1
59 10 * * * mpg321 -g 80 /home/pi/p51flyby2 >/dev/null 2>&1
00 11 * * * python /home/pi/bluey.py >/dev/null 2>&1 ########new########
12 11 * * * python /home/pi/7th.py >/dev/null 2>&1################not working!!!###############
16 11 * * * mpg321 -g 80 /home/pi/p51flyby3.mp3 >/dev/null 2>&1
17 11 * * * python /home/pi/track15.py >/dev/null 2>&1
21 11 * * * sh /home/pi/AA.sh >/dev/null 2>&1
26 11 * * * python /home/pi/gramaphone.py >/dev/null 2>&1
31 11 * * * mpg321 -g 80 /home/pi/b24landing.mp3 >/dev/null 2>&1
32 11 * * * sh /home/pi/AA.sh >/dev/null 2>&1
36 11 * * * python /home/pi/endwar.py >/dev/null 2>&1
43 11 * * * mpg321 -g 60 /home/pi/AA35mm1.mp3 >/dev/null 2>&1
44 11 * * * python /home/pi/japan.py >/dev/null 2>&1#########9 min###########
55 11 * * * sh /home/pi/AA.sh >/dev/null 2>&1
01 12 * * * python /home/pi/churchill.py >/dev/null 2>&1
19 12 * * * mpg321 -g 60 /home/pi/M113passby.mp3 >/dev/null 2>&1
20 12 * * * sh /home/pi/AA.sh >/dev/null 2>&1
30 12 * * * mpg321 -g 60 /home/pi/M1A1passby.mp3 >/dev/null 2>&1
32 12 * * * sh /home/pi/AA.sh >/dev/null 2>&1
37 12 * * * sh /home/pi/AA.sh >/dev/null 2>&1
41 12 * * * sh /home/pi/AA.sh >/dev/null 2>&1
45 12 * * * mpg321 -g 60 /home/pi/shermanpassby.mp3 >/dev/null 2>&1
49 12 * * * python /home/pi/gramaphone.py >/dev/null 2>&1
52 12 * * * python /home/pi/japan.py >/dev/null 2>&1###################9 min - replace me with something new
0 13 * * * sh /home/pi/americansWWII.sh >/dev/null 2>&1#########should no longer overlap - but remember to check###########
15 13 * * * python /home/pi/menzies.py >/dev/null 2>&1
30 13 * * * python /home/pi/shelter.py
32 13 * * * mpg321 -g 30 /home/pi/march4.mp3 >/dev/null 2>&1
33 13 * * * sh /home/pi/AA.sh >/dev/null 2>&1
34 13 * * * mpg321 -g 60 /home/pi/M1A1passby3.mp3 >/dev/null 2>&1
35 13 * * * sh /home/pi/AA.sh >/dev/null 2>&1
36 13 * * * mpg321 -g 50 /home/pi/desielturbopassby.mp3 >/dev/null 2>&1
38 13 * * * mpg321 -g 45 /home/pi/APCM2run.mp3 >/dev/null 2>&1
40 13 * * * sh /home/pi/AA.sh >/dev/null 2>&1
40 13 * * * mpg321 -g 60 /home/pi/b17.mp3 >/dev/null 2>&1
43 13 * * * sh /home/pi/105.sh >/dev/null 2>&1
44 13 * * * sh /home/pi/AA.sh >/dev/null 2>&1
45 13 * * * mpg321 -g 60 /home/pi/m60tank.mp3 >/dev/null 2>&1
55 13 * * * sh /home/pi/AA.sh >/dev/null 2>&1
0 14 * * * mpg321 -g 80 /home/pi/p51flyby.mp3 >/dev/null 2>&1
01 14 * * * python /home/pi/tuxedo.py >/dev/null 2>&1
5 14 * * * sh /home/pi/AA.sh >/dev/null 2>&1
10 14 * * * mpg321 -g 40 /home/pi/winjeelpass2.mp3 >/dev/null 2>&1
15 14 * * * mpg321 -g 40 /home/pi/march3.mp3 >/dev/null 2>&1
17 14 * * * sh /home/pi/AA.sh >/dev/null 2>&1
20 14 * * * sh /home/pi/40mm.sh >/dev/null 2>&1
21 14 * * * python /home/pi/blue.py >/dev/null 2>&1
25 14 * * * python /home/pi/menzies.py >/dev/null 2>&1########## Replace with something new########
41 14 * * * mpg321 -g 80 /home/pi/b24flyby.mp3 >/dev/null 2>&1
42 14 * * * mpg321 -g 80 /home/pi/p51flyby.mp3 >/dev/null 2>&1
43 14 * * * sh /home/pi/105.sh >/dev/null 2>&1
44 14 * * * mpg321 -g 40 /home/pi/march3.mp3 >/dev/null 2>&1
45 14 * * * mpg321 -g 40 /home/pi/p51flyby.mp3 >/dev/null 2>&1
46 14 * * * sh /home/pi/AA.sh >/dev/null 2>&1
48 14 * * * mpg321 -g 30 /home/pi/m60tank.mp3 >/dev/null 2>&1
50 14 * * * sh /home/pi/AA.sh >/dev/null 2>&1
55 14 * * * python /home/pi/7th.py >/dev/null 2>&1
03 15 * * * sh /home/pi/AA.sh >/dev/null 2>&1
10 15 * * * python /home/pi/blue.py >/dev/null 2>&1
14 15 * * * python /home/pi/japan1945.py >/dev/null 2>&1################replace with something new#####
20 15 * * * python /home/pi/AA.py >/dev/null 2>&1
24 15 * * * python /home/pi/blues.py >/dev/null 2>&1
######################## Shutdown scriptis ####################################
32 15 * * * sh /home/pi/endofday.sh >/dev/null 2>&1
0 17 * * * /sbin/shutdown -h now >/dev/null 2>&1
######################## bugle calls ##########################################
@reboot sh /home/pi/bootup.sh >/dev/null 2>&1
0 12 * * * mpg321 -g 80 /home/pi/reveille.mp3 >/dev/null 2>&1
30 15 * * * mpg321 -g 80 /home/pi/last_post.mp3 >/dev/null 2>&1
30 15 * * * rm /home/pi/detect.sh
######################## End bugle calls ######################################
######################## SPECIAL EVENTS ######################################
#Remembereance Day####
0 11 31 10 * sh /home/pi/111111/111111.sh >/dev/null 2>&1
0 11 1 11 * sh /home/pi/111111/111111.sh >/dev/null 2>&1
0 11 2 11 * sh /home/pi/111111/111111.sh >/dev/null 2>&1
0 11 11 11 * sh /home/pi/111111/111111.sh >/dev/null 2>&1
0 11 14 11 * sh /home/pi/111111/111111.sh >/dev/null 2>&1
0 11 15 11 * sh /home/pi/111111/111111.sh >/dev/null 2>&1
0 11 16 11 * sh /home/pi/111111/111111.sh >/dev/null 2>&1
#End Rememberance Day####
######################## END SPECIAL EVENTS ######################################

gramaphone.py:

Code: Select all

import RPi.GPIO as GPIO
import time
import os
GPIO.setmode(GPIO.BCM)
GPIO.setup(25, GPIO.OUT)
#
#
#
#time.sleep(10)
GPIO.output(25, False)
os.system("mpg321 -g 20 /home/pi/gramaphone.mp3")
GPIO.output(25, True)

AA.sh:

Code: Select all

sudo pkill -sigstop mpg321
#sudo service cron stop
#sudo mpg321 -g 50 /home/pi/chimeup.mp3
#sudo mpg321 -g 50 /home/pi/p51approaching.mp3
#sudo mpg321 -g 100 /home/pi/p51flyby.mp3
sudo python /home/pi/resetrelays.py
sudo /usr/bin/python /home/pi/lightshowpi/py/synchronized_lights.py --file=/home/pi/distantAAjets.mp3
#sudo python /home/pi/lightson.py
#sudo service cron start
sudo python /home/pi/resetrelays.py

resetrelays.py:

Code: Select all

import RPi.GPIO as GPIO
import time
import os
GPIO.setmode(GPIO.BCM)
GPIO.setup(18, GPIO.OUT)
GPIO.setup(23, GPIO.OUT)
GPIO.setup(24, GPIO.OUT)
GPIO.setup(25, GPIO.OUT)
time.sleep(5)
GPIO.output(18, True)
GPIO.output(23, True)
GPIO.output(24, True)
GPIO.output(25, True)

[sprinkmeier]

...
some of my scripts disable cron and restart it, so I checked "sudo service cron status" and it reported that it was indeed running.
...

A cron-script restarting cron.
That's certainly ... different.

May I ask what problem you're trying to solve?
If it's a synchronisation problem there's a good chance your not solving it, just hiding it a little, and that

Code: Select all

import lockfile

will the start of a better solution.

[BirchJD]

I have a few suggestions to locate the issue.

Rather than redirecting the output of each script to /dev/null, redirect each one to it's own log file in a dedicated directory for these events. Then when it appears to stop running, before doing anything, look at the date of the log files:
ls -lart
This will show which log was written to last, and so the most likely thing which occurred before things stopped, by checking the last entries in the log.

Also in the comments some scripts don't have redirection of output, sometimes this will make a script hang in a cron because the cron has no terminal associated with it so the output has nowhere to go. Which is why you are redirecting most to /dev/null. I would check all are redirected to a log file to prevent any issues.
e.g. python /home/pi/resetrelays.py

Plus some lines have:
2>&1##
Redirect stderr to stdout with an immediate comment, this may not be an issue, but I would make sure there is a space between each occurrence of this as the shell may misinterpret the intention.

Jason

[Dranoweb]

...
some of my scripts disable cron and restart it, so I checked "sudo service cron status" and it reported that it was indeed running.
...

A cron-script restarting cron.
That's certainly ... different.

May I ask what problem you're trying to solve?
If it's a synchronisation problem there's a good chance your not solving it, just hiding it a little, and that

Code: Select all

import lockfile

will the start of a better solution.

Apologies, poor terminology there:
Some of the shell and python scripts launched from cron, terminate cron on start, and then re-start cron when they finish.

this is to avoid two scheduled tasks overlapping, in the case of a script that only runs one day each year.

The problem I have, is that seemingly at random, one cron task simply fails to fire.

[Dranoweb]

I have a few suggestions to locate the issue.

Rather than redirecting the output of each script to /dev/null, redirect each one to it's own log file in a dedicated directory for these events. Then when it appears to stop running, before doing anything, look at the date of the log files:
ls -lart
This will show which log was written to last, and so the most likely thing which occurred before things stopped, by checking the last entries in the log.

Also in the comments some scripts don't have redirection of output, sometimes this will make a script hang in a cron because the cron has no terminal associated with it so the output has nowhere to go. Which is why you are redirecting most to /dev/null. I would check all are redirected to a log file to prevent any issues.
e.g. python /home/pi/resetrelays.py

Plus some lines have:
2>&1##
Redirect stderr to stdout with an immediate comment, this may not be an issue, but I would make sure there is a space between each occurrence of this as the shell may misinterpret the intention.

Jason

This sounds like a handy tip - Thankyou.
I have quite a few scripts as you can see, but it will be worthwhile doing the tedious work to track this down.

As for the ## issue, I had overlooked that, I will check that - that could be potentially my issue and I had not noticed.

It would probably be obvious that I'm leaving notes in cron for myself as I iron out the issues - ironic that the notations would be the problems...

[sprinkmeier]

Yeah... that's what I thought...

Your system looks a little inside-out.
instead of having a scheduler build into your program you have lots of bits of program hanging off a scheduler.
Instead of using locks to avoid concurrent executing (i.e. "import lockfile") you're stopping the scheduler and hoping that it hasn't already started something in parallel (and also, by the way, impeding all the other things the scheduler does)

[ame]

You shouldn't stop and start cron. It's going to do its best to run the scheduled tasks despite your interference. Instead, you should have each task check that it is allowed to run.

Something like this, maybe:
http://serverfault.com/questions/82857/ ... bs-running

[Dranoweb]

Yeah... that's what I thought...

Your system looks a little inside-out.
instead of having a scheduler build into your program you have lots of bits of program hanging off a scheduler.
Instead of using locks to avoid concurrent executing (i.e. "import lockfile") you're stopping the scheduler and hoping that it hasn't already started something in parallel (and also, by the way, impeding all the other things the scheduler does)

In some cases I occasionally want a concurrent execution.
The system and project is not quite as simple as I have laid out here, but I have tried to simplify the issue to avoid confusion.

I agree that stopping cron is not a good solution, but in this case it's likely to do that on only 3 occasions a year.
That being said - every answer I have ever had from almost every programming forum includes a reply that approximates the following:
"why are you doing it THAT way, it's all wrong, just don't do it, do this other thing instead"
So from that I gather there are many ways to skin a cat, and when one is not being paid to skin said cat - enthusiasm for long nights of "correct" code routine is limited.

[Dranoweb]

You shouldn't stop and start cron. It's going to do its best to run the scheduled tasks despite your interference. Instead, you should have each task check that it is allowed to run.

Something like this, maybe:
http://serverfault.com/questions/82857/ ... bs-running

Firstly, don't take this as a rant, it's not - Imagine this read by spock... That's how it's intended

As stated in the last post, in some instances I desire multiple executions.
Scripts that I use to stop cron are scheduled to run about 3 times a year (remembrance day, day before and day after)

EG: Lightshowpi runs a 3 minute sample of distant artillery fire, and flickers the cabinet lights in said museum.
2 minutes in, I'd like a t30 to come rumbling through, but instead played with mpg321, not lightshow pi - so as not to make the relays go crazy
but at the same time, I don't want this to ocurr at every instance, only at specific - and dynamic times.
(and yes I could make two scripts I guess - But if I want to adjust the timing of the tank, I have to edit both cron AND the script)

cron is easy to update on the fly, and editing scripts not so easy, but as you point out, not perfect - nor is the pay packet of a volunteer, or a museum that survives only on donations.

The main reason for driving everything from cron, is that I frequently have to reschedule events at short notice, say when the assistant minister for defense visits (as happened recently)

It's easy to run down cron and shuffle a bombardment along a couple of minutes, even if it overlaps a vintage radio broadcast.

FYI - at last check, it would appear that leaving a space between notations "#" has solved the issue, but I won't know 100% until this weekend when I get a chance to log everything to a specific file, as has been suggested above.

for the curious, a couple of videos detailing what this is all driving are here:
perhaps this might answer some questions:

https://www.youtube.com/watch?v=8PG3MSPV6FM

https://www.youtube.com/watch?v=eajmJ7I32zY

[Dranoweb]

Just to reiterate for clarity:

I think that spacing out the notation #'s seems to have worked.
but I have to wait till this weekend for the opportunity to confirm.

[Dranoweb]

May I ask what problem you're trying to solve?
If it's a synchronisation problem there's a good chance your not solving it, just hiding it a little, and that

Code: Select all

import lockfile

will the start of a better solution.

Despite what I have said, I will follow this up - it's not a method I have used before, and probably a good one to remember.
I understand sound advice when I see it.

[BirchJD]

That's a really neat project. It's good to see a project being used in real life. Thanks for providing the videos.