All Python Scripts Stop Working After ~3hrs With No Errors?

[Shirey2010]

Hi all,

Thank you for reading. I've set up my RPi3 to communicate with an Allen Bradley L32 PLC. The goal is simply to take a picture using the RPiCamera after receiving a trigger from the PLC. It will save the file with a name assigned from the PLC, upload it to a remote computer using an FTP server, then delete the picture. I've got this part working, however after around 3 hours the two scripts that I've written will just disappear from the 'Services' window.

Here's the first script that's handling all of what I described above:

Code: Select all

import ftplib
from picamera import PiCamera
import time
from Example_Program import *
import os

# Camera Settings
camera = PiCamera()
# Camera Factory Resolution Is 3280 x 2464
camera.resolution = (3280, 2464)
# camera.brightness = 45
# camera.contrast = 50
counter = 0
FileNameTag = 'Pi_Filename_With_JPG'

def doWork():
    # Handshake To PLC To Confirm Connection
    ex_write('Pi_Handshake', '1')
    if ex_read('Trigger_Shutter') is True:
        try:
        # Starting Camera And Taking Picture
            camera.start_preview()
            time.sleep(2)
            camera.capture(ex_read(FileNameTag))
            camera.stop_preview()
        # Logging Into FTP Server
            print("Trying To Login To FTP...")
            ftp = ftplib.FTP()
            ftp.connect('192.168.1.4', '800')
            ftp.login('scott', '')
            ex_write('FTP_Logon_Successful', '1')
            ftp.set_pasv(False)
            # Sending File To FTP Server
            fullname = ex_read(FileNameTag)
            name = os.path.split(fullname)[1]
            f = open(fullname, "rb")
            ftp.storbinary('STOR ' + name, f)
            f.close()
        # Remove File From Pi
            os.remove(fullname)
        except Exception:
            pass
while True:
    try:
        doWork()
    except Exception:
        doWork()

My second script that also stops working is a simple 'Restart' script that will reboot the RPi3 using a signal from the PLC. This is just to ensure I can easily get the system back to working if things go sideways. Here's that script:

Code: Select all

import time
from Example_Program import *
import os

def restart():
    if ex_read('HMI_Reset_Pb') == 1:
        try:
            os.system('sudo shutdown -r now')
        except Exception:
            pass
while True:
    try:
        restart()
    except Exception:
        restart()

I've also set these two scripts up to run at start up using the rc.local file. This is working well.

I'm very new to Python, RPi's and the like, so if there's anything glaringly obvious in this code structure, please feel free to let me know!

[B.Goode]

My hunch is that it would be necessary to examine the Example_Program library module you are using, in particular the ex_read() function that is repeatedly called by both example scripts.

Could it be that ex_read() is opening a file handle on each call but not closing it, so that eventually you exhaust the pool of such open files?

Edit: added - to prove whether it is all Python scripts as you claim, simply write a trivial looping test case that prints a progress message and sleeps for 5 minutes, without invoking the Example_Programs library....

[Shirey2010]

My hunch is that it would be necessary to examine the Example_Program library module you are using, in particular the ex_read() function that is repeatedly called by both example scripts.

Could it be that ex_read() is opening a file handle on each call but not closing it, so that eventually you exhaust the pool of such open files?

Hi B.Goode!

You're right; I should have included that in my original post.

Code: Select all

'''
Just a few examples of how to do some
basic things with the PLC
'''
from Ethernet_IP_Program import PLC



def ex_read(tag):
  '''
  simple tag read
  '''
  ret = comm.Read(tag)
  return ret



def ex_readArray(tag, length):
  '''
  read tag array
  '''
  ret = comm.Read(tag, length)
  print ret

def ex_multiRead():
  '''
  read multiple tags in a single packet
  '''
  tag1 = "Dataman_Filename_Final"
  tag2 = "Printer_Start_To_Finish"
  ret = comm.MultiRead(tag1, tag2)
  print ret

def ex_write(tag, value):
  '''
  simple tag write
  '''
  comm.Write(tag, value)

def ex_getPLCTime():
  '''
  get the PLC's clock time
  '''
  ret = comm.GetPLCTime()
  print ret

def ex_setPLCTime():
    '''
    set the PLC's clock time to the workstations time
    '''
    comm.SetPLCTime()

def ex_discover():
  '''
  discover all the Ethernet I/P devices on the network and print the
  results
  '''
  print "Discovering Ethernet I/P devices, please wait..."
  device = comm.Discover()
  print "Total number of devices found (in no particular order):", len(device)
  print ""

  for i in xrange(len(device)):
    print '(' + str(i+1) + ') ' + device[i].IPAddress
    print "     ProductName/Code - ", device[i].ProductName, "(" + str(device[i].ProductCode) + ")"
    print "     Vendor/DeviceID  - ", device[i].Vendor, "(" + str(device[i].DeviceID) + ")"
    print "     Revision/Serial  - ", device[i].Revision, device[i].SerialNumber
    print ""

def ex_getTags():
  '''
  request the tag database from the PLC and put the results in a text file
  '''
  ret = comm.GetTagList()

  # print out all the tags to a file
  with open("TagList.txt", "w") as text_file:
    for tag in ret:
      name = "Name: " + tag.TagName
      dtype = "Type: " + str(tag.DataType)
      offset= "Offset: " + str(tag.Offset)
      end = '\n'

      # some tab space formatting
      if len(name) >= 36: tabs = '\t'
      if len(name) < 36 and len(name) >= 32: tabs = '\t'*2
      if len(name) < 32 and len(name) >= 28: tabs = '\t'*3
      if len(name) < 28 and len(name) >= 24: tabs = '\t'*4
      if len(name) < 24 and len(name) >= 20: tabs = '\t'*5
      if len(name) < 20 and len(name) >= 16: tabs = '\t'*6
      if len(name) < 16 and len(name) >= 12: tabs = '\t'*7
      if len(name) < 12: tabs = '\t'*8

      line = name + tabs + dtype + '\t' + offset + end
      text_file.write(line)

# define our communication
comm = PLC()
comm.IPAddress = '192.168.1.10'
comm.ProcessorSlot = 0

Could you expand more on what you mean by closing file handlers? That sounds like something I should know about. I'll do some reading on that now.

[B.Goode]

That just pushes the puzzle back to another level of import.

The function that is being repeated is comm.Read()

comm is an instance of PLC, which has been imported from Ethernet_IP_Program. So that's where you need to look...

But maybe we are looking at open network connections not literally 'files'. (Although in the abstract Unix treats them the same.)

[Shirey2010]

That just pushes the puzzle back to another level of import.

The function that is being repeated is comm.Read()

comm is an instance of PLC, which has been imported from Ethernet_IP_Program. So that's where you need to look...

But maybe we are looking at open network connections not literally 'files'. (Although in the abstract Unix treats them the same.)

Unfortunately the Ethernet_IP_Program is 1500+ lines long so I'm unable to post that here. However, I'll post what I think is relevant for the problem:

Code: Select all

def _readTag(self, tag, elements):
    '''
    processes the read request
    '''
    self.Offset = 0
    
    if not _connect(self): return None

    t,b,i = TagNameParser(tag, 0)
    InitialRead(self, t, b)

    datatype = self.KnownTags[b][0]
    bitCount = self.CIPTypes[datatype][0] * 8

    if datatype == 211:
        # bool array
        tagData = _buildTagIOI(self, tag, isBoolArray=True)
        words = _getWordCount(i, elements, bitCount)
        readRequest = _addReadIOI(self, tagData, words)
    elif BitofWord(t):
        # bits of word
        split_tag = tag.split('.')
        bitPos = split_tag[len(split_tag)-1]
        bitPos = int(bitPos)

        tagData = _buildTagIOI(self, tag, isBoolArray=False)
        words = _getWordCount(bitPos, elements, bitCount)

        readRequest = _addReadIOI(self, tagData, words)
    else:
        # everything else
        tagData = _buildTagIOI(self, tag, isBoolArray=False)
        readRequest = _addReadIOI(self, tagData, elements)
        
    eipHeader = _buildEIPHeader(self, readRequest)
    retData = _getBytes(self, eipHeader)
    status = unpack_from('<h', retData, 48)[0]

    if status == 0 or status == 6:
        return _parseReply(self, tag, elements, retData)
    else:
        if status in cipErrorCodes.keys():
            err = cipErrorCodes[status]
        else:
            err = 'Unknown error'
        raise Exception('Read failed, ' + err)

EDIT:

Out of curiosity, we changed our Restart program to be a little cleaner. It seems to have stuck around longer than our 'Camera' program which is still the same as the original post. Here's the new program:

Code: Select all

import time
from Example_Program import *
import os


while True:
    try:
        if ex_read('HMI_Reset_Pb') == 1:
            os.system('sudo shutdown -r now')
    except Exception:
            pass

We changed up how we were doing our loop. Would it make sense that this would fix anything?

[B.Goode]

Have you considered sharing your experience and problems with the author/developer/maintainer of the library modules you are using?

[elParaguayo]

It's also not helpful having try... except... blocks that catch every exception (it's actually pretty bad practice to do this). This is probably why your code fails silently.

There may be some exceptions that it's ok to ignore but it seems clear that there are some that are causing you problems!