Andy10
Posts: 19
Joined: Fri Sep 05, 2014 11:01 pm

RasPi web scraper & txt notification

Fri Apr 08, 2016 7:16 pm

Howdy. Not sure if I should post this here or under software, but it's an automation project to me so I'll start here.

The gist of what I'm trying to do is as follows->

1) get data from a website
2) send text alert if it exceeds a certain value.
3) store the value in a text file
3) repeat

I have managed to google my way into getting the code to almost work. My issue now is that it's very ugly, probably insecure, and doesn't quite work.

When I manually run the code
$ ./cost.sh
it will store the value in the text file but when it runs through cron, it doesn't store the value.

Any thoughts/suggestions would be appreciated. More than willing to RTFM but not sure which one to start with at this point. Also not a programmer so I'm trying to keep things simple and reproducible if I want to change the code down the line. I'm also trying to keep this to bash and python as those are the only languages I've managed to get some familiarity with at this point.

Eventually if I can get this code to work, the plan will be to light a certain color of LED using the GPIO's. The color would change based on the value, but that will be in the future.

Here's my code so far:

I edited the root crontab using:
$ sudo crontab -e:

This is my crontab entry to execute my bash script every 5 minutes

Code: Select all

*/5 * * * * /home/chester/iso/working/cost.sh
Here is the main code file cost.sh

Code: Select all

#!/bin/bash
#define variables
[email protected]
Pass=password
Site=https://mysite.com/api/xml
xmltag=CostTotal
WarnMsg="cost exceeded 100$ and is "
AlarmMsg="!!!cost EXCEEDED 500$ and is"
Cost=

# find out if during the last run a message was sent
MsgSent=$(cat sent.var)

#curl command  gets xml file from mysite.com, -s suppresses output & -k  does an unsecure method
# then greps for specific tag in xml
Cost=$(curl -s -k --user $User:$Pass $Site |  \
 grep -oPm1 "(?<=<$xmltag>)[^<]+") 

# strips decimal value from Cost variable
CostInt=${Cost/.*}

# This will check if the current value is above a limit AND if we have not sent an alert yet for that limit
# If it is above a value and we haven't sent a msg, send the message and set MsgSent fla
if [ "$CostInt" -ge "500" -a  "$MsgSent" -eq "0" ]
then
	python sendmail.py $Cost $AlarmMsg
	MsgSent=1

# checks if Cost is >=100 AND we haven't sent message last time
elif [ "$CostInt" -ge "100" -a "$MsgSent" -eq "0" ]
then
	python sendmail.py $Cost $WarnMsg
	MsgSent=1

elif [ "$CostInt" -lt "100"]
then
        MsgSent=0

else
        exit 1
        echo "$(date +"%x %r %Z")       error" >> cost.txt
fi

echo $MsgSent > sent.var 
echo "$(date +"%x %r %Z")	 $Cost"  >> cost.txt
This is the python script sendmail.py that actually sends the message. It is an email to my cellular provider and they turn into a text.

Code: Select all

import sys
import smtplib

#assumes you are calling script by using ~$ python sendmail.py value1 value2
# value1 is number passed to script for LMP value
# value2 is message to be displayed

fromaddr = '[email protected]'
toaddrs = '[email protected]'

msg1 = str(sys.argv[2])  
msg2 = str(sys.argv[1])

msg = msg1 + msg2 + ' $perunit'

# Credentials (if needed)
username = '[email protected]'
password = 'mypass'

# The actual mail send
server = smtplib.SMTP('smtp.gmail.com:587')
server.starttls()
server.login(username,password)
server.sendmail(fromaddr, toaddrs, msg)
server.quit()

#print msg
Really hoping it's an easy fix to get it working but looking for any suggestions people may have to make it better as well.

Thank you in advance!

Andy10
Posts: 19
Joined: Fri Sep 05, 2014 11:01 pm

Re: RasPi web scraper & txt notification

Sat Apr 09, 2016 3:49 pm

I think I solved my own problem. After posting and sleeping on it, I started poking around some more with cron. I changed the file names to have the full path and it appears to be working based on a suggestion I found here->

http://askubuntu.com/questions/23009/re ... s-not-work

hope this might help anyone else running into this issue.

[EDIT]: - I also changed the file name to be just 'cost' instead of 'cost.sh' not sure if it made a difference or if the hardcoded file path fixed it, but it works now!

User avatar
DougieLawson
Posts: 37624
Joined: Sun Jun 16, 2013 11:19 pm
Location: Basingstoke, UK
Contact: Website Twitter

Re: RasPi web scraper & txt notification

Sat Apr 09, 2016 4:35 pm

Why not do everything in python with BeautifulSoup4? BS4 gives you a really easy way to scrape a web page.

To run stuff from cron add a file in /etc/cron.d.
Note: Having anything humorous in your signature is completely banned on this forum. Wear a tin-foil hat and you'll get a ban.

Any DMs sent on Twitter will be answered next month.

This is a doctor free zone.

Andy10
Posts: 19
Joined: Fri Sep 05, 2014 11:01 pm

Re: RasPi web scraper & txt notification

Sun Apr 17, 2016 3:06 am

DougieLawson wrote:Why not do everything in python with BeautifulSoup4? BS4 gives you a really easy way to scrape a web page.
Took a look at BS4, it looks interesting but a little above my current coding level. I'm still only good at understanding others code and modifying for my use, not so much with writing things from scratch. I'm working on getting better at python but my only formal coding experience was a c++ class 15 years ago. I am trying to work my way through the codeacademy python stuff but slow going due to time constraints.

DougieLawson wrote:To run stuff from cron add a file in /etc/cron.d.
Is this an alternate way to do the same as 'sudo crontab -e' or a better way?


Thank you for the input!

User avatar
DougieLawson
Posts: 37624
Joined: Sun Jun 16, 2013 11:19 pm
Location: Basingstoke, UK
Contact: Website Twitter

Re: RasPi web scraper & txt notification

Sun Apr 17, 2016 11:50 am

I think the BS4 stuff could be a good way to jump in at the deep end with python programming. Your old programming experience will come flooding back (it's like riding a bike, you don't forget it).

It's the way I prefer to do stuff in cron. It's easier to see what's there.

There's a whole bunch of /etc/cron* folders for hourly, daily, weekly, monthly cron jobs. Those things are driven from /etc/crontab. Take a look, it's all standard stuff on every Linux system.
Note: Having anything humorous in your signature is completely banned on this forum. Wear a tin-foil hat and you'll get a ban.

Any DMs sent on Twitter will be answered next month.

This is a doctor free zone.

Return to “Automation, sensing and robotics”