Automated JSON keyword checker


8 posts
by mailliw » Fri May 10, 2013 11:01 am
Hi,

I'm looking for a very basic JSON parser for Python and Pi, which checks multiple feeds for a certain keyword. I'd either make this a daemon or run it via cron once a minute. The only associated action is to fire a notification to my Thingm Blink(1) - it's nothing more than "check for something new which matches [keyword]. If there's something new, flash the USB light".

All the solutions I've found so far seem to be way too "big" for pi. I suppose in essence, this is just like a basic RSS notifier but instead uses JSON.

Any pointers? Is this something I can do natively in Python, or would it require a specific package? Has anyone done similar?

Thanks

Will
Posts: 33
Joined: Fri May 10, 2013 10:51 am
by scotty101 » Fri May 10, 2013 5:53 pm
What's wrong with python's build it JSON encoder and decoder?

http://docs.python.org/2/library/json.html
Building a Raspberry Pi Home. https://plus.google.com/100752687760844230040/posts/RtZzoVqDGfy
Posts: 214
Joined: Fri Jun 08, 2012 6:03 pm
by mailliw » Fri May 10, 2013 5:59 pm
Yes, sorry - I should have been much more clear. I've started using urllib and import json, I'm just after a few tips about storing the data and then checking it.

One idea was to dump each json into a list, then into a file (shelf?). Then when it runs again, dump into another temp file. If the two sizes are different, then do something (notify).

Or is there a way to store data persistently in the ram without using local file storage?
Posts: 33
Joined: Fri May 10, 2013 10:51 am
by scotty101 » Fri May 10, 2013 6:10 pm
Sure there is

Code: Select all
import json
last_read = json.loads('{"t":2,"y":3}')
current_read = json.loads('{"t":2,"y":3}')
if current_read == last_read:
    print "Both are the same"
else:
   print "Difference detected"


Does that help? Or are you looking to compare a specific key in the JSON?

EDIT: I just re-read your post and realised what you want to do, you'll need to store the previous value to a file, I'd probably do this with the raw JSON text after you've done the urllib get.
Building a Raspberry Pi Home. https://plus.google.com/100752687760844230040/posts/RtZzoVqDGfy
Posts: 214
Joined: Fri Jun 08, 2012 6:03 pm
by scotty101 » Fri May 10, 2013 6:54 pm
Just knocked up this example to compare the difference in temperature from an online weather feed. It stores the entire JSON feed to a file so that it can compare the data next time.

Code: Select all
import urllib
import json

previous_weather_file = "weather_log.txt"
previous_weather = ""
try:
   log = open(previous_weather_file,"r")
   previous_weather = log.read()
   log.close()
except:
   print "No previous data"

f = urllib.urlopen("http://api.openweathermap.org/data/2.5/weather?q=Cheltenham,uk")
weather = f.read()

log = open(previous_weather_file,'w')
log.write(weather)
log.close()



weather_json = json.loads(weather)
#print weather
#print weather_json['weather']
curr_temp = float(weather_json['main']['temp'])-273.13
print "Temperature is: %.2f degrees C" % (curr_temp)


if (not previous_weather == ""):
   prev_weather_json = json.loads(previous_weather)
   prev_temp = float(prev_weather_json['main']['temp'])-273.13
   temp_diff = curr_temp - prev_temp
   
   if not( temp_diff == 0.0):
      print "Temperature has changed by: %.2f degrees C" % (temp_diff)
Building a Raspberry Pi Home. https://plus.google.com/100752687760844230040/posts/RtZzoVqDGfy
Posts: 214
Joined: Fri Jun 08, 2012 6:03 pm
by mailliw » Sun May 12, 2013 10:59 am
Thanks for this - sorry for such a slow reply. Wish I'd seen it sooner! My version is below; not sure what try/except are, so my code is probably ludicrously ugly. The one requirement I had was to filter the feed by a certain keyword - if you've any tips about what I should do to make it cleaner, shout!

At least it works :)

Code: Select all
#!/usr/bin/python
print 'Starting'
import urllib2, json, os, bliPy

url = 'jsonurl'

# download the json string
json_string = urllib2.urlopen(url).read()
print 'Downloaded json'
# de-serialize the string so that we can work with it
the_data = json.loads(json_string)

# get the content
theposts = the_data['posts']
print 'Going through the wickets'
#filter the wickets
forfile=''
for wicket in theposts:
    it=wicket['Content']
    findit = 'WICKET' in it
    if findit > 0:
        print 'Found!!'
        forfile+= wicket['Content']

# Got the latest data, and it's inside the var forfile
# Now we need to check this against the old file
print 'opening the file to read it'
oldfile=open('/home/pi/me/spider/hello/data.txt','r')
olddata=oldfile.read()

print olddata

if forfile == olddata:
    print 'Data the same'
else:
    print 'Data not the same. We have wickets! Do something here'
    interval = 1.75
    num = 15
    colours = '100,200,0'
    bl1 = bliPy.Blink1()
    bl1.rando(num)

    # We should now write this data to the file
    writenew = open('/home/pi/me/spider/hello/data.txt','w')
    writenew.write(forfile)
    writenew.close()

print 'end'
Posts: 33
Joined: Fri May 10, 2013 10:51 am
by sprinkmeier » Sun May 12, 2013 11:12 am
Consider writing your temporary file to /dev/shm. This is a RAM disk and will avoid wear-and-tear on your SD card.
Posts: 288
Joined: Mon Feb 04, 2013 10:48 am
by mailliw » Sun May 12, 2013 11:17 am
Good tip, thanks. Presume it's persistent until a reboot?
Posts: 33
Joined: Fri May 10, 2013 10:51 am