IoT Data storing and graphing options (offline)

[whonch]

Hi,

I'm sure this topic has already come out a lots of times here, but I've been checking and I don't think there are that many options.

Basically I would like to know what options do we have when storing and creating graphs for our internet of things devices in a raspberry pi.

I am currently storing my data in CSV files and graphing them using scripts (to delimitate the time period) and Gnuplot. It kinda works, but has a few downsides: graphs need to be scheduled every few minutes if you want them to be up to date (the more graphs, more CPU usage), and those graphs are not interactive, so you cannot zoom in or out, or show a graph between two dates, without having to create another graph.

So this weekend I tried influxdb + Grafana. It kinda worked, and I loved grafana graphs, but influxdb used the 100% of the CPU and was making the whole system slow and unstable. Also, Grafana does not support CSV files, which is a pity.

I briefly tried plotly, which accepted data from CSV files, but the generation of each graph was slow (slower than gnuplot), so I don't think it's a viable option either.

I'm using a first-generation Raspberry Pi B, by the way.

So, any alternatives to gnuplot, considering I want to store all my data locally without the need for an internet connection and access the graphs from an internal web page?

[Davies]

matplotlib in python is good for graphing
https://matplotlib.org/gallery/index.html
https://matplotlib.org/xkcd/users/pyplot_tutorial.html
you can add a menu with zoom option, legend, multiple axis and have it plot from a file.

Code: Select all

import numpy as np
import matplotlib.pyplot as plt
import matplotlib.dates as mdates
from datetime import datetime


datefunc = lambda x: mdates.date2num(datetime.strptime(x, '%d %m %Y %H %M %S'))

dates, levels1 = np.genfromtxt('test.txt', delimiter=',', converters={0: datefunc}, dtype=float, unpack=True)

#plt.rcParams['toolbar'] = 'None'
fig = plt.figure()
ax = fig.add_subplot(111)

ax.set_xticks(dates)
ax.xaxis.set_major_formatter(mdates.DateFormatter('%d/%m/%Y %H:%M:%S'))

ax.plot_date(dates, levels1, label='Moisture Level', color='g', ls='-', marker='o', linewidth=5)

ax.set_title('Moisture Level')
ax.set_ylabel('Water Content')
ax.grid(True)
plt.rcParams.update({'font.size': 20})


fig.autofmt_xdate(rotation=90)
plt.tight_layout()
plt.legend(loc='upper left', numpoints=1, prop={'size': 10})
plt.locator_params(axis='x', nbins=32)
plt.show()

would read from a file named test.txt (test.txt would be so large that the x axis tick date stamp would overlap each other so i use "plt.locator_params(axis='x', nbins=32)" top ensure that only 32 ticks are listed across the bottom, which is ok for my screen size)
test.txt would contain...

Code: Select all

19 01 2016 23 10 16,18
19 01 2016 23 12 04,24
19 01 2016 23 13 51,25
19 01 2016 23 15 39,24
19 01 2016 23 17 27,25
19 01 2016 23 19 15,24
19 01 2016 23 21 03,24
19 01 2016 23 22 50,23
19 01 2016 23 24 38,23
19 01 2016 23 26 26,22
19 01 2016 23 28 14,22
19 01 2016 23 30 01,22
19 01 2016 23 31 49,22
19 01 2016 23 33 37,21
19 01 2016 23 35 24,22
19 01 2016 23 37 12,20
19 01 2016 23 39 00,20
19 01 2016 23 40 47,19
19 01 2016 23 42 35,20
19 01 2016 23 44 23,19
19 01 2016 23 46 10,19
19 01 2016 23 47 58,20
19 01 2016 23 49 46,20
19 01 2016 23 51 33,19
19 01 2016 23 53 21,19
19 01 2016 23 55 09,20
19 01 2016 23 56 56,19
19 01 2016 23 58 44,19
20 01 2016 00 00 31,19
20 01 2016 00 02 19,19
20 01 2016 00 04 07,18
20 01 2016 00 05 54,18
20 01 2016 00 07 42,19
20 01 2016 00 09 30,19
20 01 2016 00 11 17,18
20 01 2016 00 13 05,18
20 01 2016 00 14 52,18
20 01 2016 00 16 40,18
20 01 2016 00 18 28,18
20 01 2016 00 20 15,18
20 01 2016 00 22 03,17
20 01 2016 00 23 51,18
20 01 2016 00 25 38,18
20 01 2016 00 27 26,18
20 01 2016 00 29 13,17
20 01 2016 00 31 01,17
20 01 2016 00 32 48,16
20 01 2016 00 34 36,18
20 01 2016 00 36 24,16
20 01 2016 00 38 11,16
20 01 2016 00 39 59,17
20 01 2016 00 41 46,17
20 01 2016 00 43 34,16
20 01 2016 00 45 22,17
20 01 2016 00 47 09,16
20 01 2016 00 48 57,17
20 01 2016 00 50 44,16
20 01 2016 00 52 32,16
20 01 2016 00 54 19,16
20 01 2016 00 56 07,16
20 01 2016 00 57 54,16
20 01 2016 00 59 42,16
20 01 2016 01 01 30,16
20 01 2016 01 03 17,15
20 01 2016 01 05 05,16
20 01 2016 01 06 52,15
20 01 2016 01 08 40,16
20 01 2016 01 10 27,15
20 01 2016 01 12 15,16
20 01 2016 01 14 02,15
20 01 2016 01 15 50,15
20 01 2016 01 17 38,15
20 01 2016 01 19 25,15
20 01 2016 01 21 13,16
20 01 2016 01 23 00,15
20 01 2016 01 24 48,15
20 01 2016 01 26 35,15
20 01 2016 01 28 23,16
20 01 2016 01 30 10,16
20 01 2016 01 31 58,16
20 01 2016 01 33 45,15
20 01 2016 01 35 33,15
20 01 2016 01 37 21,15
20 01 2016 01 39 08,15
20 01 2016 01 40 56,15
20 01 2016 01 42 43,15
20 01 2016 01 44 31,15
20 01 2016 01 46 18,15
20 01 2016 01 48 06,15
20 01 2016 01 49 53,15
20 01 2016 01 51 41,14
20 01 2016 01 53 28,15
20 01 2016 01 55 16,15
20 01 2016 01 57 03,15
20 01 2016 01 58 51,14
20 01 2016 02 00 38,15
20 01 2016 02 02 26,14
20 01 2016 02 04 14,15
20 01 2016 02 06 01,14
20 01 2016 02 07 49,14
20 01 2016 02 09 36,14
20 01 2016 02 11 24,14
20 01 2016 02 13 11,14
20 01 2016 02 14 59,14
20 01 2016 02 16 46,14
20 01 2016 02 18 34,14
20 01 2016 02 20 21,13
20 01 2016 02 22 09,14
20 01 2016 02 23 56,14
20 01 2016 02 25 44,14
20 01 2016 02 27 31,14
20 01 2016 02 29 19,14
20 01 2016 02 31 06,13
20 01 2016 02 32 54,14
20 01 2016 02 34 41,15
20 01 2016 02 36 29,14
20 01 2016 02 38 16,14
20 01 2016 02 40 04,14
20 01 2016 02 41 51,14
20 01 2016 02 43 39,14
20 01 2016 02 45 26,14
20 01 2016 02 47 14,14
20 01 2016 02 49 01,14
20 01 2016 02 50 49,14
20 01 2016 02 52 36,13
20 01 2016 02 54 24,14
20 01 2016 02 56 11,14
20 01 2016 02 57 59,14
20 01 2016 02 59 46,14
20 01 2016 03 01 34,13
20 01 2016 03 03 21,13
20 01 2016 03 05 09,14
20 01 2016 03 06 56,13
20 01 2016 03 08 44,13
20 01 2016 03 10 46,43
20 01 2016 03 13 06,63
20 01 2016 03 15 27,63
20 01 2016 03 17 49,63
20 01 2016 03 20 13,63
20 01 2016 03 22 39,63
20 01 2016 03 25 04,63
20 01 2016 03 27 31,63
20 01 2016 03 29 59,63
20 01 2016 03 32 29,63
20 01 2016 03 34 58,63
20 01 2016 03 37 27,63

being a time/date stamp and level of moisture separated by comma (,)
i then have this data written to file in the script which monitors the moisture level
i used to have it record the info based on a timer(as above recordings) but now it only records the moisture if the moisture level has changed, as below

Code: Select all

import time

dummy_moisture = 100
current_moisture = 0

while 1:
    if not current_moisture == dummy_moisture:
        timestr = time.strftime('%d %m %Y %H %M %S')
        data = timestr + "," + ("%02d" % current_moisture) + '\n'
        with open('test.txt', 'a') as fp:  # requires for test.txt to already exist
            fp.write(data)
        dummy_moisture = current_moisture
    # script here to change value of current_moisture, ie..
    if current_moisture == dummy_moisture:
        current_moisture += 1
        if current moisture > 100:
            current_moisture = 0
    time.sleep(1)

you could have it save to the graph to a .png file then open that in your website, or even host a web.py server in the same script allowing access via the raspberry IP address

[whonch]

Thanks!

I'll give it a go.

This afternoon i found about dygraphs, and it's pretty interesting. It's basically a javascript charting library and accepts CSV files right away. Although customizing the graphs is not the easiest thing to do...

[Davies]

dygraphs does look interesting, i was going to use matplotlib via web.py webserver to access my graphs but now i think ill be using dygraphs instead.
ive not been able to pass info straight from a python script to dygraphs but can post to file then read that with the javascript

[wonx]

I've been playing with dygraphs and I think it's more or less what I needed. The main disadvantage I found is that it only accepts one CSV file as input, so you cannot compare the values from two different sensors if they are stored on different files. Maybe you could dynamically load and combine the data using javascript before loading the graph, but I wonder if it would make things too slow on the browser side.