Data Logging, Plotting Help

[xXAzazelXx]

Hi Guys,

Apologies, I'm very new to all of this. Can someone please point me in the right direction of where to start with collecting and plotting data from sensors?

I have recently purchased Enviro-Plus https://shop.pimoroni.com/products/enviro-plus and ideally would like to plot the data from the sensors over time.

I was thinking of something like smokeping, prtg, mrtg equivalent but for this?

What do you guys use to collect and plot the date from your sensors outputs?

[Gavinmc42]

I usually just collect the data into a csv file and once a day write it to SD.
Lots of way to then plot csv data.
Dygraphs.js is my webserver method.

Basic shell script is used, cron for the daily writes, watch for running sensor reading script.
Watch for grabbing the data, it calls a shell script that reads and converts it to csv and writes it to ram file.

Cron and watch have proven bullet proof running on PiCore Linux for years.

[xXAzazelXx]

I usually just collect the data into a csv file and once a day write it to SD.
Lots of way to then plot csv data.
Dygraphs.js is my webserver method.

Basic shell script is used, cron for the daily writes, watch for running sensor reading script.
Watch for grabbing the data, it calls a shell script that reads and converts it to csv and writes it to ram file.

Cron and watch have proven bullet proof running on PiCore Linux for years.

Thank you for for the advice

How are you collecting the data into csv? python? I have tried to edit the sample script into for loop but im not having much luck.

Code: Select all

#!/usr/bin/env python

import time
import csv
from bme280 import BME280
from datetime import datetime

try:
    from smbus2 import SMBus
except ImportError:
    from smbus import SMBus

dt = datetime.now()

full_date = dt.strftime("%Y-%m-%d")
full_time = dt.strftime("%H:%M:%S")
full_day = dt.strftime("%A")
year = dt.year
month = dt.month
date = dt.day
hour = dt.hour

bus = SMBus(1)
bme280 = BME280(i2c_dev=bus)



temperature = bme280.get_temperature()
pressure = bme280.get_pressure()
humidity = bme280.get_humidity()
csvWrite = full_date, full_time, full_day, year, month, date, hour, temperature, pressure, humidity
csvFile = open("/home/pi/enviroplot.csv", "a")

while True:
    with csvFile:
        writer = csv.writer(csvFile)
        writer.writerow(csvWrite)
        time.sleep(1)
        

[Gavinmc42]

Awk and Sed are your Shell script friends.
In Linux everything is a file, Awk and Sed can write/edit files.
Horrible old stuff that comes with every Linux OS and just works.

If you use a webserver then no need for all that x11 desktop bloat stuff, just a headless Linux OS.
Easy fit all this on a 128MB card.
Busybox has a webserver or you can pick from a bunch of options.
I use shell script and Awk/Sed to rewrite the CGI/HTML files on the fly.

Looping Python code, no, no, no - never really got that to work.
Memory leaks? Don't know enough Linux/Python to fix it.
If I have to use Python I call it from script and make sure it exits.
Same with shell, no loops, let cron, watch do the timing.

Software loops use 100% cpu cycles, the "watch" method max out at 2% CPU usage on a PiB+.

[OO-Dragon]

This may be a little biased but I started a project last year called "Kootnet Sensors" to gather sensor data over long periods of time with Raspberry Pi's and it supports a bunch of sensors, including the EnviroPHAT and Enviro+.

It's decent on resources once it loads up, as I have it running on a Pi Zero W using about 3% CPU (Greatly dependant on how often you set it to record sensor data). It also saves all data into an SQLite3 Database for quick and portable data access. Graphing can be done right from the web interface or the "Control Center" software (Python3 GUI to interact with sensors).

Installation is easy, as you just run a single script and let it do all the work, you just enter in the username and password you want to use for accessing the Web Interface. It also creates shortcuts under the Accessories menu to launch the management functions. The recording and HTTPS management programs start right after install but require configuration of sensors (Tell it what sensors you have installed) before it will start to work with the sensors. It even enables everything the sensors needs, AKA you can load up a fresh install of Raspbian, run the install script, setup the installed sensors under the web management interface and everything just works.

Check the download page here http://kootenay-networks.com/?page_id=236 if you are interested.

The short version of installing it on the Raspberry Pi is as follows.
1. Install Raspbian on the Flask Card.
2. Physically install your sensor(s)
3. Connect to the internet and run the following in a terminal.

Code: Select all

wget http://kootenay-networks.com/utils/koot_net_sensors/Installers/raspbian/install_update_kootnet-sensors_http.sh && sudo bash install_update_kootnet-sensors_http.sh

If you do end up giving it a try, let me know! I haven't actually publicly posted this project yet and could use some feedback. It is released under the GPL, so feel free to look at the code on GitHub. You can find a link on the download page mentioned above.

[Andyroo]

One other option is to use io.adafruit to create a basic dashboard - it only stores a limited period of data on the free level but other solutions do exist.

A search of google for IoT dashboard will turn up lots.

Another option is to look at node-red to handle the data and dashboard needs.

[amcdonley]

For simple logging sensor data to a file and then plotting it, all local:


1) Log Battery Voltage and Plot on linear scale by hour of day, and by time since first measurement
2) Log PiCamera Brightness Value and Plot on log scale by hour of day


It is also possible to tie sensor readings into the RPi-Monitor code which will serve plots to a browser.

You also can go the IoT route by following the Rosie The Red Robot blog - great stuff but involved.

[xXAzazelXx]

Thank you all for your advice !!
I will have a look in to it all.
In the meantime i managed to write a python script to get all of the readings and put them in to a .csv file and use crontab to run every 5 minutes.
Not the best solution but I think its a good start

[bensimmo]

If you want to learn some Python3 techniques have a look at the SenseHAT datalogging tutorials on the main projects website.
You then slip in the Pimoroni code as needed.
After that there are some graphing tutorials there too.

And then you can move on to IoT based setups, say sending it to some external source. Sending it via MQTT to a broker.

Have fun and the Pimoroni people will help out as well.
Also feed back any problems to them, the Enviro+ is quite new and the code may need feedback on how people use it.

[kflmiami420]

xXAzazelXx did you get your enviro+ board logging ? What did you finally decide to use to log your sensors ?

[xXAzazelXx]

xXAzazelXx did you get your enviro+ board logging ? What did you finally decide to use to log your sensors ?

Hey man,

This is pretty cool, but its not too granular.

https://github.com/nophead/EnviroPlusWeb