Nev
Posts: 33
Joined: Fri Jan 16, 2015 7:09 pm

Raspberry pi weather station build your own project

Thu Oct 11, 2018 7:35 pm

I’ve successfully followed the above project using a raspberrypi 3 linked to nine sensors which saves the sensor data on a MariaDB every 10 minutes. This I’ve located in and on a shed at the bottom of the garden which has mains power from the house.
At the moment to read the data I have to physically go to the shed and connect to the pi.
My question is I have a second pi in the house, could I connect this to the shed pi using tplink power line connectors and pull the data from MariaDB and display it via the house pi on an lcd screen. If I do this over ssh do I risk interrupting the flow of data to MariaDB?
Also would it be better to set the house pi up as a web server with MariaDB installed and pull the data off the shed pi and then use my iPad as client? Or if I make the shed pi the server would I risk over working that one and slow everything down?
I would welcome any thoughts on this one before I make any changes to my existing working system.

PhatFil
Posts: 618
Joined: Thu Apr 13, 2017 3:55 pm

Re: Raspberry pi weather station build your own project

Thu Oct 11, 2018 7:55 pm

No you shouldn't interrupt data flow to your db with an occasional ssh login via your suggested network solution

though just out of interest how often/regularly are you logging results?

Also imho run all the servers you want, if your the only client getting served the demands wont be that high, if anything the bottleneck could be the powerline network connection..

Nev
Posts: 33
Joined: Fri Jan 16, 2015 7:09 pm

Re: Raspberry pi weather station build your own project

Thu Oct 11, 2018 8:09 pm

Thanks for that. I’m logging every 5 minutes at the moment but thinking about extending it to 10. What do you mean by imho?

PhatFil
Posts: 618
Joined: Thu Apr 13, 2017 3:55 pm

Re: Raspberry pi weather station build your own project

Thu Oct 11, 2018 11:15 pm

Nev wrote:
Thu Oct 11, 2018 8:09 pm
Thanks for that. I’m logging every 5 minutes at the moment but thinking about extending it to 10. What do you mean by imho?
shorthand for 'In My Humble Opinion' as i am only an enthusiast such as yourself..

every 5-10 minutes shouldn't be a burden on the system at all ;)

jbudd
Posts: 716
Joined: Mon Dec 16, 2013 10:23 am

Re: Raspberry pi weather station build your own project

Fri Oct 12, 2018 12:22 am

I like the setup where a machine or machines read the sensors, sending the data to a central server to store and display it.
That would mean that your database resides on your indoor Pi, the Pi in the shed sends it's sensor data at intervals.

I suppose that your shed Pi could write SQL insert statements to update the database. An alternative is to send the data as MQTT messages and update the database with code on the indoor Pi.

Of course you need a reliable data connection between the house and the shed, though MQTT has quality of service levels which can build in tolerance for poor connections.

pfletch101
Posts: 232
Joined: Sat Feb 24, 2018 4:09 am

Re: Raspberry pi weather station build your own project

Fri Oct 12, 2018 3:08 pm

PhatFil wrote:
Thu Oct 11, 2018 7:55 pm
No you shouldn't interrupt data flow to your db with an occasional ssh login via your suggested network solution
I don't understand this comment! One of the points of using a database manager (like MariaDB) running as a server in this sort of project is precisely that data inserted by one application can simultaneously (or nearly so) be read by another without the two client applications interfering with one another. I have a home energy monitoring and control system which works in precisely this way, with data coming from a Davis weather station written to a weather database by weewx and data coming from indoor temperature/humidity sensors written to a separate database by my own software. The two databases are then queried for the latest data using TCP servers running on the Pi whose clients are components of an application running on my main Windows system.

Nev
Posts: 33
Joined: Fri Jan 16, 2015 7:09 pm

Re: Raspberry pi weather station build your own project

Fri Oct 12, 2018 3:34 pm

Hi thanks for the input. I’ve just been reading up on MQTT and it sounds like the way forward. I assume home pi would be the server with LAMP on it and shed pi would send 8 different streams of data to it every 5 minutes? What are the main pitfalls with this setup, I’m bound to find them but forewarned is . . .?

B.Goode
Posts: 6544
Joined: Mon Sep 01, 2014 4:03 pm
Location: UK

Re: Raspberry pi weather station build your own project

Fri Oct 12, 2018 3:51 pm

pfletch101 wrote:
Fri Oct 12, 2018 3:08 pm
PhatFil wrote:
Thu Oct 11, 2018 7:55 pm
No you shouldn't interrupt data flow to your db with an occasional ssh login via your suggested network solution
I don't understand this comment! One of the points of using a database manager (like MariaDB) running as a server in this sort of project is precisely that data inserted by one application can simultaneously (or nearly so) be read by another without the two client applications interfering with one another.

So you appear to be in agreement with my understanding of the comment.

Perhaps it could be re-phrased as "an occasional ssh login via your suggested network solution should not interrupt data flow to your database."?

pfletch101
Posts: 232
Joined: Sat Feb 24, 2018 4:09 am

Re: Raspberry pi weather station build your own project

Fri Oct 12, 2018 4:00 pm

B.Goode wrote:
Fri Oct 12, 2018 3:51 pm
pfletch101 wrote:
Fri Oct 12, 2018 3:08 pm
PhatFil wrote:
Thu Oct 11, 2018 7:55 pm
No you shouldn't interrupt data flow to your db with an occasional ssh login via your suggested network solution
I don't understand this comment! One of the points of using a database manager (like MariaDB) running as a server in this sort of project is precisely that data inserted by one application can simultaneously (or nearly so) be read by another without the two client applications interfering with one another.

So you appear to be in agreement with my understanding of the comment.

Perhaps it could be re-phrased as "an occasional ssh login via your suggested network solution should not interrupt data flow to your database."?
Yes, I now see that the comment could have been (and quite possibly was!) intended to mean what your rephrasing of it actually says.

jbudd
Posts: 716
Joined: Mon Dec 16, 2013 10:23 am

Re: Raspberry pi weather station build your own project

Sat Oct 13, 2018 1:12 am

I’ve just been reading up on MQTT and it sounds like the way forward. I assume home pi would be the server with LAMP on it and shed pi would send 8 different streams of data to it every 5 minutes? What are the main pitfalls with this setup, I’m bound to find them but forewarned is . . .?
LAMP = Linux, Apache, MySQL, PHP?

The way I would do it, because I like the visual programming interface, would be Node-Red. Of course if you already have your sensors updating a database you probably don't need this:

On the Pi in the shed use Node-Red to request readings from the sensors every minute (or 5 or 10 minutes).
Node-Red sends the data by MQTT to an MQTT broker running on the indoor Pi with topic eg shed/temp , shed/humidity etc and a payload in JSON format including timestamp, sensor reading and units.

The indoor Pi also runs Node-Red which receives the data, stores it in a database and displays it on a web page.
There are nodes to connect to many databases.
Node-Red can write an HTML page to be served by Apache or NGINX. It also offers some dashboard nodes that let you create a web page without installing another web server.

So definitely L but AMP is optional!

There are quite a few articles online describing weather stations working like this.


Do you know if powerline networking will work between the circuit in the shed and the circuit indoors? Does mains wiring for outbuildings involve an isolation transformer?

Nev
Posts: 33
Joined: Fri Jan 16, 2015 7:09 pm

Re: Raspberry pi weather station build your own project

Sat Oct 13, 2018 3:27 pm

That’s really helpful, thank you! At the moment my python 3 script on the shed pi stores 5 minutes worth of data from the sensors in different data[ ] stores, works out things like average, max/min, modal values and then sends these readings to Mariadb on the shed py. The way I retrieve the data at the moment is to connect directly into the shed pi and view it on screen.
With node-RED do I ditch Mariadb on shed pi. Install node -RED and send the separate 8 sets of modified data to node-RED on the house pi. Would I need to install Mariadb on the house pi to work with node-RED or does it have its own dB? Also does node-RED have its own MQTT broker if not can you recommend one?
Finally I already use a power line adapter from the shed to my house modem and this serves a security camera which works very well. I would use a separate adapter for for the shed pi!
Thanks again.
Nev

PhatFil
Posts: 618
Joined: Thu Apr 13, 2017 3:55 pm

Re: Raspberry pi weather station build your own project

Sat Oct 13, 2018 5:32 pm

pfletch101 wrote:
Fri Oct 12, 2018 4:00 pm
B.Goode wrote:
Fri Oct 12, 2018 3:51 pm
pfletch101 wrote:
Fri Oct 12, 2018 3:08 pm

I don't understand this comment! One of the points of using a database manager (like MariaDB) running as a server in this sort of project is precisely that data inserted by one application can simultaneously (or nearly so) be read by another without the two client applications interfering with one another.

So you appear to be in agreement with my understanding of the comment.

Perhaps it could be re-phrased as "an occasional ssh login via your suggested network solution should not interrupt data flow to your database."?
Yes, I now see that the comment could have been (and quite possibly was!) intended to mean what your rephrasing of it actually says.
So we all agree then..

PhatFil
Posts: 618
Joined: Thu Apr 13, 2017 3:55 pm

Re: Raspberry pi weather station build your own project

Sat Oct 13, 2018 6:07 pm

Nev wrote:
Sat Oct 13, 2018 3:27 pm
That’s really helpful, thank you! At the moment my python 3 script on the shed pi stores 5 minutes worth of data from the sensors in different data[ ] stores, works out things like average, max/min, modal values and then sends these readings to Mariadb on the shed py. The way I retrieve the data at the moment is to connect directly into the shed pi and view it on screen.
With node-RED do I ditch Mariadb on shed pi. Install node -RED and send the separate 8 sets of modified data to node-RED on the house pi. Would I need to install Mariadb on the house pi to work with node-RED or does it have its own dB? Also does node-RED have its own MQTT broker if not can you recommend one?
Finally I already use a power line adapter from the shed to my house modem and this serves a security camera which works very well. I would use a separate adapter for for the shed pi!
Thanks again.
Nev
whatever seems to fit you best really, you can mod your python to add an additional mqtt publishing command or commands to publish the data as a set similar to your current sql insert statement ( i assume) (probably what i would do..) or publish it on reciept and move porcessing to node red (future enhancements? but if it aint broke??)

and you could on reciept of data via mqtt in node red have subflows to handle its database insertion there are contrib node for this purpose you can easily install via the node red palette

Node red is a tool to manipulate data flows not a database and it will run quite happily along side your database on the shed pi, and you will also need a Mqtt broker and hosting your own on the same pi is again no problem.


As for sharing the same ac powerline network connection as a security camera, i think the variables are such that referencing product limits and performance expectations will only tell you so much and a suck it and see test is the best way to see if you get any significant problems. Convenience may let you live with the odd lag on a web page load?.
with your system set up access it via the network connection and judge performance before and after the camera is also connected on the same link..

Your Pi runs a sophisticated modern os and even with a very busy multiplexed system your not going to loose data once its been collected from a sensor and buffered. And your server and network systems will only be taxed when accessed by clients with everything on the shed pi.

edit.. with everything on a singe pi tho you are going to want to consider a regular backup schedule too perhaps an overnight batch job for your 2nd pi?
Last edited by PhatFil on Sat Oct 13, 2018 6:18 pm, edited 1 time in total.

Nev
Posts: 33
Joined: Fri Jan 16, 2015 7:09 pm

Re: Raspberry pi weather station build your own project

Sat Oct 13, 2018 6:17 pm

Thank you very much guys, you’ve given me a lot to think about ( and learn )!
I’m off for the next two weeks so my iPad will get some hammer jenning up on all this, then I’ll give it a whirl when I get back.
Thanks again.
Regards
Nev.

jbudd
Posts: 716
Joined: Mon Dec 16, 2013 10:23 am

Re: Raspberry pi weather station build your own project

Sun Oct 14, 2018 6:10 am

If both Pi's are on the network then it doesn't really matter which one hosts your database.
To access Mariadb from Node-Red you have to install the MySQL node.

Node-Red can both send and read MQTT messages without installing anything extra but you do need to install a broker on at least one Pi. I use Mosquitto.

User avatar
DougieLawson
Posts: 34168
Joined: Sun Jun 16, 2013 11:19 pm
Location: Basingstoke, UK
Contact: Website

Re: Raspberry pi weather station build your own project

Sun Oct 14, 2018 8:26 am

You can host MariaDB on one RPi and run your SQL on the other.

The connect() defaults to localhost but can go remote.

To do that you need to change the bind-address to 0.0.0.0 on the server and add some extra GRANT'ed privileges to the user table.

https://stackoverflow.com/questions/156 ... sql-ubuntu
Note:The use of baseball bats for educational purposes is completely disallowed on this forum.

Any DMs sent on Twitter will be answered next month.

Return to “Beginners”