View Webserver on Pi on Internet using cloud

[mesand]

Hello All,

I have encountered a lot of ambiguity around this topic and would love some clarification.

My objective:

Access a flask web server on my raspberry pi (the website reads dynamic sensor data and controls pumps) over the internet without using third party services like dataplicity or portforwarding. I would preferably like to use a cloud platform like google cloud or heroku.

My Questions:

What are my best options to accomplish this goal? I dont need a full explanation on what exactly to do. I just need someone to point me in the right direction. Ive heard I need to use MQTT to communicate with a cloud server, if this is truly what I need to do to view my webserver over the cloud? Can someone unpack this explanation for me a little? I assume I would setup an account with a cloud IOT service but what then? Any help would be incredible! Thank you.

[Heater]

One possible solution:

1) Get yourself a server instance on Google cloud service or the cloud provider of your choice. Just now we are trying out Digital Ocean and it works well.

2) Have that server run Linux. We use Debian, it's nice to have consistency over Pi, server, and other machines.

3) Create a VPN tunnel between your Pi and your cloud server instance using WireGuard https://www.wireguard.com/

4) Run the nginx web server on that cloud instance as well.

5) Have have that nginx server proxy all incoming HTTP(S) requests to the web server on your Pi. Which it can reach through the VPN tunnel. Nginx need not actually serve any content of it's own, it's just acting as a proxy passing requests to the Pi and passing the Pi's responses back the the clients.

With all this in place you are now effectively doing what DataPlicity does with it's "worm hole" feature.

Be sure to secure the thing by using HTTPS. For that you will need to register a domain name for your cloud server instance and get some certificates for your nginx server. Certs can be had for free from letsencrypt.org https://letsencrypt.org/

BUT, do you really need to have your web server on the Pi?

Instead you could put the web server part of your project on that cloud server. Have the sensor/control part of your project communicate with the web server part in the cloud via some messaging system. MQTT is popular, we use NATS https://nats.io

This more typically what people do. We generally run the nginx web server on the cloud server and have it proxy requests to "sub servers" that are running on the same machine, listening on different port numbers.

[mesand]

Great, I set up a virtual machine on linode. I transferred my flask webserver to it and installed all the dependencies I needed. Now I need the raspberry pi to send data to the linode server and input it on the web server. And if a button is pressed on the web server I would need a message sent to the pi to turn on a motor. So this is where MQTT would come in I assume. This is where get lost. I think I have to install paho or mosquito on both the pi and linode server. And whenever sensor data is read I have to send it with mqtt to the cloud, then I dont know what to do with it once it is sent to the pi.

[neilgl]

Maybe as a first step, get mqtt working on your pi using an external mqtt server (broker) i.e. mqtt.eclipse.org

Code: Select all

sudo apt install -y mosquitto mosquitto-clients

Then open two terminal windows on the pi. In the first subscribe to a topic

Code: Select all

mosquitto_sub -h mqtt.eclipse.org -t test/simple

In the second, publish a message from your pi to the mqtt server

Code: Select all

mosquitto_pub -h mqtt.eclipse.org -t test/simple -m testing

You should see "testing" appear see in the first window.

When that is working you can move on to python versions of the same, using the python library paho-mqtt, and the pi can control GPIO (LED, motor etc.) in a python script.
Then set up your own mqtt server on your linode server...

[mesand]

awesome thank you. So would my linode server be the broker? I would identify it by its IP address as the broker when using mqtt? Then the web server and raspberry pi are clients that publish and subscribe to topics(temp sensor, motion sensor, etc..)? I guess my main question is I have a server and it obviously has a specific ip address, the one I use to ssh into it with. Now when identify this server in the mqtt code do I simply refer to the server with its ip address?

[mesand]

Actually, I may use Google IoT core instead of linode since it may be easier.

[Heater]

mesand,

So would my linode server be the broker?

Yes.

You run an MQTT broker on your cloud server.

Then you Pi and your web server in the cloud, likely the same server, can communicate by publishing and subscribing to messages on that broker. You could have many Pi out here chatting through that same broker.

Mosquito is a nice and simple MQTT broker to use.

Your Pi and web server etc can all connect to that MQTT broker using whatever IP address your cloud server has. A better idea though is to register a domain name for yourself. You can do that cheaply with the likes if Godaddy: godaddy.com. Then your Pi and such connect using that domain name. If the IP address changes everything still works.

Of course you should remember to secure all of this. Otherwise anyone, anywhere can push messages into your MQTT broker and cause chaos.

That is a whole other topic....

[mesand]

I am grateful for your answers, you've clarified this greatly for me.

Yes I will dig in further on securing the mqtt communication. So far, my pi and server can communicate using certificates over ssh and I set up a firewall that only allows communication on the ssh port, so I will need to take further steps to secure the mqtt communication. My next step will be to allow communication on the SSL mqtt port on my linode server through the firewall. Then I will register the ip address on the server with a domain name like you said and use that domain to communicate with the pi over mqtt. Instead of doing this process with linode do you think it will be easier/more secure with using Google's IoT core to communicate with the pi and host my webserver on googles cloud?

[neilgl]

Did the simple test using mosquitto pub/sub work?

[mesand]

I ended up using a simple test with paho. One published a message to another that acts if the message is hello world. I did this when communicating with mosquitos server and it worked ok, a bit slow but it was communicating.

Now, I opened up my firewall on my server to communicate between the pi and the server both on port 1883. but I keep getting a connection refused error on my server.

Script 1

MQTT Publish Demo
Publish two messages, to two different topics

import paho.mqtt.publish as publish

publish.single("CoreElectronics/test", "Hello", hostname="linode ip")
publish.single("CoreElectronics/topic", "World!", hostname="linode ip")
print("Done")


Script 2
# MQTT Client demo
# Continuously monitor two different MQTT topics for data,
# check if the received data matches two predefined 'commands'

import paho.mqtt.client as mqtt

# The callback for when the client receives a CONNACK response from the server.
def on_connect(client, userdata, flags, rc):
print("Connected with result code "+str(rc))

# Subscribing in on_connect() - if we lose the connection and
# reconnect then subscriptions will be renewed.
client.subscribe("CoreElectronics/test")
client.subscribe("CoreElectronics/topic")

# The callback for when a PUBLISH message is received from the server.
def on_message(client, userdata, msg):
print(msg.topic+" "+str(msg.payload))

if msg.payload == "Hello":
print("Received message #1, do something")
# Do something


if msg.payload == "World!":
print("Received message #2, do something else")
# Do something else

# Create an MQTT client and attach our routines to it.
client = mqtt.Client()
client.on_connect = on_connect
client.on_message = on_message

client.connect("linode server", 1883, 60)

# Process network traffic and dispatch callbacks. This will also handle
# reconnecting. Check the documentation at
# https://github.com/eclipse/paho.mqtt.python
# for information on how to use other loop*() functions
client.loop_forever()

[Heater]

mesand,

do you think it will be easier/more secure with using Google's IoT core to communicate with the pi and host my webserver on googles cloud?

Whilst I have used Google cloud servers quite a lot in recent years I have no idea about their "IoT core".

Thing is I don't ever want to spend my time building something that is dependent on some particular service provider. I don't want to be tied to Google or Amazon AWS or MS Azure.

I want to be sure that I can move whatever I have somewhere else. If they have a cheaper deal, or a better service, or politics is such that I get cut off. Who knows.

As such, if any of the above, and anyone else, can offer a server instance that runs my Linux services I happy.

It's like the old days of not wanting to waste time writing programs that depended on Windows.

[mesand]

Yea exactly. That seems to be the trade off for the user interface they provide. I feel like I am pretty close to communicating via mqtt between my pi and the linode Linux server and may ride out the confusion and get acquainted. Linode advertises data privacy when compared to aws. Once I establish communication between the two, I already have a flask web server set up and will be able to fit the endpoints with mqtt so I can access it over the internet. When that day comes I am going to be so happy.

[neilgl]

OK perhaps you have "set up your own mqtt server on your linode server.." - no google IOT or AWS or Azure involved......

[mesand]

I ended up solving my issue. It as a firewall issue. Thank you all.

[Heater]

Excellent, well done.

I found it quite magical when I got my first cloud server running. I mean, you are installing your code onto a computer somewhere, God knows what it is or where it is, and there it is showing up on the internet.

I guess nobody sees that magic today, it's just expected.