User avatar
azaram
Posts: 7
Joined: Sun Mar 10, 2013 1:52 pm

Controlling a Pi over the internet (without port forwarding)

Sun Mar 10, 2013 2:12 pm

Hi Guys,

This is a kind of embrassing question for me, as I've been building web/internet applications since the mid-90s.
However, I have a project where I'd like my pi-based devices to automatically call-home to a central cloud server. I also want them to be able to be sent commands from the cloud server.
All of the web applications I've ever built have had the client making all the requests to the server.
The commands I want to send are very time-sensitive, so a polling method wouldn't work. It would need to poll 5 times a second to get the responsiveness I need.

I have no idea how to do this. What I'm trying to achieve is probably similar to any other realtime client application that you run on your PC.. eg Skype, or Twitter (using their streaming API)
There are also other devices that seem to be able to perform this function. ie the new D-Link cameras and the myDLink service.

The ideal scenario for me would be:
- each device is coded with a unique ID for identification. This is printed on the device.
- when the device is connected to the internet, it automatically contacts the cloud server and says "here I am!"
- the device owner can "claim" their device, by entering the device ID, which then binds the device to their account.
- they can see their devices in their dashboard on the cloud server, and can issue commands to them when they are online.

How to do this?

Port-forwarding.
The simplest method I can think of is to have a web server running on the device, exposing a web service API. Obviously this method only works over the internet if you know the IP and port and the port is forwarded via the router the device is sitting behind.
This is not ideal for my setup. You don't setup port forwarding for skype, and it needs to be usable wherever you connect, potentially from different networks.

An open HTTP connection?
Maybe the device keeps open a HTTP connection to the server. The server can then issue commands via the HTTP response stream.
I've never done this, so I'm not sure if this method is advisable.
I guess the potential problem here is having a huge number of mostly redundant HTTP connections to the cloud server.

Any other ideas?

I'm guessing some sort of streaming protocol is what is called for here.

I should also point out that I'm a Microsoft developer, and I'd ideally be implementing this via Mono.

Thanks!

User avatar
MrBunsy
Posts: 185
Joined: Mon Feb 20, 2012 1:48 pm
Location: Southampton, UK
Contact: Website

Re: Controlling a Pi over the internet (without port forward

Sun Mar 10, 2013 2:44 pm

Sockets sound ideal for this - all you need is the port exposed on the server side, and a permanent IP or domain for the server. Once your client pi fires up, it connects to this server and you've got two-way communication to your pi without port forwarding.

Look into TCP or UDP, there's plenty of stuff out there which should be helpful.

Twinkletoes
Posts: 210
Joined: Fri May 25, 2012 9:44 pm

Re: Controlling a Pi over the internet (without port forward

Sun Mar 10, 2013 3:30 pm

Your limitation is going to be the open sockets on the server. Make sure you don't have a thread per socket!

User avatar
azaram
Posts: 7
Joined: Sun Mar 10, 2013 1:52 pm

Re: Controlling a Pi over the internet (without port forward

Sun Mar 10, 2013 3:44 pm

I found a great post here:
http://www.leggetter.co.uk/real-time-we ... gies-guide

It references a bunch of techniques for achieving the realtime interactivity that I need.
The interesting part is that there are a bunch of 3rd party realtime API providers that can handle it for you so you can decouple your application from the realtime operations and focus on the functionality.
Pusher and Realtime.co look interesting.
This sounds like a good starting point to get realtime functionality without reinventing the wheel.

The article also mentions a good self-hosted .NET solution - Alchemy Websockets.

My direction is starting to get a bit clearer.
I just need to make some decisions about:
- do I use a 3rd party?
- host myself and setup a second realtime push server on port 80?
- how do I make it future-proof so that changes to the protocols or services used can be gracefully picked up by the client devices.

User avatar
azaram
Posts: 7
Joined: Sun Mar 10, 2013 1:52 pm

Re: Controlling a Pi over the internet (without port forward

Sun Mar 10, 2013 3:52 pm

BTW, I should note that using anything other that HTTP/port 80 makes me nervous.
I have no control over what firewalls and weird corporate networks my devices might end up in, and HTTP/80 is the only method I feel confident will get through.
Even sockets on port 80 may get blocked via packet inspection.

It seems like the pubsub style methods of HTTP communication are the direction I'm heading now.

User avatar
MrBunsy
Posts: 185
Joined: Mon Feb 20, 2012 1:48 pm
Location: Southampton, UK
Contact: Website

Re: Controlling a Pi over the internet (without port forward

Sun Mar 10, 2013 3:58 pm

azaram wrote:I found a great post here:
http://www.leggetter.co.uk/real-time-we ... gies-guide

It references a bunch of techniques for achieving the realtime interactivity that I need.
I'm a little confused now as to what you're trying to do - those all look to be for providing applications running in a web browser with socket-like communications. If you want something running on the Pi that connects to a centralised server, you don't need anything like that - pretty much every programming language will have socket support built-in, and you can use that.

If you write the protocol yourself, you can just be careful and make sure you sort out your own future proofing. Sending JSON over the socket could be a fairly nice way of doing it - you could even have a piece of data that specifies the version of the protocol that's being used.

edit: oops, you've posted since then. I suppose sending something over TCP that looks like a long-polling HTTP request could work.

User avatar
azaram
Posts: 7
Joined: Sun Mar 10, 2013 1:52 pm

Re: Controlling a Pi over the internet (without port forward

Sun Mar 10, 2013 4:49 pm

Thanks MrBunsy, but I'm pretty sure I won't be using sockets now.
I feel much more comfortable with a publish/subscribe HTTP messaging model, partly due to the firewall restrictions I mentioned previously, but also because it's more in-line with the modern concepts of cloud app communications and REST APIs.
(eg try telling a 3rd party application developer building a javascript in-browser app that he needs to use sockets to connect to my system).

I'll also be exposing a web API in addition to the push services, but using the HTTP methods will keep everything more consistent.

I'm pretty sure I have my solution for now. Looking more into those services like Pusher, PubNub and Realtime.co, they are impressive. I love that they allow me to decouple my clients and server, and it also means I can rely on someone else to provide good security over the communications rather than trying to implement it myself. Actually, I'll probably still implement my own internal message security, so I'll have multiple levels of encryption (important for my application).

User avatar
crackers
Posts: 55
Joined: Fri Dec 28, 2012 7:45 am

Re: Controlling a Pi over the internet (without port forward

Sun Mar 10, 2013 6:23 pm

Have you considered using something like Jabber/XMPP? I've got my home-brewed home automation system logged into jabber.org so I can get status and make remote commands.

User avatar
azaram
Posts: 7
Joined: Sun Mar 10, 2013 1:52 pm

Re: Controlling a Pi over the internet (without port forward

Mon Mar 11, 2013 1:04 am

Thanks for the suggestion crackers.
I was was trying to think of the name of that protocol. Some friends had used it for a messaging application.
Without looking into to it too much, it seems that Jabber/XMPP uses a distributed server system with no authoritative servers. I'm not sure of the security implications of this with my system. Encryption and security are important for my specific application, so I think I'm still leaning towards the other third party systems at this stage.
I do like that it's a decoupled messaging system though, and being a standard means that I will have a choice of servers/implementations without changing code significantly - can I just advise clients of a new server to use for communication.
Definitely food for thought.
Last edited by azaram on Tue Mar 12, 2013 4:50 pm, edited 2 times in total.

-rst-
Posts: 1316
Joined: Thu Nov 01, 2012 12:12 pm
Location: Dublin, Ireland

Re: Controlling a Pi over the internet (without port forward

Mon Mar 11, 2013 3:42 pm

azaram wrote:Thanks MrBunsy, but I'm pretty sure I won't be using sockets now.
I feel much more comfortable with a publish/subscribe HTTP messaging model, ... modern concepts of cloud app communications and REST APIs.
(eg try telling a 3rd party application developer building a javascript in-browser app that he needs to use sockets to connect to my system).
...
Hmm, HTTP is an application protocol that 'runs' on top of a transport protocol, which in most cases is TCP, which runs through a - surprise,surprise - socket. In the most typical setup the web browser opens the socket for the tcp connection and then passes the http stuff back and forth... So I am pretty sure, you end up using the sockets anyway ;)

It is of course better to design the application protocol on a higher level, but you need the underlying transport layer and the guys here only pointed out how to get your RasPi client to connect to the server on that layer... 'Socket programming' admittedly sounds like a low-level dinosaur from the early years of UNIX networks, but as you see on the page you linked : 'WebSocket' is mentioned there in many of the tools - it is just the same technique with some additional stuff on top. And yes, you can open a socket to port 80 (HTTP) from any low-level socket code too...
http://raspberrycompote.blogspot.com/ - Low-level graphics and 'Coding Gold Dust'

User avatar
azaram
Posts: 7
Joined: Sun Mar 10, 2013 1:52 pm

Re: Controlling a Pi over the internet (without port forward

Mon Mar 11, 2013 3:56 pm

Thanks,

I know what sockets are.. and by saying I won't be using sockets, I mean I won't be using raw sockets.
I wasn't trying to criticise sockets, so I'm not sure why you're springing to their defense.

I've just found a good solution that solves my problems:
- doesn't require port forwarding
- doesn't require a sustained connection to my server
- doesn't require me to reinvent any wheels or create my own protocol
- doesn't require me to implement my own security
- allows asynchronous operation
- is almost guaranteed to work on any given network,even via SPI firewalls who might otherwise block non-http traffic on port 80.

Sockets definitely have their place, but I need something more sophisticated, reliable and simpler for my application.

lofty40
Posts: 1
Joined: Sun Mar 03, 2013 10:03 pm

Re: Controlling a Pi over the internet (without port forward

Wed Mar 20, 2013 10:10 am

Hi Azaram,

I am also looking for a solution to your problem.
Can you share/give more information about your solution ?

ramjam
Posts: 3
Joined: Wed Sep 26, 2012 2:58 pm

Re: Controlling a Pi over the internet (without port forward

Wed Mar 20, 2013 11:46 am

Have you tried LogMeIn Hamachi? It works perfectly through corporate firewalls at least the one I'm behind), and there are guides around. Installing is a bit special for the Raspberry Pi, one needs to include 'force architecture'. The guide I followed was on this site somewhere. Here's one: http://www.raspberrypi.org/phpBB3/viewt ... In+Hamachi

kaantez
Posts: 1
Joined: Tue Oct 08, 2013 8:06 pm

Re: Controlling a Pi over the internet (without port forward

Tue Oct 08, 2013 8:09 pm

Hi,

I also need this kind of solution. Could you please share the solution you found.

Corez
Posts: 31
Joined: Fri Feb 03, 2012 12:59 am

Re: Controlling a Pi over the internet (without port forward

Wed Oct 30, 2013 11:29 pm

This may not really be the solution you are after but it has worked for my in the past. I had a device that was behind a firewall I had no control over so couldn't open any ports to the device directly. The way I got around this is a reverse SSH tunnel (http://www.howtoforge.com/reverse-ssh-tunneling). Basically you setup your Pi to connect to your remote server over SSH and start listening to a port on the remote server.

For example the Pi would run this command:

Code: Select all

ssh -R 19999:localhost:22 cloud-server-user@cloud.server.com
Then on the cloud server I could run this command:

Code: Select all

ssh piuser@localhost -p 19999
I would then be using the SSH terminal on the Pi so you can run any command you like. 19999 is the port that is being listened to on the cloud server, localhost:22 is the address and port that any requests on 19999 will get redirect to.

If you setup your Pis so that each Pi only used one port on your server each you could run a netstat command on the cloud server to see which Pis are online.

However, I am not sure this fits under your "no port forwarding" rule as this is technically exactly what this just backwards (as my head is now because I only just understand this myself!)

thomasamberg
Posts: 7
Joined: Thu May 02, 2013 10:01 am

Re: Controlling a Pi over the internet (without port forward

Fri Nov 22, 2013 1:20 pm

Hi azaram,

sorry for the late reply.

Our project Yaler might help solve your problem in a simple and scalable way. Here's how:
- each device is coded with a unique ID for identification. This is printed on the device.
Yaler allows devices to register a relay domain and be accessed at a well known, location independent URL.
- when the device is connected to the internet, it automatically contacts the cloud server and says "here I am!"
Just install the YalerTunnel command line tool as a service on the Raspi, to enable Web or SSH access. It forwards incoming requests to the services listening to localhost on the Raspi.
- the device owner can "claim" their device, by entering the device ID, which then binds the device to their account.
This can be done, but would probably involve some integration work between your Web servers and our backend.
- they can see their devices in their dashboard on the cloud server, and can issue commands to them when they are online.
This can be done with a front end Web server that accesses individual devices. A simpler solution would be to directly access Web services or SSH running on the individual (Raspi) devices.
How to do this? An open HTTP connection?
This is exactly what Yaler does. Building a relay server can be done in a weekend, but making it robust, highly available and scalable took us quite some time and effort.
I should also point out that I'm a Microsoft developer, and I'd ideally be implementing this via Mono.
Our .NET library has been tested with Mono, but on Raspi using the YalerTunnel command line tool is recommended.

Btw. YalerTunnel and our libraries are open source. And the Yaler relay server source is available for non-commercial use.

Please see https://yaler.net/ for details.

Kind regards,
Thomas

gresham
Posts: 4
Joined: Tue Jun 24, 2014 10:14 am

Re: Controlling a Pi over the internet (without port forward

Tue Jun 24, 2014 10:17 am

Apologies for bringing up an old post but I have the exact same requirement as Azaram.
I've just found a good solution that solves my problems:
Can you please give me some direction as to how you were able to get your working solution ?

Thanks

mitchh22
Posts: 3
Joined: Sun Dec 28, 2014 11:57 pm

Re: Controlling a Pi over the internet (without port forward

Mon Dec 29, 2014 12:05 am

Hi azaram,

Would you please share your findings on what you ended up using?

User avatar
Richard-TX
Posts: 1549
Joined: Tue May 28, 2013 3:24 pm
Location: North Texas

Re: Controlling a Pi over the internet (without port forward

Mon Dec 29, 2014 3:22 am

azaram wrote:I found a great post here:
http://www.leggetter.co.uk/real-time-we ... gies-guide

I
Time sensitive/low latency software and Java are mutually exclusive.
Richard
Doing Unix since 1985.
The 9-25-2013 image of Wheezy can be found at:
http://downloads.raspberrypi.org/raspbian/images/raspbian-2013-09-27/2013-09-25-wheezy-raspbian.zip

User avatar
iinnovations
Posts: 621
Joined: Thu Jun 06, 2013 5:17 pm

Re: Controlling a Pi over the internet (without port forward

Mon Dec 29, 2014 7:22 am

+1 for LogMeIn Hamachi for large, distributed, secure mesh or gateway networks. I use it in the day job quite a bit. It solves mucho problems, security not being the least.

I also use it at home and while working on the go. My media/web server is a gateway and lets me access all of my pis and other development projects wherever I go as if they were local. I don't even need to install hamachi on the other devices on my home network.

C
CuPID Controls :: Open Source browser-based sensor and device control
interfaceinnovations.org/cupidcontrols.html
cupidcontrols.com

User avatar
aTao
Posts: 1093
Joined: Wed Dec 12, 2012 10:41 am
Location: Howlin Eigg

Re: Controlling a Pi over the internet (without port forward

Mon Dec 29, 2014 8:21 am

Richard-TX wrote:
azaram wrote:I found a great post here:
http://www.leggetter.co.uk/real-time-we ... gies-guide

I
Time sensitive/low latency software and Java are mutually exclusive.
To go right back to the OP.

Poll 5 times a second: Nothing on the web can be relied upon to be that responsive when in a idle state. Skype was mentioned, when idle it will poll much slower or (probably) use an open request that is refreshed every minute or so.

Client issues request then waits a while (dependent on how long any NAT routers will maintain the required information [packet ID -> local address]) before issuing another request.
The server on receipt of a request will do nothing until there is information to forward or another request is received (from the same client) when it should respond to the previous request to "close" it.

as for java and speed/response then the current myth that java is not suitable is not understanding JIT compilation. Its a pain in the neck but can be worked round, all you need to do is run the code (all of it) once while the JIT compiler does is bit, then java is plenty fast enough and responsive enough.
>)))'><'(((<

Return to “General programming discussion”