IGotDT
Posts: 9
Joined: Sat Apr 18, 2015 8:21 pm

redirect only a specific web address to local server

Fri Nov 01, 2019 11:26 pm

I am currently using a Pi3 as a Wi-Fi access point for a wireless weatherstation. The Pi uses tcpdump to capture the data that the weatherstation "phones home" and logs the weather data from that call. All is working great under normal circumstances. However, when the internet goes out apparently the web requests cannot complete so tcpdump never sees the "required" data it is looking for that contains the data I want.

What I would like to do is have the Pi push the requests for ONLY that specific domain to its own webserver where I can still provide the proper response the weatherstation is looking for but still get the data it is trying to send. I would then have the Pi make the request to the REAL webserver if the internet is up.

Any ideas? Thanks.

Heater
Posts: 15949
Joined: Tue Jul 17, 2012 3:02 pm

Re: redirect only a specific web address to local server

Sat Nov 02, 2019 2:19 am

What is this weather station exactly?

From your description that weather station is making some kind of connection to a server in the cloud, ""phones home". Is this something programmed by the manufacturer or something you have done?

If the former you could be a bit stuck. If there is no internet to the internet and it's home server then it cannot establish a TCP/IP connection and will never send out any data. Redirecting IP addresses will not help.

It's a bit worrying that such a device is not using HTTPS or other security but that is another matter...

To get that to work, you would need to create a fake home server for it to connect to, from which you could collect the data. You would probably need a fake DNS server to resolve the domain name it is connecting to to and actual IP address that is under your control.

Or you would need to hack the firmware in the device.
Memory in C++ is a leaky abstraction .

User avatar
DougieLawson
Posts: 39120
Joined: Sun Jun 16, 2013 11:19 pm
Location: A small cave in deepest darkest Basingstoke, UK
Contact: Website Twitter

Re: redirect only a specific web address to local server

Sat Nov 02, 2019 8:32 am

What webserver are you running?

ProxyPass is easy to use with Apache2 to do the redirection you need.
Note: Any requirement to use a crystal ball or mind reading will result in me ignoring your question.

Criticising any questions is banned on this forum.

Any DMs sent on Twitter will be answered next month.
All non-medical doctors are on my foes list.

Heater
Posts: 15949
Joined: Tue Jul 17, 2012 3:02 pm

Re: redirect only a specific web address to local server

Sat Nov 02, 2019 8:49 am

Yes but in order to do any ProxyPassing with Apache2 or nginx one first has to get the packets sent to that Apache or nginx.

The implication of the OP is that the weather station is attempting to connect to a server belonging to the manufacturer in the cloud.

On would have to spoof DNS to make it connect to ones own Apache or nginx address instead. Or perhaps do some funky address translation on outbound connections.

If one is unlucky it may then expect some conversation with it's home server, which one would have to emulate, before it offers up it's data.
Memory in C++ is a leaky abstraction .

IGotDT
Posts: 9
Joined: Sat Apr 18, 2015 8:21 pm

Re: redirect only a specific web address to local server

Sat Nov 02, 2019 11:01 am

The weather station is is an Ambient. The data that I am grabbing with tcpdump is in the get request. Would I not get that regardless of http or https, as it's in the url?
The problem I am having seems to be getting the station establishing a connection to a server (which is what I would like to emulate). I ran tshark when everything is normal (internet is working). The station sends its syn, receives the syn-ack, then makes its GET request, receives the ack, receives the response of "OK", and then the fin-acks to end the connection. When there is no internet, it doesn't get past the initial connection (for obvious reasons) and, therefore, does not make the get request. It does this with three weather collecting sites (ambient, wunderground, and some other i cannot remember) on a regular basis.
After a night's sleep, one solution I came up with is somehow changing the DNS to point to another Pi, have that Pi pretend to be one of the servers, and then it could process the data. And if there the internet is up, it could make the REAL request but I would get the data regardless of an internet connection.
I only have two problems with this solution. First, not sure how to change the DNS for only that one website, but I could probably figure it out if needed. Second, I would really like to limit this to just the one Pi. I already have a web of Pi's and really do not want to add another strand. ;)

Heater
Posts: 15949
Joined: Tue Jul 17, 2012 3:02 pm

Re: redirect only a specific web address to local server

Sat Nov 02, 2019 11:27 am

That is what I suggested. Mess with the DNS and spoof the domain name to give an address you control and can create a server to listen to.

But here is another idea.

1) Let's assume the domain name in question is someCloudServer.com and it resolves to an address A.B.C.D. As you can find from pinging someCloudServer.com

2) You can set up another Pi with the static address of A.B.C.D on it's ethernet port. And set it's subnet address to something like A.B.C.0.

3) We connect that Pi to your WiFi access point Pi via ethernet cable.

4) On the WiFi access point Pi we configure a static IP address on it's ethernet port A.B.C.Q (Something a bit different than our target address) and we configure a route on that ethernet port to the subnet A.B.C.0

Now we should have the situation that the weather station get's the usual address from DNS but when it uses that address it's connections get routed to your second Pi.

All you need to do now is put a web server on that second Pi that makes the appropriate responses.

Ah, now I remember. You can get a Pi to resolve a domain name to any address you like simply by adding an entry to the file /etc/hosts

123.45.65.2 www.somewhere.net

The steps above describe setting up the required fake address and subnet on a second Pi. This should also be possible to do on the same Pi. Not sure how off hand.
Memory in C++ is a leaky abstraction .

IGotDT
Posts: 9
Joined: Sat Apr 18, 2015 8:21 pm

Re: redirect only a specific web address to local server

Sat Nov 02, 2019 11:42 am

DOH! I totally forgot about the hosts file. I'll play around with that and report back. I think I have a solution.

User avatar
bomblord
Posts: 266
Joined: Sun Jul 14, 2019 2:54 am

Re: redirect only a specific web address to local server

Sat Nov 02, 2019 11:53 am

Might be easiest to setup a local DNS server using the Pi and use it as your secondary. If the primary DNS can't be reached because of an internet outage your secondary can be setup to redirect the desired URL to a local address just by putting in an entry for that url with the local IP.

Edit: Actually you don't even need a second pi you can set it up on the same pi and use the loopback address (127.0.0.1) as your secondary DNS.

IGotDT
Posts: 9
Joined: Sat Apr 18, 2015 8:21 pm

Re: redirect only a specific web address to local server

Sat Nov 02, 2019 12:15 pm

Awesomeness!
1> I added to the hosts files '127.0.0.1 rtupdate.wunderground.com'.
2> Made the directory and php file that is requested by the weatherstation in the Pi's web directory.
3> The php file echos back the expected response of 'OK', logs the data, and performs a curl to the real server using the IP address rather than the URL name.
4> The php exits silently if there is no internet but I still get my data, which is what I wanted.
5> Reboot the Pi... And test... Success!
Still need to do a little work to catch when the real server's IP changes, but that will be trivial.
Thanks @Heater for the nudge I needed!
@Bomblord That would have probably worked too. I'm already using PiHole on another Pi.

Heater
Posts: 15949
Joined: Tue Jul 17, 2012 3:02 pm

Re: redirect only a specific web address to local server

Sat Nov 02, 2019 12:24 pm

Cool. That was quick.

I just tried a similar experiment and got requests to someHostName.com responded to by my nginx server.

Of course all this will fail if the address of rtupdate.wunderground.com ever changes. Maybe that is not an issue for you.
Memory in C++ is a leaky abstraction .

IGotDT
Posts: 9
Joined: Sat Apr 18, 2015 8:21 pm

Re: redirect only a specific web address to local server

Sat Nov 02, 2019 12:36 pm

I did a python script on another Pi that pings 'rtupdate.wunderground.com' to get its IP. Then sends that to the original Pi so it knows the current IP. Cron job every 15 minutes. Problem solved. :D

Heater
Posts: 15949
Joined: Tue Jul 17, 2012 3:02 pm

Re: redirect only a specific web address to local server

Sat Nov 02, 2019 12:46 pm

Excellent. I love a good quick hack that works.
Memory in C++ is a leaky abstraction .

Return to “General discussion”