fredden68
Posts: 6
Joined: Thu Sep 29, 2016 6:16 am

Trying to use raspberry pi as router between local networks

Tue Jul 09, 2019 7:21 am

I am trying to use the pi as a router between networks 192.168.1.0/24 and 192.168.10.0/24. (I am setting up a lab environment where I need to control DHCP and not clash with the existing ISP provided router's dhcp.) I need to put DHCP on a server other than the 192.168.10.1 pi node, so am not using any dhcpd products on the pi.

I have found multiple instructions for setting up WLAN as a hotspot, and some for setting up as a router, and others as a bridge. But none of them work for a wired connection.

https://www.raspberrypi.org/documentati ... s-point.md
https://medium.com/@k1d_bl4ck/butterwor ... fbe547ab8b

I have a standard interface (eth0) and a USB plugin ethernet interface (eth1) The interfaces are set as (eth0) 192.168.1.5, and (eth1) 192.168.10.1. I have configured the ipforward in the kernel parameter for ipv4 and a custom route on my machines attempting to access the ip range 192.168.10.x ips as "route add 192.168.10.0 mask 255.255.255.0 192.168.1.5" and I can ping both interfaces from a 192.168.1.x IP.

I log onto the raspberry pi and I can ping addresses in both ranges, so I know the target address is correct for 192.168.10.x device.

However any attempt to ping a 192.168.10.x device (other than the router) from 192.168.1.x ends with a hang. There must be some concept about networking that is assumed but not being included in instructions; is it permissible to route between 2 ip ranges in 192.168.x.x? is there some arcane rule that is being enforced but not elaborated on (there's always one when dealing with networks)?

/etc/network/interface:

# interfaces(5) file used by ifup(8) and ifdown(8)

# Please note that this file is written to be used with dhcpcd
# For static IP, consult /etc/dhcpcd.conf and 'man dhcpcd.conf'

# Include files from /etc/network/interfaces.d:
source-directory /etc/network/interfaces.d
auto eth0
allow-hotplug eth0
iface eth0 inet static
address 192.168.1.5
netmask 255.255.255.0
gateway 192.168.1.1

auto eth1
iface eth1 inet static
address 192.168.10.1
netmask 255.255.255.0
gateway 192.168.1.1

ip addr:
# interfaces(5) file used by ifup(8) and ifdown(8)

# Please note that this file is written to be used with dhcpcd
# For static IP, consult /etc/dhcpcd.conf and 'man dhcpcd.conf'

# Include files from /etc/network/interfaces.d:
source-directory /etc/network/interfaces.d
auto eth0
allow-hotplug eth0
iface eth0 inet static
address 192.168.1.5
netmask 255.255.255.0
gateway 192.168.1.1

auto eth1
iface eth1 inet static
address 192.168.10.1
netmask 255.255.255.0
gateway 192.168.1.1

netstat -rn:
[email protected]:~# netstat -rn
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
0.0.0.0 192.168.1.1 0.0.0.0 UG 0 0 0 eth0
192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
192.168.10.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
[email protected]:~#

iptables:
# Generated by iptables-save v1.6.0 on Tue Jul 9 18:18:00 2019
*filter
:INPUT ACCEPT [11:714]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A FORWARD -i eth0 -o eth1 -j ACCEPT
-A FORWARD -i eth1 -o eth0 -j ACCEPT
-A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
COMMIT
# Completed on Tue Jul 9 18:18:00 2019
# Generated by iptables-save v1.6.0 on Tue Jul 9 18:18:00 2019
*nat
:PREROUTING ACCEPT [2:138]
:INPUT ACCEPT [2:138]
:OUTPUT ACCEPT [1:76]
:POSTROUTING ACCEPT [0:0]
-A POSTROUTING -o eth0 -j MASQUERADE
-A POSTROUTING -o eth1 -j MASQUERADE
COMMIT
# Completed on Tue Jul 9 18:18:00 2019
# Generated by iptables-save v1.6.0 on Tue Jul 9 18:18:00 2019
*mangle
:PREROUTING ACCEPT [377:61029]
:INPUT ACCEPT [375:60854]
:FORWARD ACCEPT [2:175]
:OUTPUT ACCEPT [65:5564]
:POSTROUTING ACCEPT [72:6323]
COMMIT
# Completed on Tue Jul 9 18:18:00 2019

ipforward:
net.ipv4.ip_forward

Regards, Fred

epoch1970
Posts: 3318
Joined: Thu May 05, 2016 9:33 am
Location: Paris, France

Re: Trying to use raspberry pi as router between local networks

Tue Jul 09, 2019 8:01 am

fredden68 wrote:
Tue Jul 09, 2019 7:21 am
address 192.168.10.1
gateway 192.168.1.1
By definition, the gateway sits at the border of the network. (192.168.10.x)
But that gateway is on another network, something’s amiss. (192.168.1.1)

Also, the “gateway” keyword commands the addition of a default route. In this case it fails because the gateway is unreachable (outside the network). But if it succeeded the system would have 2 default routes and would not know which one to choose from. You can add the metric keyword, eg “metric 100” to set route priority. Default metric is 0, highest priority.
Metric 100 would mean the default route via that interface is not to be used if the other exists.

I suggest you drop the firewall until you get routing setup right.

Assuming you’re using Raspbian, I do not really understand why you’re using the interfaces file when all the available documentation makes reference to configuring dhcpcd.conf
Migrate your interfaces file custom options into dhcpcd.conf équivalents or make sure to disable dhcpcd.conf on these two interfaces.

Look in the forum for details on any of the points above.
"S'il n'y a pas de solution, c'est qu'il n'y a pas de problème." Les Shadoks, J. Rouxel

fredden68
Posts: 6
Joined: Thu Sep 29, 2016 6:16 am

Re: Trying to use raspberry pi as router between local networks

Tue Jul 09, 2019 9:23 am

@epoch70:

Thank you for your contribution. I did try 192.168.10.1 as the gateway for ip 192.168.10.1 but got the same result. By the way, the "netstat -rn" command output shows that the default route is set correctly as "eth0" is 192.168.1.1 and being able to ping IPs on both networks from the Pi shows that both routes are working.

The reason I avoid /etc/dhcpcd.conf is that it is unreliable; I have attempted to set my dns server to 192.168.1.1 and it keeps resetting to 127.0.0.1. (See below)

(config)
# Example static IP configuration:
interface eth0
static ip_address=192.168.1.5/24
#static ip6_address=fd51:42f8:caae:d92e::ff/64
static routers=192.168.1.1
static domain_name_servers=192.168.1.1

interface eth1
static ip_address=192.168.10.1/24
#static ip6_address=fd51:42f8:caae:d92e::ff/64
static routers=192.168.10.1
# static domain_name_servers=192.168.1.1

(resolv.conf)
[email protected]:~# cat /etc/resolv.conf
# Generated by resolvconf
nameserver 127.0.0.1

I have connected my laptop up via the physical interface as 192.168.10.4 and proven the reverse, I can ping 192.168.10.1 as it's gateway but cannot get past it to the 192.168.1.1. So something in the pi is set wrong.

epoch1970
Posts: 3318
Joined: Thu May 05, 2016 9:33 am
Location: Paris, France

Re: Trying to use raspberry pi as router between local networks

Tue Jul 09, 2019 9:43 am

Funny because I don't see you try to set the dns servers from the interfaces file?

Routing, name resolution, filtering are 3 differents things. Take one at a time.
(the output you posted regarding ip forwarding is unconclusive. Querying sysctl or procfs should return 1.)

And again, if you're using interfaces you must ensure dhcpcd does not interfere. Add a "denyinterfaces" keyword to dhcpcd.conf.
"S'il n'y a pas de solution, c'est qu'il n'y a pas de problème." Les Shadoks, J. Rouxel

fredden68
Posts: 6
Joined: Thu Sep 29, 2016 6:16 am

Re: Trying to use raspberry pi as router between local networks

Tue Jul 09, 2019 9:56 am

@epoch70: I have now disabled the config in /etc/network/interfaces and enabled them in /etc/dhcpcd.conf (see below). Still no change.

[email protected]:~# grep -v "^#" /etc/dhcpcd.conf


hostname

clientid

persistent

option rapid_commit

option domain_name, domain_search, host_name
option classless_static_routes
option ntp_servers
option interface_mtu

require dhcp_server_identifier

slaac private

interface eth0
static ip_address=192.168.1.5/24
static routers=192.168.1.1
static domain_name_servers=192.168.1.1

interface eth1
static ip_address=192.168.10.1/24

So I do not require the "denyinterfaces" lines to be present.

You are correct the sysctl values should be present.
[email protected]:~# sysctl -a |grep -i forward |grep ipv4
sysctl: reading key "net.ipv6.conf.all.stable_secret"
sysctl: reading key "net.ipv6.conf.default.stable_secret"
sysctl: reading key "net.ipv6.conf.eth0.stable_secret"
sysctl: reading key "net.ipv6.conf.eth1.stable_secret"
sysctl: reading key "net.ipv6.conf.lo.stable_secret"
sysctl: reading key "net.ipv6.conf.wlan0.stable_secret"
net.ipv4.conf.all.bc_forwarding = 0
net.ipv4.conf.all.forwarding = 1
net.ipv4.conf.all.mc_forwarding = 0
net.ipv4.conf.default.bc_forwarding = 0
net.ipv4.conf.default.forwarding = 1
net.ipv4.conf.default.mc_forwarding = 0
net.ipv4.conf.eth0.bc_forwarding = 0
net.ipv4.conf.eth0.forwarding = 1
net.ipv4.conf.eth0.mc_forwarding = 0
net.ipv4.conf.eth1.bc_forwarding = 0
net.ipv4.conf.eth1.forwarding = 1
net.ipv4.conf.eth1.mc_forwarding = 0
net.ipv4.conf.lo.bc_forwarding = 0
net.ipv4.conf.lo.forwarding = 1
net.ipv4.conf.lo.mc_forwarding = 0
net.ipv4.conf.wlan0.bc_forwarding = 0
net.ipv4.conf.wlan0.forwarding = 1
net.ipv4.conf.wlan0.mc_forwarding = 0
net.ipv4.ip_forward = 1
net.ipv4.ip_forward_update_priority = 1
net.ipv4.ip_forward_use_pmtu = 0

DNS resolution config should be straight forward; either in the config file along with the IP Address or switch off Network management of the interface and add it in manually to /etc/resolv.conf. Nothing I have seen in the documentation here indicates a third option that needs to be over-ridden.

epoch1970
Posts: 3318
Joined: Thu May 05, 2016 9:33 am
Location: Paris, France

Re: Trying to use raspberry pi as router between local networks

Tue Jul 09, 2019 11:08 am

Ok. So I assume routing is correct on the Pi now. (To display a routing table run "ip route show" or "route" on linux machines)

Drop the firewall, masquerade disables routing since it manipulates the source address. "iptables -Z" "iptables -Z -t nat" "iptables -F" "iptables -F -t nat" on the Pi.

From any machine on 192.168.1.0 ping both interfaces of the Pi. From any machine on 192.168.10.0 ping both interfaces from the Pi. That should work if the Pi is setup correctly.

Next ping between 2 machines, one from each network. You can install the tcpdump package and use "sudo tcpdump -n -i eth0 icmp" (-i eth1) on the Pi to see pings flow, hopefully in both directions.
If it doesn't work properly, check that:
- Machines on 192.168.10.x show 192.168.10.1 as their default gateway and have no additional routes.
- Machines on 192.168.1.0 (at least the ones you're pinging from) have an additional route via 192.168.1.5 to 192.168.10.x.

Once that works.
If you want machines on 192.168.10.x to access the Internet, and you cannot set a static route to 192.168.10.x via the Pi 192.168.1.5 in the ISP router, then you do need Masquerade in the Pi.
However a blanket masquerade statement will now break routing/pings between the 2 internal networks as every response from 192.168.10.x hosts will appear as traffic from 192.168.1.5 (the main LAN interface of the Pi).
If you want both Internet access and explicit routing, exclude the main LAN from masquerading, i.e. add "! -d 192.168.1.0/24" to the rule.

HTH
"S'il n'y a pas de solution, c'est qu'il n'y a pas de problème." Les Shadoks, J. Rouxel

fredden68
Posts: 6
Joined: Thu Sep 29, 2016 6:16 am

Re: Trying to use raspberry pi as router between local networks

Wed Jul 10, 2019 11:05 pm

I have disabled iptables masquerading and setup hosts on each of different networks.

ping from 192.168.10.x to 192.168.1.x - confirmed working
ping from 192.168.1.x to 192.168.10.1 - confirmed working (router of 192.168.10.x)

ping from 192.168.10.4 to 192.168.10.5 - confirmed working (internal to new lan)

ping from 192.168.1.x to 192.168.10.5 - fails

I can connect to the internet from any host in 192.168.10.x (via 192.168.1.1) but I cannot connect back into to 192.168.10.x.

I can see icmp requests for the 192.168.10.x coming through eth0 (192.168.1.x) using tcpdump but they are not outgoing to eth1 (192.168.10.x)

So something in the OS is stomping on the routing out of these packets. What is it?

epoch1970
Posts: 3318
Joined: Thu May 05, 2016 9:33 am
Location: Paris, France

Re: Trying to use raspberry pi as router between local networks

Thu Jul 11, 2019 10:26 am

On Linux and roughly speaking, nothing is "stomping on the routing" other than the ip_forward tunable, routes setup and iptables rules.

Deciphering this last post is a bit difficult to me.
Some behaviours (succeeding or failing) seem as expected, some I am not so sure.

Confirm masquerading is disabled on the Pi and the firewall fully open as standard.
Check a static route is defined on the machines you're using in the main network to access the inner network (.10.x) managed by the Pi.
Internet access from the inner network is expected to fail until the ISP router has a static route to that network via the Pi, or the Pi has masquerade activated (with caveat, see above.)
Do reboot the machines in order to test from a clean state.
"S'il n'y a pas de solution, c'est qu'il n'y a pas de problème." Les Shadoks, J. Rouxel

fredden68
Posts: 6
Joined: Thu Sep 29, 2016 6:16 am

Re: Trying to use raspberry pi as router between local networks

Sun Jul 21, 2019 1:26 am

For reference purposes, I have conducted some tests and the issue is remote networking.

I recreated the entire layout (1 Linux machine in 192.168.1.x, 1 Linux machine with dual IPs in 192.169.1.x and 192.168.10.x, and 1 Linux machine in 192.168.10.x only) in a virtual environment using RHEL 7.6, and the equivalent Network configurations as provided, and it worked (traffic in both directions, web access and DNS across all systems)

I then replaced the Router VM with dual IPs with the physical Raspberry Pi and it also worked between networks 192.168.1.X and 192.168.10.x. However it could not talk out to the internet.

(here is the iptables code)
iptables -t nat -F
iptables -t nat -A POSTROUTING -s 192.168.10.0/24 ! -d 192.168.10.0/24 -j MASQUERADE
iptables -A FORWARD -i eth0 -o eth1 -j ACCEPT
iptables -A FORWARD -i eth1 -o eth0 -j ACCEPT

I then decided to eliminate the D-link remote access point from the equation by removing all my equipment from the back office and setting up in the lounge, physically cabled the Raspberry Pi into the back of the router. This is the configuration that worked. I removed the physical connection and replaced with a D-link remote access point and it stopped working (did this a few times to confirm).

So, if you have all physical connections, a "router" configuration with a Pi should work for spanning networks.

I know the "what" (remote networking) of the failure, but not the "why". I suspect that this will be a "bridge" situation (as I believe this is why my separate Redhat Server cabled to a D-link remote access point can run VMs on it with a different IP range. The device "vibr0" appears in the "ip addr" command output which seems to indicate a bridge device). Or there could be some obscure switch in Raspbian OS that says "allow this sort of traffic to go through".

I will do some more research and report back on a working config using a Pi for access between 2 networks using wireless networking. However at the moment I am using a dual IP VM as a jump host and any other vm that needs to talk out to the web gets a 192.168.1.x IP.

Regards, Fred

Return to “Networking and servers”