JacobL
Posts: 76
Joined: Sun Apr 15, 2012 2:23 pm

Pi2 as NAT router/firewall?

Wed Aug 05, 2015 10:41 pm

Just got my Pi2 and a quick iperf test shows that it saturates the 100Mbps link, something that my Pi1 cannot do. Has anyone tried adding a second USB NIC and made a NAT router from a Pi2? What kind of performance does that give?

I'm considering adding 2 gigabit USB NICs, do you think that will let it break 100Mbps in routed bandwidth? What about USB3 vs USB2? I know I will not get anything from using USB3 directly, but the NICs will then still be usable for the next project. I am just wondering if I will get any compatibility issues, since mixing USB2 and USB3 seems to have triggered quite a few issues in various setups.

User avatar
mikronauts
Posts: 2722
Joined: Sat Jan 05, 2013 7:28 pm
Contact: Website

Re: Pi2 as NAT router/firewall?

Thu Aug 06, 2015 3:22 pm

I doubt you can break 100Mbps in routed bandwidth with two gig-e NIC's, however you may get close to 100Mbps routed using the on board NIC + USB NIC.

See http://www.mikronauts.com/raspberry-pi/ ... ent-howto/

for some gig-e vs on board nic benchmarks, and an explanation of why the single USB host port on the RPi's + onboard USB hub limits total bandwidth.
JacobL wrote:Just got my Pi2 and a quick iperf test shows that it saturates the 100Mbps link, something that my Pi1 cannot do. Has anyone tried adding a second USB NIC and made a NAT router from a Pi2? What kind of performance does that give?

I'm considering adding 2 gigabit USB NICs, do you think that will let it break 100Mbps in routed bandwidth? What about USB3 vs USB2? I know I will not get anything from using USB3 directly, but the NICs will then still be usable for the next project. I am just wondering if I will get any compatibility issues, since mixing USB2 and USB3 seems to have triggered quite a few issues in various setups.
http://Mikronauts.com - home of EZasPi, RoboPi, Pi Rtc Dio and Pi Jumper @Mikronauts on Twitter
Advanced Robotics, I/O expansion and prototyping boards for the Raspberry Pi

JacobL
Posts: 76
Joined: Sun Apr 15, 2012 2:23 pm

Re: Pi2 as NAT router/firewall?

Fri Sep 11, 2015 11:37 pm

Just got my Gig-E dongles, and at least part of the article matches the quick iperf test I did. I get about 170Mbps TCP Tx and about 200Mbps Rx. A simultaneous dualtest (iperf -d) gives 174Mbps Rx and 50Mbps Tx.

I ran atop during the test and it showed a single core being maxed out. For Rx it showed about 45% sys and 55% irq for this core. For Tx, the numbers for sys and irq are swapped. This suggests that the bottleneck is somewhere inside the Linux kernel, and driver optimisations might be possible, limited by the 480Mbps USB 2.0 speed + protocol overhead obviously. The components in this chain would be:
* Linux kernel generic IRQ framework
* USB host driver
* Linux kernel generic USB layer
* Driver for USB dongle. In my case, Startech.com USB 3.0 Gig-E adapter
* Linux kernel network layer

There is probably room for improvement in the drivers, but the work is tedious and probably needs GPIO debug. There might be possibilities for threading by breaking the path into a pipeline, but the large amount of time spent in the IRQ handlers severely limits the possibilities here, since the IRQ handlers must stay on a single core. Could be fun to look into though.

I will run another benchmark once I get the NAT router running and see what I can get. But there will be 3 cores practically unused from the USB/NIC layers, so I don't expect this will limit throughput much compared to the combined Rx/Tx throughput.

/Jacob

JacobL
Posts: 76
Joined: Sun Apr 15, 2012 2:23 pm

Re: Pi2 as NAT router/firewall?

Tue Sep 15, 2015 6:27 pm

And here are the routed numbers:

TCP single stream one direction: 75Mbps.

TCP simultaneous up/down: about 37-38Mbps each.

TCP down + 5Mbps UDP up: 71Mbps TCP down + 5Mbps UDP up. (I have 50/5 DSL, so the test is important to me)

UDP single stream one direction: 97Mbps.

I noticed that in this case, I have 100% time spent in IRQ context on a single core for all of the tests. It looks like the entire Linux network routing layer is run in IRQ context here.

Note how UDP performance is much higher than TCP. This suggests that the speed limit is a per-packet limit, since the many small TCP ACK packets influences the throughput much more than their size justifies.

Return to “Networking and servers”