RPi 3B+ gigabit ethernet bad download speeds.

[tkaiser]

Greater than 300 Mbit speeds both up and down using a gigabit USB dongle on a non-Pi computer with no retransmitted packets was also reported earlier on this thread, but I can't find that post anymore.

The post got censored by one of the moderators as usual. If you want to tell your users 200 Mbits/sec with tons of retransmits would be 'fine' then such reports showing the opposite are obviously something you don't want to have on 'your' forum?

But also as usual everything archived: https://forum.openmediavault.org/index. ... post171365

[jamesh]

Report so far.

We been testing this and trying various things to fix the issues in the office.

We have tried a Realtek USB dongle and see that it does not suffer the same speed issues, so that is an interesting datapoint - we know its possible!

What is interesting is that when I run test through the Gig switch on my desk I see no issues, if I run instead through the office network I do see the issue, so at least we can now replicate it. Turning on flow control on the switch apparently makes the problem go away as well.

Running through wireshark we are seeing that lan78xx driver is using standard sized packets which the USB dongle is using jumbo packets. Currently trying to figure out how to make the lan78xx use jumbo packets but no luck so far. Think we will need to get microchip to take a look.

I am off for a few days, until Wednesday next week. So will get back on it then.

[RPIgoodSBC]

ejolson dixit...

While 200 Mbit download may seem fine at first glance, these tests were performed using a low latency local area network. The retransmits have a much greater impact when transferring data across a high-latency connection to the internet.

Exactly.

All this is a consequence of the use of gigabit ethernet hanging from a USB 2.0 bus in the RPi that is not capable of supporting large flows of input data.
Or some miraculous kernel hacker is capable of a fix or it will be better or not to use the gigabit ethernet of the RPi3B + or to put it in 100 mode.

Code: Select all

sudo ethtool -s eth0 speed 100 duplex full

And good luck to everyone.

[jamesh]

Greater than 300 Mbit speeds both up and down using a gigabit USB dongle on a non-Pi computer with no retransmitted packets was also reported earlier on this thread, but I can't find that post anymore.

The post got censored by one of the moderators as usual. If you want to tell your users 200 Mbits/sec with tons of retransmits would be 'fine' then such reports showing the opposite are obviously something you don't want to have on 'your' forum?

But also as usual everything archived: https://forum.openmediavault.org/index. ... post171365

As usual? Cobblers. We hardly delete any posts on here, only if its spam, or the poster is stiring things up.

Anyway, we are not telling our users that the performance is 'ok'. At the moment, we have now replicated the low bandwidth issue and are actively looking in to it. Its now a bank holiday in the UK until Tuesday, so we will recommence work on it next week.

Even once fixed, I would expect that the transfer rate will be on average about 230Mbits/s or thereabouts, that's the speed I get without retransmits. The USB2 bus simply doesn't have the bandwidth to deal with much more than that and the chip does fire off a lot of interrupts which doesn't help.

The retransmits is possibly a red herring btw. The Gig USB adaptor we used produced about three times as many retries as the onboard, and yet still managed a much higher speed.

[jamesh]

ejolson dixit...

While 200 Mbit download may seem fine at first glance, these tests were performed using a low latency local area network. The retransmits have a much greater impact when transferring data across a high-latency connection to the internet.

Exactly.

All this is a consequence of the use of gigabit ethernet hanging from a USB 2.0 bus in the RPi that is not capable of supporting large flows of input data.
Or some miraculous kernel hacker is capable of a fix or it will be better or not to use the gigabit ethernet of the RPi3B + or to put it in 100 mode.

Adding the line to /etc/rc.local ...

Code: Select all

sudo ethtool -s eth0 speed 100 duplex full

And good luck to everyone.

Everyone knows that the USB2 bus is bandwidth limiting, but remember that the ethernet protocol has ways of dealing with it (flow control and pause frames I believe).

We don't need a miraculous kernel hacker - just some time to get to the bottom of the problem and probably some help from Microchip for the more obscure parts.

[RPIgoodSBC]

---> jamesh

Ok.

Can you put the way to boot on 100 mode that works and edit my last comment because I have tested now and is not working.

UPDATED:
People like me with problems NIC on 1000 mode can do this in:

/home/pi/.config/lxsession/LXDE-pi/autostart

and add this...

@sudo ethtool -s eth0 speed 100 duplex full

Next boot your RPi3B+ is going to start on 100 mode automatically.

[ejolson]

People like me with problems NIC on 1000 mode can do this in:

/home/pi/.config/lxsession/LXDE-pi/autostart

and add this...

@sudo ethtool -s eth0 speed 100 duplex full

While the obvious way to limit speeds to 100 Mbit is to use a four conductor cable with only two twisted pairs, I wonder what the best way in software is to limit speeds to 100Mbit.

Its now a bank holiday in the UK until Tuesday, so we will recommence work on it next week.

Woohoo! Also known as Easter.

[Bruny]

Code: Select all

sudo ethtool -s eth0 speed 100 duplex full

And good luck to everyone.

That tool also can be used to optimize rx ring.

TX/RX RAM Buffer

[mfa298]

Or some miraculous kernel hacker is capable of a fix or it will be better or not to use the gigabit ethernet of the RPi3B + or to put it in 100 mode.

Code: Select all

sudo ethtool -s eth0 speed 100 duplex full

So far for me the solution is to limit port speed on the switch to 100Mbit FD.

In both those cases unless you configure both side of the link manually you could get a mismatch (things should default to 100/Half if autonegotiate is turned off). Some managed switches will let you have autonegotiate but limit the speed (but the number of people with managed switches will be low, and the number of those that allow autonegotiate with fixed speed will be even lower).

In my case (testing PXE boot with NFS root) fixing the switch to 100/Full didn't work as the Pi came up in 100/Half and no way to change it on the Pi at that point. Setting the switch to 100/Half did work (as that's how the Pi Defaulted). I since found the magic incantation that fixes the speed and enabled auto negotiate.

I might try and do some tests a bit later to see if I can demonstrate what different types of settings do on both sides of the link. Similarly also a set of iperf tests if I can get a decent system to test against. I can't do sensible speedtest tests (my city location has worse internet options than many towns and villages). Although it might be possible to simulate something similar with some of the advanced linux kernel features.

While the obvious way to limit speeds to 100 Mbit is to use a four conductor cable with only two twisted pairs, I wonder what the best way in software is to limit speeds to 100Mbit.

Limiting the speed with a 2 pair cable doesn't work with the 3B+ plugged into a Gigabit switch, the ilnk just doesn't come up. Using a 2 pair cable with one end fixed to 100M works (but see comments above), similarly a 2 pair cable into a FastEthernet switch works. I.e. if it's going to try negotiating GigE then you need all 4 pairs, it doesn't seem to drop back to FastEthernet if 2 pairs are missing.

[ejolson]

I since found the magic incantation that fixes the speed and enabled auto negotiate.

Would you mind describing this magic incantation? I may need it soon.

[mfa298]

I since found the magic incantation that fixes the speed and enabled auto negotiate.

Would you mind describing this magic incantation? I may need it soon.

Unless you're using an ExtremeNetworks switch with XOS it probably won't help much but ...

Code: Select all

configure ports 13 auto on speed 100 

A very quick look on my netgear it doesn't seem to support that type of setup at all (only choice is 100/Half or 100/Full or Auto)

[resiakt]

I since found the magic incantation that fixes the speed and enabled auto negotiate.

Would you mind describing this magic incantation? I may need it soon.

Simply insert the following into /etc/rc.local

Code: Select all

/sbin/ethtool -s eth0 speed 100 duplex full autoneg on

Found it here: https://forum.openmediavault.org/index. ... post171474

[RPIgoodSBC]

UPDATED screenshots:
All the clues are on my screenshots.

viewtopic.php?f=63&t=208512&p=1294025#p1294025

The important point is that the problem appears using ethernet drivers that depend on the usb bus.
In the RPi3B + (any rpi) the ethernet depends on the usb bus.

Where the network works perfectly is in Tinker-OS 2.0.5 because the internal gigabit controller of the tinkerboard uses a kernel driver for gigabit ethernet that does not depend on the usb bus.

In Armbian the tinkerboard internal gigabit controller also works well but with somewhat less performance but in Raspbian the gigabit controller RPi3B+ (USB dependent do not forget) the performance is painful.Tinkerboard and RPi3B+ was tested on same net,cable,router-port,...

This leads us to see that the problem is in hanging gigabit ethernet USB bus, painful in Raspbian, better in Armbian over 130 Mbps download speeds speedtest-cli with my ISP and in Tinker-OS the same for being now kernel driver dependent on the usb bus.

But in Tinker-OS 2.0.5 with a real gigabit ethernet driver on a real bus everything works like silk with my ISP.

[jamesh]

UPDATED screenshots:
All the clues are on my screenshots.

viewtopic.php?f=63&t=208512&p=1294025#p1294025

The important point is that the problem appears using ethernet drivers that depend on the usb bus.
In the RPi3B + (any rpi) the ethernet depends on the usb bus.

Where the network works perfectly is in Tinker-OS 2.0.5 because the internal gigabit controller of the tinkerboard uses a kernel driver for gigabit ethernet that does not depend on the usb bus.

In Armbian the tinkerboard internal gigabit controller also works well but with somewhat less performance but in Raspbian the gigabit controller RPi3B+ (USB dependent do not forget) the performance is painful.Tinkerboard and RPi3B+ was tested on same net,cable,router-port,...

This leads us to see that the problem is in hanging gigabit ethernet USB bus, painful in Raspbian, better in Armbian over 130 Mbps download speeds speedtest-cli with my ISP and in Tinker-OS the same for being now kernel driver dependent on the usb bus.

But in Tinker-OS 2.0.5 with a real gigabit ethernet driver on a real bus everything works like silk with my ISP.

Yes, we know all this and are actively trying to sort it out.

It's not specifically USB bus issue though, since a USB attached GigE dongle does work at an acceptable speed in the same circumstances. Although as I said above, it's going to top out at 200-250Mbits/s maximum anyway due to downstream bandwidth issues (USB2 bus, interrupt quantity, SD card speed etc)

[beo_6]

Hello RpiForum,

I just received my new 3b+ today and I’m testing it out now.
With that I tried the ethernet speed and it resulted in that:

Code: Select all

[email protected]:~ $ speedtest-cli 
Retrieving speedtest.net configuration...
Testing from xxxxx GmbH & Co. KG (xxx.xxx.xxx.xxx)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Interoute VDC (Frankfurt) [0.65 km]: 17.125 ms
Testing download speed................................................................................
Download: 196.13 Mbit/s
Testing upload speed....................................................................................................
Upload: 145.97 Mbit/s

repeating it gives me the same results. I got a 200 Mbit/s down and 200 Mbit/s up FTTH connection. So, the results are as expected. The speedtest results match with my desktop.

It seems that they updated the driver version to 1.0.6.

Code: Select all

[email protected]:~ $ ethtool -i eth0
driver: lan78xx
version: 1.0.6
firmware-version: 
expansion-rom-version: 
bus-info: usb-3f980000.usb-1.1.1
supports-statistics: yes
supports-test: no
supports-eeprom-access: yes
supports-register-dump: no
supports-priv-flags: no

[jamesh]

Reports so far.

Been hitting the ethernet with a stick for the last few days, that stick being iperf3. I have been testing the onboard, and two different USB->GigE dongles.

The main conclusion we have come to - if your switch does NOT have flow control turned on, you will suffer a large drop in performance. This is due to the Pi being unable to service incoming data at gig speeds, and therefor a lot of retries are required for dropped frames.

Now, desktop switches have flow control on permanently, and managed switches appear to have it on by default. Big data centre switches sometimes have it turned off because there is a slight improvement in performance with it off, if all devices on the network can cope with the speed. But, if you are seeing really low speeds when downloading from the net, you really need to check your network devices to ensure flow control is turned on.

Now, that being said, we have seen some really good performance from a USB dongle with the r8153 chipset/driver. This consistently performances well above the speed of the onboard device even with flow control off. Investigating further, this simply appears to be a REALLY GOOD chip. It does a lot of offloading on to HW (for example, it appears it coalesces TCP packets from the standard 1400 size in to much larger lumps, massively reducing the load on the USB link and the CPU processing required), and that really seems to make a massive difference to overall performance. We are investigating whether we can get the Microchip devices to work in a similar way. I suspect that the amount of buffer memory on the chip is the real limiting case here. What is interesting is that this dongle, with flow control off, goes from 0 retries to over 10k when testing with iPerf over 10s. This is about 5 times that of the onboard device, but its performance is still very good....so its doing some really clever stuff which we haven't yet figured out.

Another dongle tried with an ax88179 (ASIX chipset) showed performance BELOW that off the onboard device in the same tests.

All devices tested showed degradation in performance when flow control is off.

Note, I am unable to test internet performance, since we have a max of 100Mbits/s in the office, and both the 3B and 3B+ hit that limit in our tests (well, I get mid 90's for both)

Finally, we are talking to Microchip to see what can be done to improve performance. It's their chip and their driver. However, we are still investigating.

[resiakt]

Now, that being said, we have seen some really good performance from a USB dongle with the r8153 chipset/driver.

Interesting. I already heard about this RTL8153 before. Here in this forum. Here in this thread. But you deleted it (see at the bottom of this archived version). Why exactly?

[ejolson]

Now, that being said, we have seen some really good performance from a USB dongle with the r8153 chipset/driver.

Interesting. I already heard about this RTL8153 before. Here in this forum. Here in this thread. But you deleted it (see at the bottom of this archived version). Why exactly?

What I find interesting about those deleted but archived tests with the RTL8153 is the fact that iperf indicated no retransmissions. Does the Pi 3B+ also experience zero retransmissions under the same conditions (when flow control is turned on) using the same dongle? Also, is flow control effective when downloading from a high-latency internet connection or only for local area networks?

[jamesh]

Now, that being said, we have seen some really good performance from a USB dongle with the r8153 chipset/driver.

Interesting. I already heard about this RTL8153 before. Here in this forum. Here in this thread. But you deleted it (see at the bottom of this archived version). Why exactly?

Why are you here? You been banned twice for being abusive and ignoring request to keep it civil, and yet you keep coming back, again with the snark. Odd.

Your post will have been deleted not because it contained useful information, but before it was either a) Abusive b) Snarky. Because we have one real rule here. Keep it civil, which you seen unable to do. Your banning was a reflection on YOU and the incivility of your posts, not on the content of them, or the moderators just being a bit mean.

[jamesh]

Now, that being said, we have seen some really good performance from a USB dongle with the r8153 chipset/driver.

Interesting. I already heard about this RTL8153 before. Here in this forum. Here in this thread. But you deleted it (see at the bottom of this archived version). Why exactly?

What I find interesting about those deleted but archived tests with the RTL8153 is the fact that iperf indicated no retransmissions. Does the Pi 3B+ also experience zero retransmissions under the same conditions (when flow control is turned on) using the same dongle? Also, is flow control effective when downloading from a high-latency internet connection or only for local area networks?

iperf3 with flow control on the switch turned OFF, showed many many retries - 5 times that of the onboard device. (10k vs 2k). With flow control turned ON, retries under all circumstances for all devices was minimal. The mentioned archived tests didn't go in to enough detail to really make any assessments - it's a straight forward performance test, which did not reflect the actual problem, which we believe is the lack of flow control. Nothing to do with eee, which is a different problem entirely.

The interesting thing is that even with 10k retries it was still impressively fast, so that chip is clearly doing something very clever at the firmware level. Of course, being at the firmware level, apart from the buffer coalescing mentioned above (which we saw by using a managed switch that could mirror ports activity, then plugging in and comparing results from the mirror with those seen on the Pi using wireshark; results which were surprisingly different) , we have no idea what might be going on to get that sort of throughput even under the harsh conditions of no flow control. Hopefully Microchip can be a little more clever, since they are the network experts.

[resiakt]

Your post will have been deleted not because it contained useful information, but before it was either a) Abusive b) Snarky.

Interesting. Which part was abusive or snarky?

Isn't it in reality what you've been bragging here about (explaining the censorship shit show going on here all the time): 'how they are so upset I kept deleting their posts referencing (with pictures) Orange Pi's'

This was indeed my first post you censored (at the bottom): https://archive.fo/6kzg0 (the post contained two bits of information that could be valuable to your users and one picture of a heatsink). In the post with the iperf3 numbers and testing both direct attached RTL8153 and 'Raspberry Pi situation' with the unavoidable USB hub in between I also used the magic words 'Orange Pi'. I guess that's what the censor triggered, right?

You sell millions of boards each year and fear the words 'Orange Pi'? Seriously?

Well, at least it's a great way to keep your forum 'clean' since people know they always have to fear censorship and better not waste their time writing posts here.

[rhildinger]

resiakt (tkaiser),

In all honesty, your posts are becoming childishly tiresome and pointless. What good information you are adding to the thread is being outweighed by your obsession over nonexistent censorship. Please, give it a rest...

-Robert

[Bruny]

iperf3 with flow control on the switch turned OFF, showed many many retries - 5 times that of the onboard device. (10k vs 2k). With flow control turned ON, retries under all circumstances for all devices was minimal.

I would also compare that with dongle connected to desktop USB 2.0 port and Pi Desktop vs other distro.

[mfa298]

The interesting thing is that even with 10k retries it was still impressively fast, so that chip is clearly doing something very clever at the firmware level. Of course, being at the firmware level, apart from the buffer coalescing mentioned above (which we saw by using a managed switch that could mirror ports activity, then plugging in and comparing results from the mirror with those seen on the Pi using wireshark; results which were surprisingly different) , we have no idea what might be going on to get that sort of throughput even under the harsh conditions of no flow control. Hopefully Microchip can be a little more clever, since they are the network experts.

Just want to say the effort you guys go to in tracking down bugs and trying to find fixes (or at least tell the suppliers what issues they need to fix) is appreciated. Having done plenty of server/networking stuff I know how much work can go into trawling through packet captures trying to decipher what may (or may not) be going on.

[ejolson]

What good information you are adding to the thread is being outweighed by your obsession over nonexistent censorship.

I am happy to have just learned how the design of a heatsink intended for passive use is different than one intended for use with a fan.

The use of online communications can result in trolls and flames. For possibly different reasons Facebook also leads to unhappiness. Hopefully the same is not true about Raspberry Pi. It has been observed that Pi is often used as a generic term for single board computer. While this signifies a tremendous market success, it also presents a difficulty. Just like IBM was unable to assert trademark on the term PC so the letter Pi appears difficult to trademark. Even so, this forum is specifically designed to answer questions about Raspberry Pi and not about other single board computers. While it can be useful to make comparisons, it is also important not to troll the moderators or to flame individual forum members.

At any rate, while important, such a discussion does not help solve Ethernet packet retransmissions or bad download speeds. I'll be connecting a Pi 3B+ directly to a Linux PC next week using a direct Ethernet cable no switch in between. While this is the opposite of a high-latency Internet connection, I'll post some iperf numbers and try some latency simulations using Linux netem.