RPi 3B+ gigabit ethernet bad download speeds.

[RPIgoodSBC]

I wasn't expecting much change in overall speed performance, I've not seen improvements by turning off eee. What I have seen is that the number of drop outs went from 5 or 6 per 1.5GB transfer down to zero. So I am expecting the link to be more robust, but not faster.

When we have tested for speed in the office, we definitely see a major improvement on internal transfers for the 3B+, but we cannot check internet speed because our office link is capped at 100Mbits/s, so both the B and B+ hit about the same speed.

And a quick lesson on definitions. The Linux kernel is monolithic, this means that drivers are inherently part of the kernel (even if loaded as a module, but the lan78xx is built in anyway), so there is no real distinction between being a kernel problem and a driver problem. A driver problem IS a kernel problem.

Ok.Learning.

The very odd is that gigabit NIC on RPi3B+ performs better on 100 mode than on 1000 mode.

sudo ethtool -s eth0 speed 100 duplex full

100 mode is about 80 Mbits/s in my ISP and 1000 mode is about 40 Mbits/s.

This can only see people with about 200 ISP download speeds and up and when testing.
Very good ISP connections can disguise the problem. Better ping and much download speed less you see the loss.
The old RPi3B not plus has a different kernel driver and is 10/100 card but have the same problem that RPi3B+ on iperf3.
All this points to a Raspbian kernel problem on pi.

[jamesh]

So, just for reference, and of not much use since our link is capped at 100Mb/s, I get upload 90Mb/s, uploads 87Mb/s.

When I do an rsync from Ubuntu box to Pi via local network (just one switch) actually hit the limits of the SD card bandwidth which slows the network transfer down, but occasionally get up to 25MB/s (200Mbits/s) or more. Going to try a different mechanism for real world copying so I can dump the results to null to avoid that.

But in precis I'm just not seeing any major speed issues with the 3B+ yet.

[Bruny]

Does anyone know another linux system that can run on RPi3B + with ethernet enabled so I can try it out and see if the problem is the raspbian ethernet kernel driver?

They are ready from day one. There is example of UDP test with bandwidth flag.

Now on RPi3B+
iperf3 -c YYY.YYY.Y.YYY ...Now your PC is receiving and retr column must be 0 on a good network.

What about public servers?

[jamesh]

Curious.

Using netcat to transfer from an Ubuntu PC to the Pi,

Code: Select all

yes | pv -rab | nc [ip of pi] 4444

and running this on the Pi

Code: Select all

nc -l 4444 > /dev/null 

Gives me an average throughput of 27.5 MB/s, which is 220Mbits/s.

Which seem fine to me. Can you try it to see what happens?

[RaTTuS]

^ i get the same - though for some reason I cannot go the other way it exits after 64k

[ejolson]

Now let's simulate RPi behaviour and put an USB hub in between

Interesting idea to insert a USB hub in between. It is possible that packet loss in the 3B+ is occurring at the hub level. The built-in USB hub being responsible for the packet loss is further suggested by the fact that similar behaviour is observed by the person making the original post when using an external gigabit USB device. However, a marginal networking cable might explain the same thing.

This article describes the difference between single-TT and multi-TT hubs and how one type can lead to packet loss when multiple devices are connected. Note that all USB hubs based on the Genesys GL850 chipset dropped packets. You might consider connecting a mouse, keyboard and thumb drive to the USB hub and see if that decreases the reliability of the Ethernet dongle. What happens if the mouse and thumb drive are in use by a different process when the iperf test is being performed?

[el_grappaduro]

@ejolson seem you won't get an answer: https://forum.openmediavault.org/index. ... post171310

[ejolson]

@ejolson seem you won't get an answer

That's too bad. I think permanent bans can be damaging to the forum and to the people who are banned. A couple years ago there was an enthusiastic young member who identified performance issues related to an interrupt bug in the Bluetooth driver. I think he or she was permanently banned for something in the off-topic section. They subsequently made a few paranoid posts from a different account then never came back. I expect that person could make a positive contribution to this thread if still around.

[jamesh]

Wasn't me doing the banning this time...

It's all very well knowing your stuff, but presentation matters. I have had about 10 different reports on this posters posts (more than anyone for years), and was wasting a lot of time dealing with the fallout. It a shame that some clearly intelligent people are unable to behave in a civilised manner, even when asked to, and seem to think that when they do come back after a temporary ban, that being sarcastic and having a go at the mods is a good idea.

Sometimes you just have to bite the bullet and in the words of Elsa, just Let it Go. It's not as if there is a lack of smart people both in the office and on this forum to step up to the plate and fill any gaps.

[RPIgoodSBC]

---> ejolson

I have limited time, now can do now can not.
Here a few test:

Non stress on usb buses test iperf3:

- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 265 MBytes 222 Mbits/sec 3497 sender
[ 4] 0.00-10.00 sec 265 MBytes 222 Mbits/sec receiver

iperf Done.

1...Stress test writing to RPi3B+ host raspbian Sandisk usb extreme 3.0
2...And at the same time to external HD-2.5"-1Tb on a hub belkin 4 ports
3...And at the same time running iperf3 test.

Terminal 1:

sudo dd if=/dev/zero of=/tmp/testfile bs=2000k count=1k && sync ; sudo rm -f /tmp/testfile

Terminal 2:

sudo dd if=/dev/zero of=/media/pi/512-EXT4/testfile bs=2000k count=1k && sync ; sudo rm -f /media/pi/512-EXT4/testfile

Terminal 3:

sudo iperf3 -c XXX.XXX.X.XXX -R

Results:

- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 132 MBytes 111 Mbits/sec 2756 sender
[ 4] 0.00-10.00 sec 132 MBytes 111 Mbits/sec receiver

iperf Done.

------------------
Hd external 1Tb:

[email protected]:/media/pi/512-EXT4 $ sudo dd if=/dev/zero of=/media/pi/512-EXT4/testfile bs=2000k count=1k && sync ; sudo rm -f /media/pi/512-EXT4/testfile
1024+0 registros leídos
1024+0 registros escritos
2097152000 bytes (2,1 GB, 2,0 GiB) copied, 109,785 s, 19,1 MB/s


Sandisk extreme usb 3.0 usb host raspbian system:

[email protected]:~ $ sudo dd if=/dev/zero of=/tmp/testfile bs=2000k count=1k && sync ; sudo rm -f /tmp/testfile
1024+0 registros leídos
1024+0 registros escritos
2097152000 bytes (2,1 GB, 2,0 GiB) copied, 111,432 s, 18,8 MB/s
---
The Retr column is a very high numbers stress or not...about 2700/3400 retransmissions.

[RPIgoodSBC]

Curious.

Using netcat to transfer from an Ubuntu PC to the Pi,

Code: Select all

yes | pv -rab | nc [ip of pi] 4444

and running this on the Pi

Code: Select all

nc -l 4444 > /dev/null 

Gives me an average throughput of 27.5 MB/s, which is 220Mbits/s.

Which seem fine to me. Can you try it to see what happens?

About 220Mbits/s receiving it implies retransmissions.
See my iperf3 output:
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 265 MBytes 222 Mbits/sec 3497 sender
[ 4] 0.00-10.00 sec 265 MBytes 222 Mbits/sec receiver
---------------------------------------------
You must have about 300 Mbits/sec receiving.

This is sending... and sending on pi is OK

[email protected]:~ $ sudo iperf3 -c XXX.XXX.X.XXX ... sending from pi to my big PC

Connecting to host big PC, port 5201
[ 4] local RPi3B+ port 54914 connected to big PC port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 37.9 MBytes 318 Mbits/sec 0 257 KBytes
[ 4] 1.00-2.00 sec 37.7 MBytes 316 Mbits/sec 0 270 KBytes
[ 4] 2.00-3.00 sec 37.8 MBytes 317 Mbits/sec 0 284 KBytes
[ 4] 3.00-4.00 sec 37.7 MBytes 316 Mbits/sec 0 298 KBytes
[ 4] 4.00-5.00 sec 37.5 MBytes 314 Mbits/sec 0 317 KBytes
[ 4] 5.00-6.00 sec 37.7 MBytes 316 Mbits/sec 0 317 KBytes
[ 4] 6.00-7.00 sec 37.7 MBytes 316 Mbits/sec 0 331 KBytes
[ 4] 7.00-8.00 sec 37.7 MBytes 316 Mbits/sec 0 349 KBytes
[ 4] 8.00-9.00 sec 37.6 MBytes 315 Mbits/sec 0 387 KBytes
[ 4] 9.00-10.00 sec 37.7 MBytes 316 Mbits/sec 0 387 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 377 MBytes 316 Mbits/sec 0 sender
[ 4] 0.00-10.00 sec 376 MBytes 315 Mbits/sec receiver

-----------

You must have about 300 Mbits/sec receiving and sending.

[HiassofT]

[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 265 MBytes 222 Mbits/sec 3497 sender
[ 4] 0.00-10.00 sec 265 MBytes 222 Mbits/sec receiver

Don''t focus too much on retransmit values in iperf3, it''s normal that you see these when the RPi is on the receiving side.

The sender will try to transmit at full 1Gbit/sec bandwidth, but of course the RPi can't handle that - due to the USB2.0 bandwidth limitation. So packets will get dropped.

Try limiting the bandwidth in iperf3, eg by using "iperf3 -b 200M -c IP.OF.YOUR.RPI" on your PC and you'll see that the retransmits are gone.

so long,

Hias

[jdb]

Wasn't me doing the banning this time...

It was me. Said poster annoyed me sufficiently to motivate me to wield the banhammer with aplomb, so it was thusly wielded. Pro tip: even if you post technically useful information or analysis on our forums, if you act like you would be kicked out of a house dinner party by the host then you can expect a swift application of the banhammer.

Post all the bugs you want, but lose the attitude. I simply do not have time to deal with forum drama.

Edit: Oh, and ban evasion gets you special attention. Vindictive attention, you might say.

[RPIgoodSBC]

[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 265 MBytes 222 Mbits/sec 3497 sender
[ 4] 0.00-10.00 sec 265 MBytes 222 Mbits/sec receiver

Don''t focus too much on retransmit values in iperf3, it''s normal that you see these when the RPi is on the receiving side.

The sender will try to transmit at full 1Gbit/sec bandwidth, but of course the RPi can't handle that - due to the USB2.0 bandwidth limitation. So packets will get dropped.

Try limiting the bandwidth in iperf3, eg by using "iperf3 -b 200M -c IP.OF.YOUR.RPI" on your PC and you'll see that the retransmits are gone.

so long,

Hias

[email protected]:~ $ sudo iperf3 -b 200M -c YYY.YYY.Y.YYY -R

Connecting to host YYY.YYY.Y.YYY, port 5201
Reverse mode, remote host YYY.YYY.Y.YYY is sending
[ 4] local XXX.XXX.X.XXX port 46162 connected to YYY.YYY.Y.YYY port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 24.7 MBytes 207 Mbits/sec
[ 4] 1.00-2.00 sec 23.8 MBytes 200 Mbits/sec
[ 4] 2.00-3.00 sec 22.4 MBytes 188 Mbits/sec
[ 4] 3.00-4.00 sec 24.5 MBytes 206 Mbits/sec
[ 4] 4.00-5.00 sec 24.1 MBytes 202 Mbits/sec
[ 4] 5.00-6.00 sec 24.3 MBytes 204 Mbits/sec
[ 4] 6.00-7.00 sec 22.7 MBytes 191 Mbits/sec
[ 4] 7.00-8.00 sec 24.9 MBytes 209 Mbits/sec
[ 4] 8.00-9.00 sec 23.0 MBytes 193 Mbits/sec
[ 4] 9.00-10.00 sec 24.2 MBytes 203 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 239 MBytes 200 Mbits/sec 2965 sender
[ 4] 0.00-10.00 sec 239 MBytes 200 Mbits/sec receiver

iperf Done.

---

Not gone.

[asavah]

This is a real shame and shows poor testing on RPF side because this should have been detected and dealt with _before_ going into production.

Now let's get technical:

I'm running latest as of the moment of writing custom built 4.14.y kernel @ https://github.com/raspberrypi/linux/co ... a976c0a5b0

TX side is good.
But RX is awful, it's the worst SBC network-wise I've ever seen, and I own quite a collection.

As suggested above running iperf -c pi.ip -b 200M is good, 250 and higher is bad,
lots of retries, poor throughput, very poor, copying a file over nfs takes like forever.
My switch shows absolutely no errors on the pi port.
When sending from the pi (TX) everything is fine without any limits.

dtparam=eee=off does not help here at all.

I have a proper network at home and the 3b+ is connected to a L2 managed gigabit switch, which works perfectly.
So far for me the solution is to limit port speed on the switch to 100Mbit FD.
At 100 mbit forced link speed the network is stable, no drops, retrs, works fine, just like the old good rpi3 without plus.

----------------------------
Warning: rant
----------------------------
And sadly I have to agree with @tkaiser, I have a few poor cheapo-crapo wannabe pi boards which can max gigabit.

Pi network "architecture" over single USB port has to die. Period.
This may have been good when BCM2835 was designed, but can I remind you what year it is?
For your favorite god's sake - put a proper RGMII into the SoC ...

I know, I know what you might say - pi is for education and yada-yada, nowadays trying to teach computers with utterly broken network is pointless.

[RPIgoodSBC]

Curious.

Using netcat to transfer from an Ubuntu PC to the Pi,

Code: Select all

yes | pv -rab | nc [ip of pi] 4444

and running this on the Pi

Code: Select all

nc -l 4444 > /dev/null 

Gives me an average throughput of 27.5 MB/s, which is 220Mbits/s.

Which seem fine to me. Can you try it to see what happens?

Same test to me is...

27,1MiB/s average that is 227,3 Mbps

[RPIgoodSBC]

---> Bruny...What about public servers?

I think they are very saturated.
I do not think they will be useful in this case, but there is a test:

UPDATED...I was wrong this server works very well.But not in mi pi.

[email protected]:~ $ iperf3 -c bouygues.iperf.fr -p 5209 -R

Connecting to host bouygues.iperf.fr, port 5209
Reverse mode, remote host bouygues.iperf.fr is sending
[ 4] local Rpi3B+ port 40310 connected to 89.84.1.222 port 5209
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 3.62 MBytes 30.3 Mbits/sec
[ 4] 1.00-2.00 sec 2.35 MBytes 19.7 Mbits/sec
[ 4] 2.00-3.00 sec 2.41 MBytes 20.2 Mbits/sec
[ 4] 3.00-4.00 sec 2.11 MBytes 17.7 Mbits/sec
[ 4] 4.00-5.00 sec 1.88 MBytes 15.7 Mbits/sec
[ 4] 5.00-6.00 sec 1.85 MBytes 15.5 Mbits/sec
[ 4] 6.00-7.00 sec 1.94 MBytes 16.3 Mbits/sec
[ 4] 7.00-8.00 sec 1.56 MBytes 13.1 Mbits/sec
[ 4] 8.00-9.00 sec 1.87 MBytes 15.7 Mbits/sec
[ 4] 9.00-10.00 sec 1.77 MBytes 14.9 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 23.1 MBytes 19.3 Mbits/sec 41 sender
[ 4] 0.00-10.00 sec 21.5 MBytes 18.0 Mbits/sec receiver

iperf Done.

Very limited bandwidth in public servers.Too many users?

UPDATED...I was wrong this server works very well.But not in mi pi.

Same test on my big PC tested 1 hour later

big PC ~ $ iperf3 -c bouygues.iperf.fr -p 5209 -R

Connecting to host bouygues.iperf.fr, port 5209
Reverse mode, remote host bouygues.iperf.fr is sending
[ 4] local big PC port 48034 connected to 89.84.1.222 port 5209
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 15.1 MBytes 127 Mbits/sec
[ 4] 1.00-2.00 sec 22.5 MBytes 188 Mbits/sec
[ 4] 2.00-3.00 sec 23.2 MBytes 195 Mbits/sec
[ 4] 3.00-4.00 sec 24.5 MBytes 206 Mbits/sec
[ 4] 4.00-5.00 sec 24.7 MBytes 207 Mbits/sec
[ 4] 5.00-6.00 sec 24.5 MBytes 206 Mbits/sec
[ 4] 6.00-7.00 sec 24.7 MBytes 207 Mbits/sec
[ 4] 7.00-8.00 sec 24.7 MBytes 207 Mbits/sec
[ 4] 8.00-9.00 sec 21.4 MBytes 180 Mbits/sec
[ 4] 9.00-10.00 sec 22.3 MBytes 187 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 230 MBytes 193 Mbits/sec 17 sender
[ 4] 0.00-10.00 sec 229 MBytes 192 Mbits/sec receiver

iperf Done.

Same server at same minute and much more bandwith on my big PC.

big PC bandwith 192 Mbits/sec

Rpi3B+ bandwith 14,3 Mbits/sec

Then is not my ISP=200down/20up.

Tinkerboard screenshot on same net, cable, router-port...Tinker-OS 2.0.5 linaro system...

iperf3-remote-server-test-on-tinkerboard-on-same-net-cable-router-port-that-rpi3b-plus-before-tested.png (238.73 KiB) Viewed 3339 times

RPi3B+ screenshoot...

iperf3-remote-server-test-on-rpi3b-plus.png (212.75 KiB) Viewed 3327 times

Armbian latest test on tinkerboard...

speedtest-cli-and-iperf3-test-on-latest-armbian.jpg (251.27 KiB) Viewed 3197 times

[asavah]

I correct myself a bit:

If I enable flow control on the port of the pi on my switch the RX path works ok-ish , only a few retrs with iperf3, nfs works ok too.

Code: Select all

iperf3 -c rpi3plus -t 60
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-60.00  sec  1.69 GBytes   242 Mbits/sec    8             sender
[  4]   0.00-60.00  sec  1.69 GBytes   242 Mbits/sec                  receiver

But this is still not acceptable for people who have cheap unmanaged gigabit switches.

[jamesh]

Curious.

Using netcat to transfer from an Ubuntu PC to the Pi,

Code: Select all

yes | pv -rab | nc [ip of pi] 4444

and running this on the Pi

Code: Select all

nc -l 4444 > /dev/null 

Gives me an average throughput of 27.5 MB/s, which is 220Mbits/s.

Which seem fine to me. Can you try it to see what happens?

Same test to me is...

27,1MiB/s average that is 227,3 Mbps

Again, once you take into account the usb2 bus which this attached to, this is pretty good. Do you think it should be faster? We expect a 2 to 3 times speedup over the 3b, so these numbers look about right.

[jamesh]

I correct myself a bit:

If I enable flow control on the port of the pi on my switch the RX path works ok-ish , only a few retrs with iperf3, nfs works ok too.

Code: Select all

iperf3 -c rpi3plus -t 60
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-60.00  sec  1.69 GBytes   242 Mbits/sec    8             sender
[  4]   0.00-60.00  sec  1.69 GBytes   242 Mbits/sec                  receiver

But this is still not acceptable for people who have cheap unmanaged gigabit switches.

Interesting data point, I'll take a look tomorrow.

As for usb2 needing to die, that has been abundantly clear for a couple of years at least. However, doing something about it that works for the Pi ecosystem takes longer than that.

Meanwhile, we do the best we can with what's available.

[asavah]

Interesting data point, I'll take a look tomorrow.

As for usb2 needing to die, that has been abundantly clear for a couple of years at least. However, doing something about it that works for the Pi ecosystem takes longer than that.

Meanwhile, we do the best we can with what's available.

I know you do, and I know you (the team) will fix this if it's at all posible, the engineers here are very skilled,
if any patches or workarounds are found I will be more than happy to help test anything, I have skill and equipment, I lack time tho.
However please understand that as network engineer (SNA) I find it very frustrating to stumble into poor working network on my new shiny 3b+

[RPIgoodSBC]

Test people:

Code: Select all

sudo apt-get install -y iperf3

Code: Select all

iperf3 -c bouygues.iperf.fr -p 5209 -R

This server works very well.
Test on RPi3B+ and on other PC gigabit NIC.

viewtopic.php?f=63&t=208512&start=75#p1294025

[jamesh]

Interesting data point, I'll take a look tomorrow.

As for usb2 needing to die, that has been abundantly clear for a couple of years at least. However, doing something about it that works for the Pi ecosystem takes longer than that.

Meanwhile, we do the best we can with what's available.

I know you do, and I know you (the team) will fix this if it's at all posible, the engineers here are very skilled,
if any patches or workarounds are found I will be more than happy to help test anything, I have skill and equipment, I lack time tho.
However please understand that as network engineer (SNA) I find it very frustrating to stumble into poor working network on my new shiny 3b+

I'm no network expert, so can you give me some idea of what you think might be happening? From the comment above, is the transmitter sending out at 1000T but since the Pi cannot rx at that speed, it's losing packets which need retransmitting? So turning on flow control stops the sender from over sending?

Are you aware of any ability in receivers to broadcast the fact that is rx speed is limited to avoid the need for each end to understand/enable flow control? Or can receivers tell senders to use flow control? Or can rx'ers tell senders a maximum data rate to use?

I can have a dig in the chip datasheet tomorrow to see if anything pops out, although TBH, Microchip should be sorting this sort of thing out - it's their chip, firmware and driver.

[asavah]

I've borrowed a USB 2.0 to gigabit dongle (ax88178) from one of our managers and plugged it into my ubuntu workstation and to a cisco 3750g switch, flow control off.
Everything worked fine, no RX issues.
So I'd say that it's effectively microchip IC and/or driver issue or some compatibility issue between LAN/HUB chip and SoC USB.
Seeing kernel oopses that were reported (now triaged in rpi-4.14.y branch) with lan78xx driver I'd dare to say that microchip driver is in poor state and needs to be properly fixed by the vendor.

Sadly I can't bring the dongle home to run some tests in the same environment as the pi.

I'm not that good with low level stuff, which we don't usually need, and right now I can't offer any advice.
I'll investigate this on the weekend and report back if I find something.

[ejolson]

I've borrowed a USB 2.0 to gigabit dongle (ax88178) from one of our managers and plugged it into my ubuntu workstation and to a cisco 3750g switch, flow control off.
Everything worked fine, no RX issues.

Greater than 300 Mbit speeds both up and down using a gigabit USB 2.0 dongle on a non-Pi computer with no retransmitted packets was also reported earlier on this thread, but I can't find that post anymore. The person who started this thread tried an external gigabit USB networking device in the 3B+ but still experienced retransmissions and slow 200 Mbit download. Unfortunately, no single person has used the same dongle in both 3B+ and non-3B+ systems, so results are inconclusive.

While 200 Mbit download may seem fine at first glance, those tests were performed using a low-latency local area network. Retransmits have a much greater impact when transferring data across a high-latency connection to the internet.