sladflob
Posts: 10
Joined: Sun Nov 03, 2019 1:23 am

Sudden drop in network speed

Sun Nov 03, 2019 1:41 am

I've had a Pi 4 for a while now and have been experimenting with using it as a NAS/download box. It is currently running Raspbian. I've noticed that after some indeterminate period of time (sometimes weeks, sometimes days) the download speeds achieved by the box drop quite drastically and stay that way until I reboot. For example, I am normally able to max out my 100 Mb/s connection and achieve over 11 MB/s downloads, but when the Pi gets into this state the max speed I see is 5.2 MB/s.

I have eliminated any local networking issues as another PC connected to the same router is able to download from the same site at maximum speed. I am only using the ethernet interface, not WiFi. I have tried restarting the network both by doing "ip link set eth0 down/up" and "ifconfig eth0 down/up" and neither restores the network speed. It seems like a reboot is the only thing that fixes it.

Could there be something wrong with this unit? What are some other diagnostics I can do?

Thanks!

epoch1970
Posts: 4123
Joined: Thu May 05, 2016 9:33 am
Location: Paris, France

Re: Sudden drop in network speed

Sun Nov 03, 2019 12:33 pm

Does speed drop over the local network as well? E.g. PI to PC?
"S'il n'y a pas de solution, c'est qu'il n'y a pas de problème." Les Shadoks, J. Rouxel

sladflob
Posts: 10
Joined: Sun Nov 03, 2019 1:23 am

Re: Sudden drop in network speed

Mon Nov 04, 2019 6:16 am

Does speed drop over the local network as well?
Good question. I was fortunate to have the Pi in "slow mode" so tried doing a file transfer using FTP from my Synology NAS. In slow mode, the Pi was, at best, able to achieve about 4 MB/s. I rebooted the Pi and tried again and it downloaded the same file at 64.2 MB/s. As a comparison, my Windows laptop downloaded the file at around 85 MB/s. So there's a pretty severe slow-down when the Pi is in slow mode.

My other thought was that maybe something is going strange with the SSD I'm downloading to? I have the root file system mounted from an NVMe SSD connected via a powered hub. Could that be the culprit? Possibly not but something to keep in mind at least. How could I tell if it was network or disk limiting the download speed?

iznobe
Posts: 315
Joined: Sun Feb 05, 2017 4:14 pm
Location: Avignon , FRANCE

Re: Sudden drop in network speed

Mon Nov 04, 2019 8:59 am

Hi ,

could you look at this:

Code: Select all

cat /var/log/syslog
when the slow down speed appears ?

maybe there will be some indications about what happened .

renice123
Posts: 138
Joined: Tue Oct 15, 2019 3:56 pm

Re: Sudden drop in network speed

Mon Nov 04, 2019 12:34 pm

Modern SSD disks have their own algorithm for working with files. In particular, the disk controller does not immediately write the file to memory cells, but caches data in its own RAM.
Perhaps your SSD drive works according to its own "proprietary" algorithm, which when working with Debian leads to errors.
Also check the status of the SSD drive, because old drives can work poorly due to the large number of “broken” cells.
Often this behavior is observed in Samsung SSDs and in this case, some recommend trying to change the swap settings of the area

In any case, it seems to me that you should check the SSD drive for critical hardware errors
sudo apt-get install gnome-disk-utility
»  SMART Data
sudo apt-get install smartmontools
sudo smartctl -a /dev/sd** > smart-results.txt

sladflob
Posts: 10
Joined: Sun Nov 03, 2019 1:23 am

Re: Sudden drop in network speed

Mon Nov 04, 2019 1:17 pm

Hi, I managed to narrow down the time period where the slowdown occurred to a window of a few minutes (I was downloading some files - one download was fast, the next one was slow) and unfortunately there doesn't seem to be anything in the syslog for that time period. I also looked in /var/log messages.

sladflob
Posts: 10
Joined: Sun Nov 03, 2019 1:23 am

Re: Sudden drop in network speed

Mon Nov 04, 2019 1:23 pm

The output of smartctl says that the device lacks SMART capability, so that doesn't sound too useful. It's pretty much a brand new drive, I bought it when I got the Pi 4.

iznobe
Posts: 315
Joined: Sun Feb 05, 2017 4:14 pm
Location: Avignon , FRANCE

Re: Sudden drop in network speed

Mon Nov 04, 2019 2:55 pm

sladflob wrote:
Mon Nov 04, 2019 1:17 pm
Hi, I managed to narrow down the time period where the slowdown occurred to a window of a few minutes (I was downloading some files - one download was fast, the next one was slow) and unfortunately there doesn't seem to be anything in the syslog for that time period. I also looked in /var/log messages.
how do you do theses downloads ?

command line ? GUI ?

the downloads are in local network , on a samba share ? on a NFS share ? the downloads are for same type of file ?

in slow mode, the speed is it fast at the beginning of the download , then slowed down ?

i don't think the SSD is in cause , SSD are capable to write 350 Mb/s easily now ( read specifications of SSD write capabilities ) .

but it is possible there is an autonegotiation on ethernet for the speed capability , i ' m not sure about this ( or a power management )

sladflob
Posts: 10
Joined: Sun Nov 03, 2019 1:23 am

Re: Sudden drop in network speed

Tue Nov 05, 2019 12:15 am

The downloads are from a remote Usenet server controlled via the web interface to SABnzbd. But I don't think there's anything special about the downloads that is causing the problem. Sometimes the Pi has been idle for a few days and when I come back to it I find that it has gone into slow mode. Once it is slow it's slow for all kinds of network traffic, including LAN traffic. I've tried using ethtool to check the status of the interface and it looks fine. Also, ifconfig does not show any dropped packets on the interface. So I'm a bit stumped.

epoch1970
Posts: 4123
Joined: Thu May 05, 2016 9:33 am
Location: Paris, France

Re: Sudden drop in network speed

Tue Nov 05, 2019 1:46 pm

Out of RAM? Too many open connections/files?
Given the use you make of the machine, I would just leave it at that and have cron reboot it every day.
"S'il n'y a pas de solution, c'est qu'il n'y a pas de problème." Les Shadoks, J. Rouxel

dustnbone
Posts: 146
Joined: Tue Nov 05, 2019 2:49 am

Re: Sudden drop in network speed

Tue Nov 05, 2019 11:35 pm

Could it possibly be related to the dhcpcd issues discussed here?

https://www.raspberrypi.org/forums/view ... 8&t=255887

I was having similar slowdowns the other day, noticed dhcpcd pulling 100% of a CPU core. Disabling dhcpcd fixed it for me, I didn't troubleshoot it any further because I don't need it on that Pi.

sladflob
Posts: 10
Joined: Sun Nov 03, 2019 1:23 am

Re: Sudden drop in network speed

Thu Nov 07, 2019 12:36 am

epoch1970 wrote: Out of RAM? Too many open connections/files?
Given the use you make of the machine, I would just leave it at that and have cron reboot it every day.
Rebooting daily is not really a solution - I'd like to get to the bottom of what's going on. I'm fairly certain it has nothing to do with sabnzbd per se, other than that it's the process that uses the network a lot so it's where I first notice the slowdown. As I mentioned in an earlier post, once it's in that state even FTP on the local network is terribly slow. Next time it happens I'll try killing the sab process and see if it makes any difference.
dustnbone wrote: Could it possibly be related to the dhcpcd issues discussed here?
Possibly! I have updated dhcpcd so will see what happens. Of course now it's not in the slow network state and who knows how long it will stay "good", which makes it very difficult to tell if it's been fixed or not.

sladflob
Posts: 10
Joined: Sun Nov 03, 2019 1:23 am

Re: Sudden drop in network speed

Mon Dec 30, 2019 1:14 am

Hi everyone, I'm resuscitating this thread because I'm still having problems with my Pi's network speed suddenly dropping.

Since my last post I've been away for a while so didn't use the Pi, but since returning I fired it up again and within a couple of days I'm now back in the "slow" state again. Again, it's not just remote downloads that are slow but also local LAN transfers. I've tried restarting the network service and also taking the eth0 interface down and then up again and it hasn't fixed things. It seems that the only thing that fixes it is rebooting.

I'm just wondering if there are any other diagnostics I can perform to try and see what's happening?

If no-one else sees this problem then is it possible it's a hardware problem? I don't even know if it's possible to return the Pi if it has this intermittent problem.

Sorry if I'm repeating myself but maybe some fresh eyes can give me a clue on what to do. Thanks!

epoch1970
Posts: 4123
Joined: Thu May 05, 2016 9:33 am
Location: Paris, France

Re: Sudden drop in network speed

Mon Dec 30, 2019 11:01 am

SABnzbd Would be my prime suspect. Try suspending the use of that thing, reboot, and see if performance slows down over time.
I don’t think an intermittent hardware issue with Ethernet is possible.
"S'il n'y a pas de solution, c'est qu'il n'y a pas de problème." Les Shadoks, J. Rouxel

sladflob
Posts: 10
Joined: Sun Nov 03, 2019 1:23 am

Re: Sudden drop in network speed

Mon Dec 30, 2019 11:21 pm

epoch1970 wrote: SABnzbd Would be my prime suspect. Try suspending the use of that thing, reboot, and see if performance slows down over time.
I don’t think an intermittent hardware issue with Ethernet is possible.
It's not just sabnzbd - when it's in the "slow state", FTP transfers over my local network are also affected. Or do you think it's possible that sabnzbd is somehow causing the network interface to go into the slow state?

dustnbone
Posts: 146
Joined: Tue Nov 05, 2019 2:49 am

Re: Sudden drop in network speed

Tue Dec 31, 2019 7:22 pm

I wouldn't expect sabnzbd to have any effect when it's not actively downloading from Usenet. It doesn't do anything unless you give it something to do, aside from running a web service. You could try stopping the sabnzbd service for troubleshooting but I really can't see it being the source of your problem.

I assume you're tested using another cable/port on your ethernet switch?

sladflob
Posts: 10
Joined: Sun Nov 03, 2019 1:23 am

Re: Sudden drop in network speed

Thu Jan 02, 2020 7:45 am

dustnbone wrote:
Tue Dec 31, 2019 7:22 pm
I assume you're tested using another cable/port on your ethernet switch?
Yes, have tried that, and also tried bypassing the switch and plugging directly into the router.

I left it for a few days to see if it got better but if anything it got worse. Downloads slowed down to around 240 KB/s. In the end it was taking too long to even download Raspbian updates so I rebooted. Now it's back up to full speed again. I really don't know what to do at this stage.

sladflob
Posts: 10
Joined: Sun Nov 03, 2019 1:23 am

Re: Sudden drop in network speed

Wed Jan 08, 2020 2:32 am

Just to add more information...

I disabled sabnzbd and wrote a bash script to repeatedly download a file from my local network using curl via ftp. I rebooted the pi and ran the script and it was downloading happily at about 100 MB/s for about two days. I checked it last night and it was still going ok. This morning I checked and the speed had dropped to around 40 MB/s. Nothing has changed on the server or on my network. Memory use remains unchanged at around 1.4 GB.

So that eliminates sab from the equation. I'm still not sure whether to blame hardware or software. I'm reluctant to do it, but maybe I should do a clean install of Rasbian just to be sure. Or should I contact the shop for replacement?

Return to “Troubleshooting”