grindax
Posts: 8
Joined: Mon Mar 11, 2013 1:47 pm

Very slow SFTP / SCP performance [obsolete]

Mon Mar 11, 2013 2:04 pm

Hi all, I hope someone can provide some assistance with a problem I'm experiencing.

It seems that for some reason SFTP / scp performance is extremely slow on both of my Raspberry Pis.

One is running Raspbian and the other is running Raspbmc, and both are using wired connections directly to my router.

If I connect from my Windows PC to either using WinSCP, then SFTP transfers top out at around 1,800 KiB/s. If I use scp directly from the one RPi to copy a large file to the other RPi, then scp reports the performance to be on average 2.3 MB/s.

General network performance, and in particular bandwidth, seems to be fine to/from the RPis. If I run iperf between them, I get reasonably good speeds:

Code: Select all

pi@raspberrypi ~ $ iperf -c 192.168.1.3 -f M
------------------------------------------------------------
Client connecting to 192.168.1.3, TCP port 5001
TCP window size: 0.02 MByte (default)
------------------------------------------------------------
[  3] local 192.168.1.4 port 50036 connected with 192.168.1.3 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  93.2 MBytes  9.31 MBytes/sec
pi@raspberrypi ~ $
Likewise, testing SD card performance, I see reasonably good performance:

Code: Select all

pi@raspberrypi ~ $ sudo dd if=/dev/mmcblk0 of=/dev/null bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 5.14085 s, 20.4 MB/s
While a large SFTP transfer is underway, I used 'top' to have a look at CPU performance, and it's not being maxed out. CPU usage hovers between 49-55%.

So if network speed, CPU utilization, and SD card performance can be ruled out, what could be causing sftp and scp to be so slow?

joschh
Posts: 4
Joined: Wed Apr 10, 2013 1:31 pm

Re: Very slow SFTP / SCP performance

Tue Apr 16, 2013 9:02 pm

Hi,

same problem here...
Transferring files from my PC to the Raspberry via scp gives me a speed about ~2MB/s.
Any tips to speed it up? (encryption is already disabeld)

Thanks!

michauko
Posts: 2
Joined: Fri May 03, 2013 5:09 pm

Re: Very slow SFTP / SCP performance

Fri May 03, 2013 5:12 pm

Hi there,
Same thing here. 128kB/s max via SFTP
I think the CPU is the main bottleneck. It's not powerful at all (I'm not talking about the video decoder chip) and, as SFTP requires some encryption, you can't go faster than 2Mb/s
Activate FTP in the raspbmc settings menu
Do the same transfer.... I'm reaching 3 MB/s (~25 Mb/s).... on a 100Mb/s switch.
Sounds reasonnable

Regards

michauko
Posts: 2
Joined: Fri May 03, 2013 5:09 pm

Re: Very slow SFTP / SCP performance

Fri May 03, 2013 7:58 pm

about cpu usage, 50% must mean one core full. raspberry is a two core architecture if I remember well
and sftp is probably not multi threaded

User avatar
pluggy
Posts: 3635
Joined: Thu May 31, 2012 3:52 pm
Location: Barnoldswick, Lancashire,UK
Contact: Website

Re: Very slow SFTP / SCP performance

Fri May 03, 2013 8:06 pm

Nope, only one core. Some of the later ARM processors used in phones are multi core.
Don't judge Linux by the Pi.......
I must not tread on too many sacred cows......

M33P
Posts: 199
Joined: Sun Sep 02, 2012 1:14 pm

Re: Very slow SFTP / SCP performance

Fri May 03, 2013 8:08 pm

michauko wrote:about cpu usage, 50% must mean one core full. raspberry is a two core architecture if I remember well
and sftp is probably not multi threaded
Your information is faulty. There is one (1) ARM CPU on the Pi.

SCP will be slow because of the overhead of encryption. Be careful with the "CPU Usage" reported - linux reports userspace/kernelspace/waiting for IO usage as separate percentages adding up to 100%.

joscch: Are you sure you've disabled encryption? Even the minimal fallback (3DES?) will have some overhead.

ddurdle
Posts: 61
Joined: Sat May 04, 2013 12:20 am

Re: Very slow SFTP / SCP performance

Sat May 04, 2013 9:05 am

You might want to try rsync.

I rsync between computers and the raspberry pi and I encounter 20MB/s transfer rates.

ddurdle
Posts: 61
Joined: Sat May 04, 2013 12:20 am

Re: Very slow SFTP / SCP performance

Sun May 05, 2013 9:41 pm

Sorry, I'm mistaken, I'm getting 2.5MB/s over rsync.

ddurdle
Posts: 61
Joined: Sat May 04, 2013 12:20 am

Re: Very slow SFTP / SCP performance

Sun May 19, 2013 3:50 pm

Actually, now I'm not sure what is the cause of the capped SCP/SFTP/rsync (over ssh) on the raspberry pi. I noticed rsyncing between two core 2 duo systems with 8GB of ram had a 2.5MB/s transfer rate as well. This was over wifi, so perhaps 20mbps over G (within close promixity of eachother and with strong wifi signal) is some limitation I'm not aware (since G is supposed to be rated at 54mbps).

I'll have to try some more tests over Ethernet with and without the raspberry pi.

alcachi
Posts: 4
Joined: Tue Sep 03, 2013 11:46 am

Re: Very slow SFTP / SCP performance

Tue Sep 03, 2013 11:57 am

The problem seems to be related to the RPi CPU not being able to cope with the additional encryption overhead required by scp. Monitoring with top I can see all CPU resources are being used while doing the transfer but this is also common in other systems with better CPU.

One quick test I tried is changing the cypher algorithm to "blowfish", that it is supposed to be faster, but I still get 3.5MB/s transfer rates.

It would be nice to find some tweak that allows to come closer to the 10MB/s limit.

sjiveson
Posts: 2
Joined: Tue Jan 06, 2015 4:07 pm

Re: Very slow SFTP / SCP performance

Tue Jan 06, 2015 4:38 pm

This is not to do with CPU or encryption overheads, rather a fundamental flaw in the Linux SSH library.

Using the tar and netcat commands like so (in the order shown) will achieve much better results;

Pi B (recipient): nc -l 8080 | tar xf - -C /dir/
Pi A (sender): tar cf - filename | nc -w1 'Pi B IP address' 8080

Obviously you can change the port 8080 to anything of your choosing and /dir/ should be a suitable directory on the recipient.

The above offers no security or authentication but requires you have suitable privileges on both hosts.

You can enable compression by adding a z before the f - with each tar command on each Pi. I find this doesn't help much but it may if your files are mostly text.

sjiveson
Posts: 2
Joined: Tue Jan 06, 2015 4:07 pm

Re: Very slow SFTP / SCP performance

Wed Jan 07, 2015 9:53 am

Actually, I've realised tar is only good if you have multiple files/directories or if compression matters. If not, I've found this works just fine, same caveats as before;

Sender: nc -l 8080 < file_name
Recipient: nc ip_address 8080 > file_name

In testing (not on a Pi I'm afraid) this is at least twice as fast as scp.

DieHimmbeere
Posts: 3
Joined: Thu Feb 12, 2015 1:39 pm

Re: Very slow SFTP / SCP performance

Sat Feb 14, 2015 4:58 pm

So i bought a Raspberry PI 2 B to move my SFTP-server from my router (OpenWRT) to the pi.
Well.... performance is worse than from my router (which has like 10% of the CPU/RAM-perf of the pi).

I write/read to/from an ntfs hdd which is mounted to a chrooted dir in which the sftp-users are "jailed". Server is OpenSSH-SFTP in default configuration.
Download speed: ~1,1MiB/s, Upload speed: ~5,5MiB/s.
Pi is running the newest build of Raspbian (upgraded today), nothing except Ethernet, power and the HDD is plugged in.

If i try to download something via WLAN to my Android phone i'll get ~350KiB/s which is just miserable. My Router could at least do ~700KiB/s.

Anybody able to help me here? Share your awesome tricks please.

klricks
Posts: 7779
Joined: Sat Jan 12, 2013 3:01 am
Location: Grants Pass, OR, USA
Contact: Website

Re: Very slow SFTP / SCP performance

Sat Feb 14, 2015 5:12 pm

DieHimmbeere wrote:So i bought a Raspberry PI 2 B to move my SFTP-server from my router (OpenWRT) to the pi.
Well.... performance is worse than from my router (which has like 10% of the CPU/RAM-perf of the pi).

I write/read to/from an ntfs hdd which is mounted to a chrooted dir in which the sftp-users are "jailed". Server is OpenSSH-SFTP in default configuration.
Download speed: ~1,1MiB/s, Upload speed: ~5,5MiB/s.
Pi is running the newest build of Raspbian (upgraded today), nothing except Ethernet, power and the HDD is plugged in.

If i try to download something via WLAN to my Android phone i'll get ~350KiB/s which is just miserable. My Router could at least do ~700KiB/s.

Anybody able to help me here? Share your awesome tricks please.
Loose the NTFS and use a Linux format ext3, ext4. There is a huge performance hit using NTFS as the RPi has to burn CPU to translate.
Unless specified otherwise my response is based on the latest and fully updated RPiOS Buster w/ Desktop OS.

ghans
Posts: 7885
Joined: Mon Dec 12, 2011 8:30 pm
Location: Germany

Re: Very slow SFTP / SCP performance

Sat Feb 14, 2015 5:35 pm

SFTP is not meant to be fast , it's meant to be secure. Inside a LAN unencrypted (and therefore faster) protocols should be
acceptable.

ghans
• Don't like the board ? Missing features ? Change to the prosilver theme ! You can find it in your settings.
• Don't like to search the forum BEFORE posting 'cos it's useless ? Try googling : yoursearchtermshere site:raspberrypi.org

DieHimmbeere
Posts: 3
Joined: Thu Feb 12, 2015 1:39 pm

Re: Very slow SFTP / SCP performance

Sun Feb 15, 2015 11:48 am

klricks wrote:Loose the NTFS and use a Linux format ext3, ext4. There is a huge performance hit using NTFS as the RPi has to burn CPU to translate.
Tested with a fast 32GB USB stick formatted as ext4: starts acceptable fast with 6-7MiB/s then hangs for 5sec, falls down to 4,5MiB/s for about 10seconds, then hangs again for 5-10sec, after that upload-speed will be about 3MiB/s for 10seconds, then hang (0MiB/s) for 10seconds, 3MiB/s for 10sec, 0MiB/s for 10sec and so on....
CPU-load looks like this:
Image
USB is automounted via fstab:

Code: Select all

/dev/sda1 /home/ftp/SFTPfiles ext4 rw,umask=0002 0 0
Download is fine with ~6,5MiB/s (does not hang).

No idea whats wrong here.
ghans wrote:SFTP is not meant to be fast , it's meant to be secure. Inside a LAN unencrypted (and therefore faster) protocols should be
acceptable.

ghans
Well the SFTP is accessable from outside too ;)

grindax
Posts: 8
Joined: Mon Mar 11, 2013 1:47 pm

Re: Very slow SFTP / SCP performance

Thu Feb 19, 2015 10:11 pm

Just to revisit this topic, and summarize:

I originally misinterpreted the CPU stats in top because I didn't add up the different categories of CPU performance reported. Basically, the CPU in the Raspberry Pi 1 was maxed out, and the maximum SFTP throughput that it can handle is in the region of 2MB/s.

My question now is: can someone with a Raspberry Pi 2 tell me how much better it is at handling SFTP transfers?

User avatar
skalada
Posts: 7
Joined: Sun Feb 22, 2015 2:19 pm
Location: Germany
Contact: Website

Re: Very slow SFTP / SCP performance

Sun Feb 22, 2015 2:44 pm

My question now is: can someone with a Raspberry Pi 2 tell me how much better it is at handling SFTP transfers?
Hello,

Im using Raspbian in a Raspberry Pi 2 (in the next image are the details)
Image

I tested 2 scenarios:

- 1. I connect to the network via WPA2 protected wifi (with an Edimax wifi USB EW-7811Un and Wireles switch that supports IEEE 802.11g too) and the SCP transfer indicates rates of 2.1MB/s (but file trasnfer is really slow). It doesnt seem to be CPU's overhead while doing the SCP (only 5-6% CPU Usage);
Image
I tried before with NFS shared folders and by using vsftpd and the transfer rates where very slow too.

- 2. I connect my Raspberry direct to the Ethernet cable and the SCP rates still show 2,1MB/s but the real transfer of the files is extremely faster than via wifi. And I get a 90% CPU ussage;
Image

In both scenarios SCP says speed rates are 2,1MB/s but in the case of using Ethernet connection files are transfered really faster than in the case of Wireless-connection. Looks like I can not trust this rates, next times will contrast them with bmon ressults in scp "server" host.
Image

I tried with another wifi USB Adapter, and had more ressults; Seems like the Wireless Adapter limits the CPU usage to itś capacity or sth like that. SCP seems to show always 2,1MB/s evenso speed really seems very different in each case so I used bmon to check the speeds and I got this;

Image

Seems that somewhere in the wifi interface there is a bottleneck (more than in the CPU).

Any explanation or idea about it or any more tests to do?

Regards,
Last edited by skalada on Sun Feb 22, 2015 11:44 pm, edited 1 time in total.

tobidope
Posts: 2
Joined: Sun Feb 22, 2015 5:54 pm

Re: Very slow SFTP / SCP performance

Sun Feb 22, 2015 7:30 pm

I have the same problem and it seems to be connected to an extreme high rate of dropped packets over wifi. I'm also using the Edimax wifi adapter. Could you look at the number of dropped packets in the output of

Code: Select all

ifconfig wlan0

User avatar
skalada
Posts: 7
Joined: Sun Feb 22, 2015 2:19 pm
Location: Germany
Contact: Website

Re: Very slow SFTP / SCP performance

Mon Feb 23, 2015 7:15 pm

Thanks, You're right

Edimax
wlan0 RX packets:4123 errors:0 dropped:1537 overruns:0 frame:0
TX packets:7342 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:537355 (524.7 KiB) TX bytes:10873559 (10.3 MiB)

Wifisky
wlan0 RX packets:4767 errors:0 dropped:0 overruns:0 frame:0
TX packets:4915 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:314826 (307.4 KiB) TX bytes:7131043 (6.8 MiB)

Ethernet cable
eth0 RX packets:5144 errors:0 dropped:0 overruns:0 frame:0
TX packets:9112 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:288191 (281.4 KiB) TX bytes:13449645 (12.8 MiB)

Just for curiosity, any idea about why does it happen?
Does it happen just in the Raspberry or it's an Edimax EW-7811Un USB-toggle specific problem?

Regards,

tobidope
Posts: 2
Joined: Sun Feb 22, 2015 5:54 pm

Re: Very slow SFTP / SCP performance

Mon Feb 23, 2015 7:21 pm

Tach skalada,
to be honest I have no clue. I'm still searching. Do you also just recently bought your raspberry pi?

User avatar
skalada
Posts: 7
Joined: Sun Feb 22, 2015 2:19 pm
Location: Germany
Contact: Website

Re: Very slow SFTP / SCP performance

Mon Feb 23, 2015 7:41 pm

Yes, I just got it last week... I don't have any previous model... to check if it happens the same.

Anyway, I found many threads speaking about Edimax and Raspberry slow rates... but didn't find clear conclusions...

I configured wpa_supplicant by following tutorials, didnt understand 100% what I was doing but I think it doesn't look like a consequence of a wrong configuration there.

grindax
Posts: 8
Joined: Mon Mar 11, 2013 1:47 pm

Re: Very slow SFTP / SCP performance

Tue Feb 24, 2015 3:32 pm

To bring this discussion back on topic:

So it's accurate to say that on a wired connection the Pi 2 is just as slow as the Pi 1 when transferring data using SFTP ?

User avatar
skalada
Posts: 7
Joined: Sun Feb 22, 2015 2:19 pm
Location: Germany
Contact: Website

Re: Very slow SFTP / SCP performance

Tue Feb 24, 2015 4:15 pm

So it's accurate to say that on a wired connection the Pi 2 is just as slow as the Pi 1 when transferring data using SFTP ?
Thinking that SFTP and SCP performance are the same. I think not, is not accurate. Here I try to explain why I think so.

Focusing on the tests I posted; https://db.tt/921VehQU the last one (Wired SCP with Raspberry 2)

SCP reports ----> 2.1 MB/s <--- I didnt whant o say that cos it sounds like a very hard accusation but I think SCP transfer rate reports lie
bmon -------------> 7,82MiB/s <--- for this experiments I trust bmon rates more.

As you can see in the picture I posted, in the Ethernet wired scenario, the saturated CPU usage seems to be the limiting factor for the SCP copy speed.

Image

It's just my theory; I don't have a previous model to test it but I understand there is a direct relation between CPU power and SCP transfer rates. So, I understand 2 B model is faster. And comparing the 7,82 MiB/s with the results from Pi 1 models I can find in this thread, seems to confirm this.

Regards

ghans
Posts: 7885
Joined: Mon Dec 12, 2011 8:30 pm
Location: Germany

Re: Very slow SFTP / SCP performance

Tue Feb 24, 2015 6:40 pm

Is open WiFi + SCP faster ?

ghans
• Don't like the board ? Missing features ? Change to the prosilver theme ! You can find it in your settings.
• Don't like to search the forum BEFORE posting 'cos it's useless ? Try googling : yoursearchtermshere site:raspberrypi.org

Return to “Troubleshooting”