janekpolak
Posts: 8
Joined: Sun Aug 23, 2020 8:30 pm

Raspberry 4 - NAS, slow copying over the network

Tue Aug 25, 2020 10:54 am

I have problem with copying backup speed over network.
How can I confirm that this is the maximum data backup speed?
How can I confirm (debug) that the problem is with the disk?

I have a new disk: Seagate Barracuda Pro
Network: 1000 mbps

When I copy file locally I have 71.17MB/s from SD to HDD but over network I have only 36MB/s.

##COPY FILE LOCALLY
user@raspberrypi:/tmp# dd if=/dev/urandom of=500M bs=10M count=50
50+0 records in
50+0 records out
524288000 bytes (524 MB, 500 MiB) copied, 26.8808 s, 19.5 MB/s
user@raspberrypi:/tmp# rsync -avh --progress 500M /mnt/hdd1/
sending incremental file list
500M
524.29M 100% 71.17MB/s 0:00:07 (xfr#1, to-chk=0/1)

sent 524.42M bytes received 35 bytes 69.92M bytes/sec
total size is 524.29M speedup is 1.0

##COPY OVER NETWORK
[user@xyz tmp]# dd if=/dev/urandom of=500M bs=10M count=50
50+0 records in
50+0 records out
524288000 bytes (524 MB, 500 MiB) copied, 3.24353 s, 162 MB/s
[user@xyz tmp]# rsync -avh --progress 500M user@raspberry:/mnt/hdd1/
user@raspberry's password:
sending incremental file list
500M
524.29M 100% 36.52MB/s 0:00:13 (xfr#1, to-chk=0/1)

sent 524.42M bytes received 35 bytes 25.58M bytes/sec
total size is 524.29M speedup is 1.00

Kendek
Posts: 265
Joined: Thu Jul 25, 2019 4:39 pm
Location: Kaposvár, Hungary

Re: Raspberry 4 - NAS, slow copying over the network

Tue Aug 25, 2020 12:10 pm

Pi4 doesn't have hardware AES accelerator (the best option is chacha20-poly1305), so the rsync over SSH will be slower. If you don't need encryption, use the rsync daemon:

Code: Select all

> rsync -avh --progress 500M raspberry::mnt/hdd1

janekpolak
Posts: 8
Joined: Sun Aug 23, 2020 8:30 pm

Re: Raspberry 4 - NAS, slow copying over the network

Tue Aug 25, 2020 12:37 pm

Kendek wrote:
Tue Aug 25, 2020 12:10 pm
Pi4 doesn't have hardware AES accelerator (the best option is chacha20-poly1305), so the rsync over SSH will be slower. If you don't need encryption, use the rsync daemon:

Code: Select all

> rsync -avh --progress 500M raspberry::mnt/hdd1
For sure I want start samba on my raspberry and make backups data from my network (max: 2-3 devices), but I don't know why my xxx is slow please check my last post here --> viewtopic.php?f=28&t=283569&p=1717435#p1717435, please check section " ### Rsync files over network (cable)" and " ### Rsync ISO file over network (cable)".

LTolledo
Posts: 4106
Joined: Sat Mar 17, 2018 7:29 am
Location: Anime Heartland

Re: Raspberry 4 - NAS, slow copying over the network

Tue Aug 25, 2020 1:41 pm

just sharing....
RPi4B-4G rsync from RPi4NASHDD to local SSD minrate.jpg
RPi4B-4G rsync from RPi4NASHDD to local SSD minrate.jpg (30.28 KiB) Viewed 408 times
RPi4B-4G rsync from RPi4NASHDD to local SSD maxrate.jpg
RPi4B-4G rsync from RPi4NASHDD to local SSD maxrate.jpg (30.71 KiB) Viewed 408 times
RPi4B-4G rsync from RPi4NASHDD to local SSD finished.jpg
RPi4B-4G rsync from RPi4NASHDD to local SSD finished.jpg (40.7 KiB) Viewed 408 times

feel free to interpret these based on your beliefs..... ;)
"Don't come to me with 'issues' for I don't know how to deal with those
Come to me with 'problems' and I'll help you find solutions"

Some people be like:
"Help me! Am drowning! But dont you dare touch me nor come near me!"

Kendek
Posts: 265
Joined: Thu Jul 25, 2019 4:39 pm
Location: Kaposvár, Hungary

Re: Raspberry 4 - NAS, slow copying over the network

Tue Aug 25, 2020 2:15 pm

rsync over SSH; 100% one core CPU usage @ sshd + 40-60% @ rsync --server:

Code: Select all

> rsync -ah --progress root@192.168.1.2:/data/testfile .
root@192.168.1.2's password: 
receiving incremental file list
testfile
          4.25G 100%   59.26MB/s    0:01:08 (xfr#1, to-chk=0/1)
rsync daemon; 70-100% one core CPU usage @ rsync --daemon:

Code: Select all

> rsync -ah --progress 192.168.1.2::data/testfile .
receiving incremental file list
testfile
          4.25G 100%  100.06MB/s    0:00:40 (xfr#1, to-chk=0/1)

janekpolak
Posts: 8
Joined: Sun Aug 23, 2020 8:30 pm

Re: Raspberry 4 - NAS, slow copying over the network

Wed Aug 26, 2020 9:00 am

I did another test, I did a network backup to an SSD drive (Adata su 600), the transfer is the same about 40MB/s.
The same environment (network, router), two different HDD/SSD disk controller.
Where is the problem? Why is it so slow? How debug my case where is bottleneck?

Kendek
Posts: 265
Joined: Thu Jul 25, 2019 4:39 pm
Location: Kaposvár, Hungary

Re: Raspberry 4 - NAS, slow copying over the network

Wed Aug 26, 2020 9:50 am

janekpolak wrote:
Wed Aug 26, 2020 9:00 am
Where is the problem? Why is it so slow? How debug my case where is bottleneck?
You can do a network test with iperf3. These results are mine; Pi4-wire-router-wire-notebook and Pi4-wire-router-WiFi-notebook.

janekpolak
Posts: 8
Joined: Sun Aug 23, 2020 8:30 pm

Re: Raspberry 4 - NAS, slow copying over the network

Wed Aug 26, 2020 8:17 pm

Kendek wrote:
Wed Aug 26, 2020 9:50 am
janekpolak wrote:
Wed Aug 26, 2020 9:00 am
Where is the problem? Why is it so slow? How debug my case where is bottleneck?
You can do a network test with iperf3. These results are mine; Pi4-wire-router-wire-notebook and Pi4-wire-router-WiFi-notebook.
I think network is OK, please see my screenshot. But still too slow. what else can I do?
Maybe the problem is between USB 3.0 and Ethernet port? Firmware, update something? Really I don't know.
cable.png
cable.png (102.75 KiB) Viewed 281 times

Kendek
Posts: 265
Joined: Thu Jul 25, 2019 4:39 pm
Location: Kaposvár, Hungary

Re: Raspberry 4 - NAS, slow copying over the network

Sat Sep 12, 2020 2:43 pm

Rsync, SSH (encryption) and ethernet (interrupts) consumes a lot of CPU.
Try to move eth0 to another processor core, away from USB.

Code: Select all

for i in $(grep eth0 /proc/interrupts | cut -d : -f 1) ; do echo 8 >/proc/irq/$i/smp_affinity ; done
If you just want to transfer files within your local network, then try to use a simpler protocol. I found that SmbFTPD is a very great FTP daemon.

/usr/local/etc/smbftpd.conf

Code: Select all

PidFile "/var/run/smbftpd.pid"
EmptyPasswdLogin Yes
ShowSymlinks Yes
SupportUTF8Client Yes
UsingUTF8FileSystem Yes
AnonymousLogin Yes
AnonymousOnly Yes
AnonymousReadOnly No
/etc/passwd

Code: Select all

ftp:x:0:0::/:/usr/sbin/nologin
/etc/rc.local

Code: Select all

smbftpd -Ds /usr/local/etc/smbftpd.conf
I have created a simple Bash script that uses SmbFTPD (for performance) and rsync daemon (for metadatas) on RPi side, and lftp/rsync on client side. This script is able to synchronize a directory up and down, or download/upload a single file. For file transfers it uses FTP protocol to achieve the best copy speed.

Forget Samba and the copying through mounted shared folder. NFS is better, but the FTP provides the best performance with the least CPU usage on RPi side. The rsync daemon is also fast, and very compatible and convenient, but uses slightly more CPU.

ejolson
Posts: 5980
Joined: Tue Mar 18, 2014 11:47 am

Re: Raspberry 4 - NAS, slow copying over the network

Sat Sep 12, 2020 3:59 pm

janekpolak wrote:
Tue Aug 25, 2020 10:54 am
I have a new disk: Seagate Barracuda Pro
If it's really the Barracuda Pro and not simply the Barracuda, then it will not employ shingled magnetic recording techniques and should not significantly slow down in backup contexts. More about Seagate SMR drives is at

https://www.seagate.com/internal-hard-d ... -smr-list/

If the Seagate drive was purchased in a USB enclosure, you may want to verify what's inside as well as scan the model number using hdparm and smartctl.

Since backups contain public keys and passwords, encryption is important for backups, even if you believe that your local area network is secure. Even if you actively monitor the local network with the tools needed to check security, that only provides a little ground to stand on when claiming it's secure. In particular, one of the ways ransom malware works is by attacking backups first.

My preferred backup scheme launches rsync from the backup server to collect changed files from the other systems at regularly scheduled time intervals. In this case the systems being backed up are never used to log into the server, but only the other way around. It is, of course, possible to set up a secure way for remote systems to push backup images to the server. Unless you also follow data encrypted at rest policies, the backup server needs to be trusted anyway and a pull approach seems easier.

The backup server is a surprisingly high value target. For this reason it should not be running unnecessary services, used for other things or even allow remote login from other machines. One nice thing about a using a Pi as a backup server is that it's cheap enough to devote to that one function. Thus, there is very little incentive from a cost point of view, to use the Pi as some sort of converged do everything appliance, and every reason not to in order to keep backups safe.

If you set up the backups in this way, the Raspberry Pi can schedule them to run in the middle of the night. In that case, the difference between 35 MB/sec versus 100 MB/sec may not matter. In particular, although 35 MB/sec is not much to brag about, it isn't so bad for me to think that anything is broken either.

Return to “Troubleshooting”