jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 28920
Joined: Sat Jul 30, 2011 7:41 pm

How to check the speed of encryption

Mon Jun 14, 2021 5:42 pm

I don't often start new posts, but I've been pondering the performance impact of only having SW encryption on the Pi range, and how to find out what sort of performance hit that gives.

I've been comparing rcp against scp, to null, to SD card, to USB3 drives etc. I'm basically seeing no difference between rcp and scp, if fact the figures jump around a bit, sometimes scp is faster, sometimes rcp. Finding replicatable results is a bit hit and miss.

I've also tried copying files to encrypted folders vs non-encrypted, I do see a tiny slow down, but there isn't much in it.

Now, I would expect some sort of performance hit, so I am wondering if my testing process isn't showing up the difference. Has anyone really seen any performance impact of having to do encryption in SW? If so, what circumstances showed it up?
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Working in the Application's Team.

bls
Posts: 1564
Joined: Mon Oct 22, 2018 11:25 pm
Location: Seattle, WA

Re: How to check the speed of encryption

Mon Jun 14, 2021 5:53 pm

I followed the earlier note about SW encryption on the Pi potentially being a performance issue, and was thinking about spinning up an IPSEC VPN to do some testing against straight network. Will try to get to that today or tomorrow.

As far as rcp and scp, maybe I'm missing something, but it appears that they are exactly the same binary.

Code: Select all

pisrv1~# which rcp
/usr/bin/rcp
pisrv1~# l /usr/bin/rcp
lrwxrwxrwx 1 root root 21 Sep 25  2019 /usr/bin/rcp -> /etc/alternatives/rcp
pisrv1~# l /etc/alternatives/rcp
lrwxrwxrwx 1 root root 12 Sep 25  2019 /etc/alternatives/rcp -> /usr/bin/scp
Does the code do something different based on the command that started it?
Pi tools:
Quickly and easily build customized-just-for-you SSDs/SD Cards: https://github.com/gitbls/sdm
Easily run and manage your network's DHCP/DNS servers on a Pi: https://github.com/gitbls/ndm
Easy and secure strongSwan VPN installer/manager: https://github.com/gitbls/pistrong
Lightweight Virtual VNC Config: https://github.com/gitbls/RPiVNCHowTo

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 28920
Joined: Sat Jul 30, 2011 7:41 pm

Re: How to check the speed of encryption

Mon Jun 14, 2021 9:37 pm

bls wrote:
Mon Jun 14, 2021 5:53 pm
I followed the earlier note about SW encryption on the Pi potentially being a performance issue, and was thinking about spinning up an IPSEC VPN to do some testing against straight network. Will try to get to that today or tomorrow.

As far as rcp and scp, maybe I'm missing something, but it appears that they are exactly the same binary.

Code: Select all

pisrv1~# which rcp
/usr/bin/rcp
pisrv1~# l /usr/bin/rcp
lrwxrwxrwx 1 root root 21 Sep 25  2019 /usr/bin/rcp -> /etc/alternatives/rcp
pisrv1~# l /etc/alternatives/rcp
lrwxrwxrwx 1 root root 12 Sep 25  2019 /etc/alternatives/rcp -> /usr/bin/scp
Does the code do something different based on the command that started it?
rcp should be unencrypted, scp should be encrypted. I have no idea how they might do different things depending on invocation.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Working in the Application's Team.

bls
Posts: 1564
Joined: Mon Oct 22, 2018 11:25 pm
Location: Seattle, WA

Re: How to check the speed of encryption

Mon Jun 14, 2021 10:09 pm

jamesh wrote:
Mon Jun 14, 2021 9:37 pm
bls wrote:
Mon Jun 14, 2021 5:53 pm
I followed the earlier note about SW encryption on the Pi potentially being a performance issue, and was thinking about spinning up an IPSEC VPN to do some testing against straight network. Will try to get to that today or tomorrow.

As far as rcp and scp, maybe I'm missing something, but it appears that they are exactly the same binary.

Code: Select all

pisrv1~# which rcp
/usr/bin/rcp
pisrv1~# l /usr/bin/rcp
lrwxrwxrwx 1 root root 21 Sep 25  2019 /usr/bin/rcp -> /etc/alternatives/rcp
pisrv1~# l /etc/alternatives/rcp
lrwxrwxrwx 1 root root 12 Sep 25  2019 /etc/alternatives/rcp -> /usr/bin/scp
Does the code do something different based on the command that started it?
rcp should be unencrypted, scp should be encrypted. I have no idea how they might do different things depending on invocation.
Indeed...should. I located one source at https://github.com/openssh/openssh-port ... ster/scp.c but it just says "This is basically patched BSD rcp which uses ssh to do the data transfer (instead of using rcmd)."

If this is the definitive source then invoking using rcp simply uses scp and ssh, which would make some sense, since there is no rcp/rcmd listener process running on the remote system.

In fact, using rcp from one pi to another appears to start an ssh session, supporting the theory that rcp is just an alias for scp to help fingers programmed for rcp do "the right thing". :roll:
Pi tools:
Quickly and easily build customized-just-for-you SSDs/SD Cards: https://github.com/gitbls/sdm
Easily run and manage your network's DHCP/DNS servers on a Pi: https://github.com/gitbls/ndm
Easy and secure strongSwan VPN installer/manager: https://github.com/gitbls/pistrong
Lightweight Virtual VNC Config: https://github.com/gitbls/RPiVNCHowTo

bls
Posts: 1564
Joined: Mon Oct 22, 2018 11:25 pm
Location: Seattle, WA

Re: How to check the speed of encryption

Mon Jun 14, 2021 10:49 pm

I don't have enough time to test over a VPN today, but here's an interesting test. I copied a 5+GB ISO from an X64 Ubuntu server to an SSD-based Pi4 twice, once using cp over NFS (which is NOT encrypted) and once using scp (which IS encrypted).

Both systems were idle.

Code: Select all

p83/tmp# time cp /k/win10/Win10_1909_English_x64.iso . ; pitemp

real    1m3.903s
user    0m0.031s
sys     0m38.799s
temp=50.1'C
p83/tmp# pitemp
temp=46.2'C
p83/tmp# rm -f Win10_1909_English_x64.iso 
p83/tmp# pitemp
temp=45.7'C
p83/tmp# time scp mondo:/k/win10/Win10_1909_English_x64.iso . ; pitemp
Win10_1909_English_x64.iso                                                                                                              100% 5170MB  45.9MB/s   01:52    

real    1m53.118s
user    1m1.351s
sys     1m28.398s
temp=52.1'C
p83/tmp# pitemp
temp=47.7'C
p83/tmp# rm -f Win10_1909_English_x64.iso 
p83/tmp# pitemp
temp=46.7'C
p83/tmp# time cp /k/win10/Win10_1909_English_x64.iso . ; pitemp

real    0m56.228s
user    0m0.061s
sys     0m38.633s
temp=49.6'C
p83/tmp# rm -f Win10_1909_English_x64.iso 
p83/tmp# pitemp
temp=46.7'C
p83/tmp# time scp mondo:/k/win10/Win10_1909_English_x64.iso . ; pitemp
Win10_1909_English_x64.iso                                                                                                              100% 5170MB  43.7MB/s   01:58    

real    1m58.854s
user    1m0.109s
sys     1m36.929s
temp=50.6'C

The elapsed time increased by nearly 50%, and the user and kernel times increased significantly as well.

It's pretty clear that scp, which is encrypted, uses a LOT more CPU time to do the copy.
Pi tools:
Quickly and easily build customized-just-for-you SSDs/SD Cards: https://github.com/gitbls/sdm
Easily run and manage your network's DHCP/DNS servers on a Pi: https://github.com/gitbls/ndm
Easy and secure strongSwan VPN installer/manager: https://github.com/gitbls/pistrong
Lightweight Virtual VNC Config: https://github.com/gitbls/RPiVNCHowTo

incognitum
Posts: 762
Joined: Tue Oct 30, 2018 3:34 pm

Re: How to check the speed of encryption

Mon Jun 14, 2021 11:48 pm

Not sure if encryption will slow down network communication that much in practice, as 1 gigabit is relatively slow, and can also become the bottleneck before CPU does.

When using something with faster bandwidth -like USB storage- you do can see that it is only is capable of reading at 110 MByte/sec from encrypted storage, while it does more than double that if unencrypted.


Pi 400 with 64-bit kernel and Samsung T7 SSD.

Without encryption:

Code: Select all

pi@raspberrypi:~ $ bonnie++
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.98       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
raspberrypi   7424M  137k  99  270m  94  120m  51  809k  99  273m  42  5980 188
Latency             91662us     111ms     431ms   10448us   16387us   10235us
Version  1.98       ------Sequential Create------ --------Random Create--------
raspberrypi         -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 16384  94 +++++ +++ 16384  91 16384  94 +++++ +++ 16384  91
Latency               615us     664us    1399us     519us      45us    1734us
1.98,1.98,raspberrypi,1,1623712786,7424M,,8192,5,137,99,276846,94,122658,51,809,99,279636,42,5980,188,16,,,,,16127,94,+++++,+++,19256,91,16428,94,+++++,+++,18292,91,91662us,111ms,431ms,10448us,16387us,10235us,615us,664us,1399us,519us,45us,1734us
With LUKS full disk encryption: ("encrypt disc" checkbox in Berryboot)

Code: Select all

pi@raspberrypi:~ $ bonnie++
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.98       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
raspberrypi   7424M  136k  99  207m  78 76.9m  26  784k  99  110m  20  5115 156
Latency               121ms     971ms     972ms   11007us   16644us    8145us
Version  1.98       ------Sequential Create------ --------Random Create--------
raspberrypi         -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 16384  91 +++++ +++ 16384  88 16384  90 +++++ +++ 16384  87
Latency               602us     665us    1479us     507us      52us    1116us
1.98,1.98,raspberrypi,1,1623709958,7424M,,8192,5,136,99,212176,78,78765,26,784,99,113037,20,5115,156,16,,,,,15653,91,+++++,+++,19079,88,15934,90,+++++,+++,17809,87,121ms,971ms,972ms,11007us,16644us,8145us,602us,665us,1479us,507us,52us,1116us
(Not sure why writing is less difference. E.g. if writing figures are distorted by caching, or that it is able to use more than one core with that).

cleverca22
Posts: 3895
Joined: Sat Aug 18, 2012 2:33 pm

Re: How to check the speed of encryption

Tue Jun 15, 2021 12:55 am

it also depends on which encryption algo your using

Code: Select all

[clever@amd-nixos:~]$ openssl speed
...
Doing rc4 for 3s on 16 size blocks: 107696599 rc4's in 2.99s
Doing rc4 for 3s on 64 size blocks: 36617020 rc4's in 3.00s
Doing rc4 for 3s on 256 size blocks: 9933271 rc4's in 3.00s
Doing rc4 for 3s on 1024 size blocks: 2559330 rc4's in 2.99s
Doing rc4 for 3s on 8192 size blocks: 323474 rc4's in 3.00s
Doing rc4 for 3s on 16384 size blocks: 160504 rc4's in 3.00s
Doing des cbc for 3s on 16 size blocks: 11612148 des cbc's in 3.00s
Doing des cbc for 3s on 64 size blocks: 3016880 des cbc's in 3.00s
Doing des cbc for 3s on 256 size blocks: 760797 des cbc's in 3.00s
Doing des cbc for 3s on 1024 size blocks: 191139 des cbc's in 3.00s
Doing des cbc for 3s on 8192 size blocks: 23917 des cbc's in 3.00s
Doing des cbc for 3s on 16384 size blocks: 11966 des cbc's in 3.00s
Doing des ede3 for 3s on 16 size blocks: 4413332 des ede3's in 3.00s
Doing des ede3 for 3s on 64 size blocks: 1119151 des ede3's in 3.00s
Doing des ede3 for 3s on 256 size blocks: 281487 des ede3's in 3.00s
Doing des ede3 for 3s on 1024 size blocks: 70558 des ede3's in 3.00s
Doing des ede3 for 3s on 8192 size blocks: 8813 des ede3's in 3.00s
Doing des ede3 for 3s on 16384 size blocks: 4405 des ede3's in 3.00s
Doing aes-128 cbc for 3s on 16 size blocks: 30109263 aes-128 cbc's in 3.00s
Doing aes-128 cbc for 3s on 64 size blocks: 7920369 aes-128 cbc's in 3.00s
Doing aes-128 cbc for 3s on 256 size blocks: 2031437 aes-128 cbc's in 3.00s
Doing aes-128 cbc for 3s on 1024 size blocks: 505577 aes-128 cbc's in 3.00s
Doing aes-128 cbc for 3s on 8192 size blocks: 63472 aes-128 cbc's in 3.00s
Doing aes-128 cbc for 3s on 16384 size blocks: 31908 aes-128 cbc's in 3.00s
...
thats all using the routines within openssl, which are then used by all other crypto programs

Code: Select all

[clever@amd-nixos:~]$ ssh -v system76
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
and i think my ssh client is using chacha20-poly1305 to encrypt things

Code: Select all

[clever@amd-nixos:~]$ openssl speed -evp chacha20-poly1305
Doing chacha20-poly1305 for 3s on 16 size blocks: 27406692 chacha20-poly1305's in 2.99s
Doing chacha20-poly1305 for 3s on 64 size blocks: 12991022 chacha20-poly1305's in 3.00s
Doing chacha20-poly1305 for 3s on 256 size blocks: 10424408 chacha20-poly1305's in 3.00s
Doing chacha20-poly1305 for 3s on 1024 size blocks: 3058188 chacha20-poly1305's in 3.00s
Doing chacha20-poly1305 for 3s on 8192 size blocks: 414411 chacha20-poly1305's in 3.00s
Doing chacha20-poly1305 for 3s on 16384 size blocks: 209732 chacha20-poly1305's in 3.00s
OpenSSL 1.1.1j  16 Feb 2021
built on: Tue Feb 16 15:24:01 2021 UTC
options:bn(64,64) rc4(8x,int) des(int) aes(partial) idea(int) blowfish(ptr) 
compiler: gcc -fPIC -pthread -m64 -Wa,--noexecstack -Wall -O3 -DOPENSSL_USE_NODELETE -DL_ENDIAN -DOPENSSL_PIC -DOPENSSL_CPUID_OBJ -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DKECCAK1600_ASM -DRC4_ASM -DMD5_ASM -DAESNI_ASM -DVPAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DX25519_ASM -DPOLY1305_ASM -DNDEBUG
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
chacha20-poly1305   146657.88k   277141.80k   889549.48k  1043861.50k  1131618.30k  1145416.36k
which you can then benchmark alone like so

my pi isnt currently plugged in, but its simple enough to run the same commands on a pi and then compare models and see how those numbers compare to the network speeds

swampdog
Posts: 718
Joined: Fri Dec 04, 2015 11:22 am

Re: How to check the speed of encryption

Tue Jun 15, 2021 3:48 am

My main PC died so can't do any tests but probably a good comparison would be comparing NFS against SSH <-> a decent NAS using some known data sets, zero, random, text, etc. I guess the ssh key being used needs to be taken into account.

Back in the day when my upload speed was slow I used to tunnel back home from work using "ssh -C" and have a tolerable rdp session onto win2003 boxes. As the manual says though "-C" only slows things down if the network is fast enough. Indeed, when my upload speed doubled I was much better off without it.

Perhaps rsync also. Maybe even just have the rpi talk to itself? Actually that is a test I can do..

Code: Select all

foo@pi18:~ $ dd if=/dev/urandom of=/wrk/z bs=1024 count=1048576
foo@pi18:~ $ time scp /wrk/z foo@localhost:/dev/null
z                                             100% 1024MB  41.3MB/s   00:24    

real	0m27.708s
user	0m17.193s
sys	0m11.985s

foo@pi18:~ $ time scp /wrk/z foo@localhost:/dev/null
z                                             100% 1024MB  41.2MB/s   00:24    

real	0m25.171s
user	0m16.940s
sys	0m12.523s

foo@pi18:~ $ time scp /wrk/z foo@localhost:/dev/null
z                                             100% 1024MB  41.1MB/s   00:24    

real	0m25.241s
user	0m17.153s
sys	0m12.368s

foo@pi18:~ $ time cp -v /wrk/z /dev/null
'/wrk/z' -> '/dev/null'

real	0m1.766s
user	0m0.002s
sys	0m1.763s

foo@pi18:~ $ time cp -v /wrk/z /dev/null
'/wrk/z' -> '/dev/null'

real	0m1.773s
user	0m0.001s
sys	0m1.771s

foo@pi18:~ $ time cp -v /wrk/z /dev/null
'/wrk/z' -> '/dev/null'

real	0m1.742s
user	0m0.011s
sys	0m1.729s
..standard (aka ssh-keygen -t rsa) key. I'm too sleepy to figure out how to get rsync to write to /dev/null.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 28920
Joined: Sat Jul 30, 2011 7:41 pm

Re: How to check the speed of encryption

Tue Jun 15, 2021 10:25 am

Thanks all, I was at a loss to explain why rcp was the same as scp, but seems clear if they are now using the same backend. Will adjust my tests accordingly.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Working in the Application's Team.

LTolledo
Posts: 5534
Joined: Sat Mar 17, 2018 7:29 am
Location: Anime Heartland

Re: How to check the speed of encryption

Tue Jun 15, 2021 12:17 pm

Hi jamesh..

as the user base of RPi is greatly increasing...
will this feature (encryption) be implemented as an option in future OS releases?

a "could be" or "maybe", or "not being considered" answer is good enough for me :D
(and after answering ....no need to elaborate further...) ;)
"Don't come to me with 'issues' for I don't know how to deal with those
Come to me with 'problems' and I'll help you find solutions"

Some people be like:
"Help me! Am drowning! But dont you dare touch me nor come near me!"

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 28920
Joined: Sat Jul 30, 2011 7:41 pm

Re: How to check the speed of encryption

Tue Jun 15, 2021 2:05 pm

LTolledo wrote:
Tue Jun 15, 2021 12:17 pm
Hi jamesh..

as the user base of RPi is greatly increasing...
will this feature (encryption) be implemented as an option in future OS releases?

a "could be" or "maybe", or "not being considered" answer is good enough for me :D
(and after answering ....no need to elaborate further...) ;)
You can already use encryption, it's just that it is done in software, not HW as the ARM core in the 2711 (and older) SoC's do not have that particular feature included. What I am trying to do is nail down the performance impact of having to do it in software. So its not a question of OS release, it a question of will future Pi have HW encryption. Which I cannot answer.

If you want to encrypt folders, I was using fscrypt - pretty easy to use.

These are my notes.
sudo apt install fscrypt
sudo apt install libpam-fscrypt

# Enable encryption on the device

sudo fscrypt
sudo fscrypt setup /

sudo fscrypt <name of folder to encrypt>

I selected [2] and added a new passphrase


Use

fscrypt unlock <folder>

fscrypt lock <folder>
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Working in the Application's Team.

User avatar
jojopi
Posts: 3490
Joined: Tue Oct 11, 2011 8:38 pm

Re: How to check the speed of encryption

Tue Jun 15, 2021 2:56 pm

jamesh wrote:
Tue Jun 15, 2021 10:25 am
Thanks all, I was at a loss to explain why rcp was the same as scp, but seems clear if they are now using the same backend.
You just have not installed any rlogin packages, so the rcp symlink is pointing to scp ?

Installing rsh-client will provide netkit-rcp and should change the symlink via update-alternatives.

Obviously you would need a server too. It is good that these packages are not installed by default in modern distributions.

LTolledo
Posts: 5534
Joined: Sat Mar 17, 2018 7:29 am
Location: Anime Heartland

Re: How to check the speed of encryption

Tue Jun 15, 2021 9:54 pm

jamesh wrote:
Tue Jun 15, 2021 2:05 pm
LTolledo wrote:
Tue Jun 15, 2021 12:17 pm
Hi jamesh..

as the user base of RPi is greatly increasing...
will this feature (encryption) be implemented as an option in future OS releases?

a "could be" or "maybe", or "not being considered" answer is good enough for me :D
(and after answering ....no need to elaborate further...) ;)
You can already use encryption, it's just that it is done in software, not HW as the ARM core in the 2711 (and older) SoC's do not have that particular feature included. What I am trying to do is nail down the performance impact of having to do it in software. So its not a question of OS release, it a question of will future Pi have HW encryption. Which I cannot answer.

If you want to encrypt folders, I was using fscrypt - pretty easy to use.

These are my notes.
sudo apt install fscrypt
sudo apt install libpam-fscrypt

# Enable encryption on the device

sudo fscrypt
sudo fscrypt setup /

sudo fscrypt <name of folder to encrypt>

I selected [2] and added a new passphrase


Use

fscrypt unlock <folder>

fscrypt lock <folder>
thanks for answering....
got more than what I asked :D
"Don't come to me with 'issues' for I don't know how to deal with those
Come to me with 'problems' and I'll help you find solutions"

Some people be like:
"Help me! Am drowning! But dont you dare touch me nor come near me!"

Return to “Advanced users”