Raspberry Pi 3 - enabling hardware crypto support?

[GainfulShrimp]

I've been trying to enable the hardware crypto support that I believe is present in the Cortex-A53 cpu (used in the Pi 3)...

After a struggle (because I've not done much compilation before), I think I've got cryptodev-linux kernel module working, and I have compiled a fresh OpenSSL from source, with the flags to use cryptodev engine.
I have also compiled OpenVPN to use my new OpenSSL, as OpenVPN is what I want to use the accelerated crypto for.

I have two questions:

1. Do my speed test results below indicate that the hardware crypto is indeed enabled (the times for the accelerated version look odd!)?

2. How do I set up a self-compiled copy of OpenVPN so that it can run as a service etc? (I've always previously used apt-get rather than built it myself, so the init script etc have just been sorted out for me...)
Do I need to install it via apt-get and then overwrite it with my own version?

Here are my speed tests - first with the stock OpenSSL:

Code: Select all

pi@raspberrypi3:~ $ openssl speed -evp aes-256-cbc
Doing aes-256-cbc for 3s on 16 size blocks: 5543752 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 64 size blocks: 1629278 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 256 size blocks: 424968 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 1024 size blocks: 107249 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 8192 size blocks: 13438 aes-256-cbc's in 3.00s
OpenSSL 1.0.1k 8 Jan 2015
built on: Tue Mar  1 16:38:12 2016
options:bn(64,32) rc4(ptr,char) des(idx,cisc,16,long) aes(partial) blowfish(ptr)
compiler: -I. -I.. -I../include  -fPIC -DOPENSSL_PIC -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -DL_ENDIAN -DTERMIO -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -Wl,-z,relro -Wa,--noexecstack -Wall -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DAES_ASM -DGHASH_ASM
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256-cbc      29566.68k    34757.93k    36263.94k    36607.66k    36694.70k

And here is the same test using my cryptodev-enabled copy of OpenSSL:

Code: Select all

pi@raspberrypi3:~ $ /usr/local/ssl/bin/openssl speed -evp aes-256-cbc
Doing aes-256-cbc for 3s on 16 size blocks: 596490 aes-256-cbc's in 0.25s
Doing aes-256-cbc for 3s on 64 size blocks: 424524 aes-256-cbc's in 0.24s
Doing aes-256-cbc for 3s on 256 size blocks: 200017 aes-256-cbc's in 0.09s
Doing aes-256-cbc for 3s on 1024 size blocks: 64655 aes-256-cbc's in 0.05s
Doing aes-256-cbc for 3s on 8192 size blocks: 8710 aes-256-cbc's in 0.00s
OpenSSL 1.0.2g  1 Mar 2016
built on: reproducible build, date unspecified
options:bn(64,32) rc4(ptr,char) des(idx,cisc,16,long) aes(partial) idea(int) blowfish(ptr)
compiler: gcc -I. -I.. -I../include  -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -DHAVE_CRYPTODEV -DUSE_CRYPTODEV_DIGESTS -march=armv7-a -Wa,--noexecstack -O3 -Wall -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DAES_ASM -DBSAES_ASM -DGHASH_ASM
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256-cbc      38175.36k   113206.40k   568937.24k  1324134.40k         infk

FYI, here is the version details for the OpenVPN that I've compiled to use my custom OpenSSL:

Code: Select all

pi@raspberrypi3:/opt/openvpn/sbin $ ./openvpn --version
OpenVPN 2.3.10 armv7l-unknown-linux-gnueabihf [SSL (OpenSSL)] [LZO] [EPOLL] [MH] [IPv6] built on Mar 23 2016
library versions: OpenSSL 1.0.2g  1 Mar 2016, LZO 2.09
Originally developed by James Yonan
Copyright (C) 2002-2010 OpenVPN Technologies, Inc. <sales@openvpn.net>
Compile time defines: enable_crypto=yes enable_crypto_ofb_cfb=yes enable_debug=yes enable_def_auth=yes enable_dlopen=unknown enable_dlopen_self=unknown enable_dlopen_self_static=unknown enable_fast_install=yes enable_fragment=yes enable_http_proxy=yes enable_iproute2=no enable_libtool_lock=yes enable_lzo=yes enable_lzo_stub=no enable_management=yes enable_multi=yes enable_multihome=yes enable_pam_dlopen=no enable_pedantic=no enable_pf=yes enable_pkcs11=no enable_plugin_auth_pam=yes enable_plugin_down_root=yes enable_plugins=yes enable_port_share=yes enable_selinux=no enable_server=yes enable_shared=yes enable_shared_with_static_runtimes=no enable_small=no enable_socks=yes enable_ssl=yes enable_static=yes enable_strict=no enable_strict_options=no enable_systemd=no enable_win32_dll=yes enable_x509_alt_username=no with_crypto_library=openssl with_gnu_ld=yes with_lzo_headers=/usr/local/include with_lzo_lib=/usr/local/lib with_mem_check=no with_plugindir='$(libdir)/openvpn/plugins' with_sysroot=no

Thanks in advance for any advice/tips.

[Paul Webster]

I don't have any answers for this but it does look very interesting.
Posting just to register that interest and subscribing to the thread for updates.

[Bony]

I tried to compile version 1.8 of cryptodev-linux using git clone and it failed on 'make ' and 'make install'. Could you explain how to do this please. It certainly appears to be working to me. It is around 4 times faster that my Synology NAS (DS2013) which has a 2GHz ARM5 processor with crypto in hardware.

[GainfulShrimp]

After doing some further tests, I'm not convinced that I've got this working, sadly.

For my latest tests, I used the '-elapsed' option, which means "use wall clock time rather than CPU time". And I used the 'time' command to measure how long the tests took to run, shown at the bottom of each test.

Here is the test using the stock openssl:

Code: Select all

pi@raspberrypi3:~ $ time openssl speed -evp aes-256-cbc -elapsed
You have chosen to measure elapsed time instead of user CPU time.
Doing aes-256-cbc for 3s on 16 size blocks: 5661113 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 64 size blocks: 1636347 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 256 size blocks: 427167 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 1024 size blocks: 107979 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 8192 size blocks: 13531 aes-256-cbc's in 3.00s
OpenSSL 1.0.1k 8 Jan 2015
built on: Tue Mar  1 16:38:12 2016
options:bn(64,32) rc4(ptr,char) des(idx,cisc,16,long) aes(partial) blowfish(ptr)                               
compiler: -I. -I.. -I../include  -fPIC -DOPENSSL_PIC -DOPENSSL_THREADS -D_REENTR                               ANT -DDSO_DLFCN -DHAVE_DLFCN_H -DL_ENDIAN -DTERMIO -g -O2 -fstack-protector-stro                               ng -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -Wl,-z,relro -Wa,--noexe                               cstack -Wall -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM                                -DSHA512_ASM -DAES_ASM -DGHASH_ASM
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256-cbc      30192.60k    34908.74k    36451.58k    36856.83k    36948.65k

real    0m15.029s
user    0m15.020s
sys     0m0.000s

And here is the same test using my 'custom' openssl:

Code: Select all

pi@raspberrypi3:~ $ time /usr/local/ssl/bin/openssl speed -evp aes-256-cbc -elap                               sed
You have chosen to measure elapsed time instead of user CPU time.
Doing aes-256-cbc for 3s on 16 size blocks: 592983 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 64 size blocks: 420447 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 256 size blocks: 198687 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 1024 size blocks: 64264 aes-256-cbc's in 3.00s
Doing aes-256-cbc for 3s on 8192 size blocks: 8658 aes-256-cbc's in 3.00s
OpenSSL 1.0.2g  1 Mar 2016
built on: reproducible build, date unspecified
options:bn(64,32) rc4(ptr,char) des(idx,cisc,16,long) aes(partial) idea(int) blo                               wfish(ptr)
compiler: gcc -I. -I.. -I../include  -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN                                -DHAVE_DLFCN_H -DHAVE_CRYPTODEV -DUSE_CRYPTODEV_DIGESTS -march=armv7-a -Wa,--noe                               xecstack -O3 -Wall -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA2                               56_ASM -DSHA512_ASM -DAES_ASM -DBSAES_ASM -DGHASH_ASM
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-256-cbc       3162.58k     8969.54k    16954.62k    21935.45k    23642.11k

real    0m15.082s
user    0m0.620s
sys     0m14.400s

Openssl seems to be using cryptodev, as the time is all spent in kernel ("sys" time) rather than user time.

But the overall performance appears to be worse than the stock/non-cryptodev openssl.

Maybe I need some sort of kernel driver for the hardware crypto facility, for cryptodev to hook in to?
Perhaps the disappointing result is because I've compiled this on the current Raspian Jessie, which is based on armhf/ARMv7/32-bit and I need arm64/ARMv8/AArch64 to be able to use the new crypto stuff?

All these questions are well outside my level of knowledge, unfortunately... so I'm hoping others can tell me if this is worth persuing, or if I'm barking up the wrong tree?

I tried to compile version 1.8 of cryptodev-linux using git clone and it failed on 'make ' and 'make install'. Could you explain how to do this please.

I'll dig my notes out and try to explain how I did it. (Possibly a classic case of the blind leading the blind though!)

[GainfulShrimp]

Apologies for the delay.

I used a combination of these guides to muddle my way through the kernel compilation process, followed by the cryptodev compilation process. Unlike the advice on these pages though, I did it all on my Pi 3, rather than cross-compiling from a more powerful Linux machine, as they recommend...

http://lostindetails.com/blog/post/Comp ... berry-pi-2
http://bchavez.bitarmory.com/archive/20 ... ry-pi.aspx
http://stackoverflow.com/questions/2016 ... spberry-pi

But, again... I suspect we're wasting our time with this, for now. I think we need an ARMv8/AArch64/arm64 OS before we can truly make use of the Cortex-A53's crypto instructions.
Reading around further, I've learned that the ARMv8 crypto acceleration is based on special instructions (e.g. a similar approach to AESNI on recent Intel CPUs) rather than a hardware module such as some other SoC solutions.
I *believe* that the ARMv8 optimised code is already included in the latest OpenSSL code, but I think you need to be on a proper ARMv8-based OS to make use of them properly.

Just to underline the issue with using cryptodev without proper supporting hardware, here is a benchmark comparison between 'standard'/packaged OpenVPN on my RPi2, and a custom OpenVPN - using cryptodev - on my RPi3. Usually, you'd expect the RPi3 to be faster just from its beefier CPU, but the overhead of cryptodev in this case costs us dearly!

Code: Select all

root@raspberrypi2:~# time openvpn --test-crypto --secret /etc/openvpn/easy-rsa/keys/ta.key --verb 0 --tun-mtu 20000 --cipher aes-256-cbc

real    1m36.805s
user    1m36.770s
sys     0m0.030s

root@raspberrypi3:/opt/openvpn/sbin# time ./openvpn --test-crypto --secret /etc/openvpn/easy-rsa/keys/ta.key --verb 0 --tun-mtu 20000 --cipher aes-256-cbc

real    6m9.189s
user    1m10.330s
sys     4m58.820s

[Bony]

Yes, I see your point it is slower. I was hoping for a speed improvement for openvpn, as well.
I guess this speed-up would also affect https decodes, so it has a wide area of application.
That is every time you use Duckduckgo, and www.raspberrypi.org as a minimum. Perhaps some kind person will compile the proper crypto into the next kernel release so that everybody can use it.

[Bony]

As a comparison, here is the same measurement made on my DS213:

root@VPNServer:~# time openvpn --test-crypto --secret /volume1/@appstore/VPNCenter/etc/openvpn/keys/ta.key --verb 0 --tun-mtu 20000 --cipher aes-256-cbc

real 1m0.356s
user 0m50.690s
sys 0m2.690s

[GainfulShrimp]

That's a useful comparison Bony, thanks.

I've just received a new microSD card today, which I'm going to use to setup an OpenVPN server on my Pi3 using the latest available packages for Raspbian Jessie Lite.
Once I've set it up, I'll share the same benchmark results for that particular combo.

I used to have my home OpenVPN server running on my router (a Ubiquiti Edgerouter Lite ERL-3), but the Pi 2 is *much* faster, in my experience. The Pi 3 should be faster still (even in 32bit mode).

I think we'll have to be patient to see the ARMv8 crypto acceleration in action.
If some clever person/organisation makes an easy-to-use Debian-arm64 image and includes the right instructions/firmware for the Pi 3 to boot in 64bit mode, that would be a fantastic step towards getting it working. But I haven't got the knowledge or skills to do all that myself, unfortunately, even if I had the time...

[Bony]

Here is a final indication of the openssl speed:
root@VPNServer:~# openssl speed -evp aes-256-cbc
Doing aes-256-cbc for 3s on 16 size blocks: 195944 aes-256-cbc's in 0.42s
Doing aes-256-cbc for 3s on 64 size blocks: 180031 aes-256-cbc's in 0.47s
Doing aes-256-cbc for 3s on 256 size blocks: 133113 aes-256-cbc's in 0.38s
Doing aes-256-cbc for 3s on 1024 size blocks: 82842 aes-256-cbc's in 0.23s
Doing aes-256-cbc for 3s on 8192 size blocks: 15263 aes-256-cbc's in 0.01s
OpenSSL 1.0.2g-fips 1 Mar 2016
built on: reproducible build, date unspecified
options:bn(64,32) rc4(ptr,char) des(idx,cisc,16,long) aes(partial) idea(int) blowfish(ptr)
compiler: /usr/local/arm-marvell-linux-gnueabi/bin/arm-marvell-linux-gnueabi-ccache-gcc -I. -I.. -I../include -fPIC -DOPENSSL_PIC -DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN -DHAVE_DLFCN_H -Wa,--noexecstack -DSYNOPLAT_F_ARMV5 -DSYNO_MARVELL_88F6281 -O2 -DBUILD_ARCH=32 -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -DSYNO_PLATFORM=MARVELL_88F6281 -DSYNO_RUNNING_DSM_BUILD_SYSTEM -g -pipe -fstack-protector --param=ssp-buffer-size=4 -Wformat -Wformat-security -D_FORTIFY_SOURCE=2 -O2 -Wno-unused-result -DL_ENDIAN -DHAVE_CRYPTODEV -mcpu=marvell-f -Wl,-z,relro -Wl,--as-needed -Wl,--no-undefined -O3 -Wall -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_GF2m -I/usr/local/arm-marvell-linux-gnueabi/arm-marvell-linux-gnueabi/libc/usr//include -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DAES_ASM -DBSAES_ASM -DGHASH_ASM
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-256-cbc 7464.53k 24514.86k 89676.13k 368826.99k 12503449.60k
root@VPNServer:~#
This is for a 2GHz ARM5 with hardware crypto.

I think I shall wait until cryptodev is added to the kernel and openssl and openvpn have been compiled with ARMv8. I have just got FTTC and now have a 9Mb/s upload speed, I was also going to go for elliptic curve diffie-hellman to try to speed-up the key exchange. I think I am going to find it easier in a few months perhaps.

[deekue]

Suse have published a 64 bit OS image for the Rpi3 http://betanews.com/2016/11/22/64-bit-o ... pberry-pi/

I want to run Raspbian but this at least means there's a functional 64 bit kernel available

[deekue]

but then this post indicates the RPi3 CPU doesn't actually have the AES & SHAx instructions
https://www.phoronix.com/forums/forum/h ... post861842

[todorcolov]

Have you enabled the HW accelerator for aes?
To use crypto-dev, you must have enabled the HW AES modules in the linux kernel.

Can you paste the following output:
cat /proc/crypto
lsmod

[ghans]

No Raspberry Pi has hardware acceleration for crypto operations , or at least not via
the ARMv8 crypto instruction set.

ghans

[rpdom]

Old post.