OpenSSL has ARM assembly code to accelerate encryption/decryption of the most important ciphers. It's enabled in OpenSSL's default ARM configurations, but unfortunately not in Debian's or Raspbian's OpenSSL package. These optimizations have a *huge* impact.
A Debian bug includes a patch to enable these optimizations, and also some benchmarks. Needless to say, I'd really like to see them enabled by default in Raspbian. I've been using OpenSSL compiled with those optimizations for a while, and didn't have any problems. I definitely saw a nice speedup in various applications, e.g. ssh/scp.