If you want to use your Pi as a server you might want to tweak your settings and OS a little. The default Raspbian image is build to display all the Pi has to offer and that's a load more than you want on your server. There are several mini Raspbian images like
Minibian which has only a minimal selection of software, it doesn't even include a graphical desktop environment. You won't need that on a server anyways since it's easy to run commands and access files using SSH.
On this forum you can find tutorials for setting up any type of server you like using mostly terminal commands so there's no need for a desktop anyway.
If you don't use a desktop you can also dedicate more GPU to the system itself which makes it way faster. This can be configured using:
And if you've got a fast USB key why not use it as a swap drive??... Just set the swappiness as explained in previous posts and keep the swap partition's size below 512MB for more won't be used.
Advanced Users and Daredevils:
Another good tip is to check what packages will be installed every time you apt-get something. If you see a lot of packages you don't need apply
--no-install-recommends to your apt-get command and check again. Be sure to check the previous selections to make sure you're not missing anything before you commit the changes. Specialy on webservers you can save tons of disk space and unneeded daemons starting automatically at boot. (disabling daemons you don't use is also a good mem saver)