True, that is generally the down-side of source based distributions, and also the reason why I generally recommend installing via a stage 4 tarball
(otherwise you either have to cross compile or copy the sources for the WiFi/Ethernet software over USB to your Pi before you can connect, and then compile some).
Depending on how many and what libraries you need and how often you have to update, compilation time may not be that big an issue after your initial install. I myself use a lot of python libraries, and save for some of the lapack packages many do not require any compilation at all (the files just get copied, meaning it's as fast as binary distributions, only you get live packages instead of nightyl/weekly binaries) ^^.
RPi and Gentoo user. Check out my overview article on Gentoo for the RPi: http://tutorials.chymera.eu/blog/2014/05/21/gentoo-for-the-raspberry-pi/