I just started looking at a similar personal project for the Pi: Using "jackaudio" and "netjack".
jackaudio is a low latency audio layer, somewhat similar to pulse audio, that has an available plug in called "netjack" (there are actually two different versions of "netjack", not to be confused with the fact that there are also two different versions of jackaudio itself!), that should be able to provide this functionality as well.
In my case, I'm hoping to implement a wireless digital "audio snake" using two RPi's with Wifi, and two outboard USB audio devices such as "Presonus Audiobox" products (I've successfully used an "Audiobox 1818VSL under Linux and jack in another project).
I just started working on this under Raspbian, where installing "jack" was simply an apt-get away, but can not get jackd to start (actually to stay running) using the built in audio of the RPi. I'll be trying it with some outboard USB audio devices soon, as I am guessing the the RPi audio implementation may not be 100% ALSA compliant.
"Jackaudio" has been ported to other OS's. I once tried getting it running under Windows without success however.
Granted using "jackaudio" may be a more complicated solution than what you want, but it is a very versatile solution as lots of other software play nicely with jack and jack can nicely mix and route audio throughout its "clients" in a manor similar to a traditional physical "patchbay" which is its original intention.