Though this doesn't solve the issue of I/O on a single Pi, I have been quite successful using two Pies, one for input (USB) and one for output and then networking them. I have used both netsend~ and streamout~ for this and I get little to no latency with decent quality. Still a few slips and crackle...