First thing, this Linux will probably not have an ASIO driver yet (or something similar).
If it does, the latency of this device will be significantly higher than from a PC, why?, because the CPU has very little, to no L-cache! Also the cpu is pretty slow.
Good enough to run a single instrument through, but once you start to have polyphonic, multi layer instruments, that have a variety of analog control settings; like attack and sustain variables depending on how hard you hit a key and keep it pressed in, this CPU might not handle the stress.
Even then it would be advisable, to find a flash drive (non SD) that has high iops, and connect it via a bus other than USB. USB will increase latency considerably too!
But it is possible. If you want best results, you'd probably have to plug in a fast USB stick into the USB port, without HUB, as sample drive, and connect your USB midi device to the other port (or connect a HUB inbetween so you can add a mouse and keyboard, and other low bandwidth devices).
I think the biggest issue is finding software that will run on the raspberry's os, and preferably software that is optimized for the pi. Also software that has a 'learn' mode for midi knobs, and a midi controller that will be recognized out of the box!
Some of the more advanced midi tools over USB need drivers. You don't want to run wine on the raspberry (doesn't work I heard, and even if it does, would be too slow).