I think that it is a great idea, but not with Unity as the default desktop.
I got my Raspberry running Debian fairly easily, but have hit a couple of glitches, which I think are down to the nature of Debian as much as anything. If I want to do non-multimedia stuff on a computer, I go for Debian. But, for a lot of multimedia stuff, simply getting it to work can be a nightmare on Debian, whereas that stuff often works out of the box with Ubuntu.
I use XFCE, because I am not that keen on LXDE. Guess it is because I am a crusty Gnome 2.x die-hard. You don't have to run Unity, that is what Xubuntu is for, so there's no reason why you can't run a lightweight version of Ubuntu, rather than Unity, Gnome3, KDE, etc.
I eventually got sound via the 3.5mm jack working, so I can use it to play the music from my file server through the stereo without cables going across the room, and control it from a laptop, tablet or mobile phone. A few kinks to iron out: such as why the Edimax WiFi dongle won't start unless the Pi is plugged into the router via ethernet cable. Not finding it easy to figure that one out.
The point is, though, to run it remotely without cable, or monitor. I suspect it is a power issue, although I bought this dongle because it is supposed to work without the need for an external usb hub and (apart from the temporary ethernet cable & phono jack), is the only other thing plugged in now.
The audio clicks now and then, which is a bit annoying. But, not a bad little music player. One could use an external hard-drive instead, of course, but that is not the point, it is meant to be there to play music in the living room which sits on the file server in the office. The audio libraries and codecs in Debian ARM look a bit of a mess (get nervous about installing gstreamer libraries with names that end in 'ugly' and 'bad'...), although they may be a mess on the x86 branch as well for all I know, because I never usually use Debian for multimedia, as most things just work out of the box with Ubuntu.
I remember the hassle of getting Broadcom WiFi working on my old celeron laptop using an NDIS wrapper - still use the same laptop, got WiFi working under arch on it, and while Debian was still a hassle to get WiFi working (although at least not having to use wrappers), Ubuntu just does it without any problem - just as it does multimedia stuff.
So, maybe an Ubuntu/XFCE version of the GNU/linux-debian ARM port for the Raspberry would be nice, although that would limit what one might learn while setting it up. Which kind of defeats the purpose of the RPi, doesn't it? It would be nice though, so that when you've finished messing about with it, you could have something that works without needing fiddling with (for example, where the WiFi dongle doesn't entail having to have the ethernet plugged in for some strange reason, and where if you do pull the WiFi dongle out it doesn't kill the power to the RPi, and where you can listen to audio through a 3.5mm jack without crackles between tracks...).