I am actually not gunning for any VR effect at all, at least initially.
Having a HMD for a raspberry pi using an android phone and a $15 vr headset is the initial goal. Of course, something like that won't be super useful for people who can't touch-type. Seems to me being able to drop an app on an android phone, put on a cheap VR headset and then VNC into a pi might prove handy to more than a few people.
Anyway, where it goes beyond that is not important presently. If interacting with the desktop is simply untenable, time to bail- if it's acceptable, then I will expand upon it.
I have connected the phone to the PI using VNC, and then dropped the phone into the VR headset (sans stereoscopic capability)- the view of the desktop is actually pretty decent over wireless.
I have also connected he phone to the with VNC, started up the camera on the pi and held it up, then held it against the phone and moved around to emulate vr headset use, and see how laggy the view felt in this fashion. I think it would do as a pass-through, though there's definitely a lag.