Well, it's not actually a problem. I use my Raspberry Pi for a whole bunch of different stuff, so it helps to keep it just beside my router, along with an external hd and some other goodies. One of the things I use it, however, is to test the library I work on (SDLU)
http://bitbucket.org/sdlu/sdlu, an extension library to SDL2. For that, I need proper OpenGL ES 2 support. SDL2 for the Raspberry Pi does not build with GLX (at least by default). So, my VNC connection via tightvnc to the X server is of no use.
Moving the rpi from the router and hooking it up to my TV is at least 10 minutes of work (while the real testing takes about 1 minute). So, I want to somehow access the framebuffer remotely, and do the testing from there. Hope I've made my setup understood.