There are a number of us with experience in X when it was known as Project Athena at MIT and GL going back to Original Flavor (on the very first SGI IRIS workstations).
OpenGL is the correct route since it"s already widely used in many other environments, and there are benefits for some applications to having 2-D X FBs that can be further manipulated in 3-D (think about being able to rotate the face of a cube that has X and other active or saved screen views on each face as the view is changed, e.g., various desktop environments).
All text typeface (of which font family is just one element) and graphics operations must be performed within the GPU portion of RAM since video comes out of the GPU directly. The only code that should be in ARM CPU RAM is that which doesn"t perform any graphics functionality and instead makes OpenGL calls into the GPU.
Has anyone looked at GLX, the OpenGL extensions for X:
http://www.opengl.org/sdk/docs.....XIntro.xml
I believe all of the GL-side work is done, and integration of the particular X server in the Linux distros probably needs to be performed, if we"re inheriting X down trees in which GLX hasn"t been integrated.
I haven"t dealt with this for quite some time and that wasn"t for ARM, but, if there are particular problems that can"t be figured out, I and others with experience can help. If we need more head count to do the grunt work, I can see if we can pull in folks who have done this for many other platforms - they would be much faster than people who haven"t done this. Failing that in our desired timeframe, perhaps we can crowd-source it among all those college/university students around the world who seem to have unlimited quantities of time, hypercaffeinated beverages, and a lack of need for sleep, the opposite sex, etc. iOS developers who have done OpenGL ES work (e.g., games) would probably be most useful.
The best things in life aren't things ... but, a Pi comes pretty darned close!

"Education is not the filling of a pail, but the lighting of a fire." -- W.B. Yeats
In theory, theory & practice are the same - in practice, they aren't!!!