Simon's accelerated X development thread


406 posts   Page 4 of 17   1, 2, 3, 4, 5, 6, 7 ... 17
by spurious » Sun May 20, 2012 10:25 am
This sounds really interesting work teh_orph.
Have you put the code anywhere public, so people can have a look.
And keep up the good work!
Posts: 343
Joined: Mon Nov 21, 2011 9:29 pm
by theHetman » Sun May 20, 2012 2:40 pm
I'm also following this project with interest and wish I could help more but my Linux knowledge is minimal and my understanding of the various graphics APIs is very outdated.

Now that the forum has been spread out, with many more topics perhaps this thread should be moved to Projects / Graphics, audio and Multimedia. There is nothing in that topic yet.
Posts: 78
Joined: Tue Jan 10, 2012 5:42 pm
by teh_orph » Sun May 20, 2012 3:45 pm
Yep sounds good. However this thread was primarily a call out to other people working on the project. Perhaps I should start a development thread elsewhere.
spurious wrote:This sounds really interesting work teh_orph.
Have you put the code anywhere public, so people can have a look.
And keep up the good work!

I would like to do this however I'm not sure where to host. I can get a free git through my github account, but I've never used that version control system before. Also many of the important parts of the project are simply patches for existing libraries (with specific version numbers). How should I host this? Suggestions please!
User avatar
Posts: 345
Joined: Mon Jan 30, 2012 2:09 pm
Location: London
by Hexxeh » Sun May 20, 2012 3:50 pm
Probably a series of GitHub repos is best. If you can find existing Git repos for the versions you've patched, fork those, and then add your patches as a separate commit on top. That way anyone who wants a patch can generate one from the commit diff, any anyone who wants the patched version can just clone it.

If you need help with any of this, let me know.
Posts: 90
Joined: Thu Apr 05, 2012 3:07 pm
by KeithSloan » Sun May 20, 2012 5:04 pm
Well I wish you luck. But have my doubts.
I once worked on some software which was to split an X-Server into two parts. A C++ part and a Java part. i.e. Make the X-Server also client server. My experience was that. The X protocol ( what gets sent from X-Client to X-Server ) is not efficient. It was designed by computer scientists with their theory hats on and their performance hats off. X Client applications don't care about performance. Take the sample X Client X-Calc the way X protocol works each icon ( Calculator button ) is a Window and each time the mouse gets moved into and out of an icon/window there is the potential of a message being sent server to client. Could the programmer have disabled this message - yes, but do they No!!! Why Not - because programmers are sufficiently removed from the protocol i.e. what gets sent between the client and server that they don't care only worry that their program works. Is the default to send the message - Yes because the application may want to change the look of the icon/window when the mouse enters or leaves. Will offloading to a GPU help. Not in this case as its the passing of messages between Client and Server that is the problem. Don't believe me, run an xtrace and look at the superfluous messaging going on.
Re GPU there is a cost of involving the GPU in the case of 3D there is sufficient saving to justify the extra initial cost. With 2D there will be savings but not as much as with 3D. Bliting a scene from a movie, fine. Bliting a lot of small icons each with a setup cost, not so much.
Like I said at the beginning I wish you luck and hope it all works out, but for me the glass is less than half empty.
Posts: 175
Joined: Tue Dec 27, 2011 9:09 pm
by teh_orph » Sun May 20, 2012 10:21 pm
I agree with you here that there is a lot of communication between the X server and clients. This however appears to be common to allow windowing environments. Why should this set-up be worse than a different windowing system? (I know where you're going with this, I just wanna coax it out of you ;))

After profiling X server / X client interaction it is surprising how much rendering is actually done on the client side of things. This circumvents the X way of doing things and makes the optimisation process a whole lot more tricky.

The thing I would like to improve though is the *scaling of performance with respect to the number of pixels rendered*. In your calculator example, a mouse-over event is a one-off. It does not depend on the number of pixels. When the resolution is low the ratio of computing power expended handling this mouse-over compared to drawing the button being highlighted is low. But bump up the resolution and this ought to improve. This is what I am trying to tackle, not the fixed X client/server interaction. Running the rpi at a high resolution makes it chug.
User avatar
Posts: 345
Joined: Mon Jan 30, 2012 2:09 pm
Location: London
by AndrewS » Mon May 21, 2012 12:07 am
I wonder if KeithSloan's having fun using https://en.wikipedia.org/wiki/Network_e ... dow_System ? ;)
User avatar
Posts: 3626
Joined: Sun Apr 22, 2012 4:50 pm
Location: Cambridge, UK
by spurious » Mon May 21, 2012 6:27 am
teh_orph wrote:I would like to do this however I'm not sure where to host. I can get a free git through my github account, but I've never used that version control system before. Also many of the important parts of the project are simply patches for existing libraries (with specific version numbers). How should I host this? Suggestions please!

I think Hexxeh's approach would work.
You never know, you may get some assistance then. ;)
Although I'd love to help myself, I'm not sure I have the knowledge of X that would be needed.. but I will have a look.
Posts: 343
Joined: Mon Nov 21, 2011 9:29 pm
by KeithSloan » Mon May 21, 2012 9:49 am
I agree with you here that there is a lot of communication between the X server and clients. This however appears to be common to allow windowing environments. Why should this set-up be worse than a different windowing system? (I know where you're going with this, I just wanna coax it out of you ;))

Because X was designed so that on one X server screen you could have a number of clients displayed where the clients are actually all running on different machines and different from the X Server. i.e. communication is via the TCP/IP stack, where as in other systems for example Windows there is no network stack involved. With X you pay for this feature/flexibility even if its not used 99% of the time.
Posts: 175
Joined: Tue Dec 27, 2011 9:09 pm
by teh_orph » Mon May 21, 2012 10:14 am
When local it uses UNIX sockets and shared memory.
Bosh, question one: http://wiki.x.org/wiki/FAQ
User avatar
Posts: 345
Joined: Mon Jan 30, 2012 2:09 pm
Location: London
by KeithSloan » Mon May 21, 2012 10:35 am
When local it uses UNIX sockets and shared memory.
Bosh, question one: http://wiki.x.org/wiki/FAQ


It can do it configured properly, but still more expensive than subroutine calls.
Posts: 175
Joined: Tue Dec 27, 2011 9:09 pm
by teh_orph » Mon May 21, 2012 11:38 am
I guess this is why many applications now do client-side rendering, to improve the interface latency.
User avatar
Posts: 345
Joined: Mon Jan 30, 2012 2:09 pm
Location: London
by theHetman » Mon May 21, 2012 8:21 pm
I know this isn't the area that is being looked at currently, but I've been looking into what is available in OpenVG 1.1. It has all the lines, curves and filled shapes that you would expect, as well as image scaling and rotating. All these can be clipped to a viewport and there are various blend modes. 1.1 adds font rendering, including font hinting (which is expensive) and so should be able to render high quality fonts even as small point sizes. If font rendering could be off loaded to the GPU then that might be quite a win.
Posts: 78
Joined: Tue Jan 10, 2012 5:42 pm
by Jim Manley » Mon May 21, 2012 9:38 pm
The long-time X-perts out there already know this, but, for those who have arrived more recently, to be completely technically accurate, an X server actually runs on what would normally be considered a client system. That is, when you run X on your local PC/Mac/workstation/mobile-device/Pi/etc., you're actually running an X server, not a client. You can then create connections to remote systems via the network to share the resources between systems. That's why there can be multiple user views into other X systems (e.g., 192.168.1.2:0, 192.168.1.2:1, 192.168.1.2:2, etc.), while with technologies such as Windoze Remote Desktop Protocol (RDP), the display on the remote system being logged into disappears locally and is instead displayed on the remote system's video output.

Knowing this makes it much easier to wrap your head around what's actually happening with X and, as mentioned above, why the X "client" on your local system seems so heavyweight, code-wise - it's really a server running a pretty demanding stack of services, whether you need all of them, or not (most of them have to do with reconstruction of the remote views (plural, optionally) from amazingly sparse streams of display element descriptors and values). I would highly recommend going back to the original MIT Project Athena papers (or historical summaries that contain the project goals) and reading about what they were attempting to do and why it was so challenging over relatively slow networks (compared with today's) as opposed to the solely text-oriented network protocols (e.g., telnet, ftp, mail, etc.) running on the early Unix systems up to that point. If you delve into the network sockets details, you will learn more about socket-based networking fundamentals than you could doing probably anything else.
The best things in life aren't things ... but, a Pi comes pretty darned close! :D
"Education is not the filling of a pail, but the lighting of a fire." -- W.B. Yeats
In theory, theory & practice are the same - in practice, they aren't!!!
User avatar
Posts: 1358
Joined: Thu Feb 23, 2012 8:41 pm
Location: SillyCon Valley, California, USA
by jannis » Tue May 22, 2012 6:09 am
theHetman wrote:... If font rendering could be off loaded to the GPU then that might be quite a win.

This is possible with Qt. A bit unstable with 4.X but 5.X will definitely work and be stable. Take a look at the "QtonPi" SD-card-image if you want a demonstration :)
Posts: 55
Joined: Tue Jan 17, 2012 3:48 pm
by ArborealSeer » Tue May 22, 2012 8:45 am
the font stuff will have an overheard though as opengl es itself does not have inbuilt font support
Pi Status > Farnell, Arrived 24/5- RS, Arrived 1/6
User avatar
Posts: 292
Joined: Tue Jan 24, 2012 9:48 am
by jannis » Tue May 22, 2012 8:52 am
Qt can use OpenVG as backend
Posts: 55
Joined: Tue Jan 17, 2012 3:48 pm
by ArborealSeer » Tue May 22, 2012 9:01 am
But thats still not directly handled by the GPU, its still software which as someone posted before may well be layered on top of OpenGL ES.
Pi Status > Farnell, Arrived 24/5- RS, Arrived 1/6
User avatar
Posts: 292
Joined: Tue Jan 24, 2012 9:48 am
by faceless » Tue May 22, 2012 11:29 am
I have absolutely no technical expertise to contribute here, but if a donation would help to speed this up then I'd happily contibute. I have a touchscreen kiosk project planned and 2D performance is going to be the make or break.
Posts: 4
Joined: Tue May 22, 2012 11:19 am
by dom » Tue May 22, 2012 12:49 pm
ArborealSeer wrote:But thats still not directly handled by the GPU, its still software which as someone posted before may well be layered on top of OpenGL ES.


OpenVG is a first class interface handled by the GPU. It is not a wrapper on top of OpenGL ES.
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 4104
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge
by ArborealSeer » Tue May 22, 2012 2:19 pm
*shoulda read back through this post as you'd said that in the very first one*


but to answer..

theHetman wrote: If font rendering could be off loaded to the GPU then that might be quite a win.


as I suspected... dom also confirmed (ish) the fonts aren't GPU accelerated

dom wrote:I"d guess it"s mainly fonts (freetype rendered by ARM)
Pi Status > Farnell, Arrived 24/5- RS, Arrived 1/6
User avatar
Posts: 292
Joined: Tue Jan 24, 2012 9:48 am
by theHetman » Tue May 22, 2012 3:45 pm
Dom said that in the context of XBMC. He said that they had ported an existing version that ran on OpenGL ES 2.0 on another platform. He didn't mention that they were also using OpenVG. He has also said that it's a native API on the Pi and could be useful for speeding up X.

Given that the whole point of OpenVG is to accelerate 2D primitives I would hope that it is using hardware to render. We will have to write some demos to confirm this.
Posts: 78
Joined: Tue Jan 10, 2012 5:42 pm
by jannis » Tue May 22, 2012 4:30 pm
Well at least there is a libOpenVG.so in https://github.com/raspberrypi/firmware ... opt/vc/lib
and it wouldn't make much sense to ship it with the other closed-source stuff it it wasn't supported at all by the GPU
Posts: 55
Joined: Tue Jan 17, 2012 3:48 pm
by khulat » Tue May 22, 2012 4:42 pm
To Quote dom from a few posts above:

dom wrote:OpenVG is a first class interface handled by the GPU. It is not a wrapper on top of OpenGL ES.


Highlights by me.
Posts: 105
Joined: Sun Feb 12, 2012 9:43 pm
by theHetman » Tue May 22, 2012 4:51 pm
jannis wrote:Well at least there is a libOpenVG.so in https://github.com/raspberrypi/firmware ... opt/vc/lib
and it wouldn't make much sense to ship it with the other closed-source stuff it it wasn't supported at all by the GPU

Good point and if you can render Bezier Curves in hardware then basic fonts aren't too much harder. Font hinting is another matter and is needed to stop characters rendered in small point sizes from looking like smudges. The OpenVG 1.1 spec is very clear that font hinting should be supported so that the API can be used in eBook readers.

Oh I've just found this post from 31 Jul 2011:
eben wrote:We'll have to take a look. As James mentions, we support hardware-accelerated OpenVG, and have had Flash Lite running incredibly fast. Personally, I'd like to get the official hardware-accelerated Flash 10 going on there, running against OpenGL ES 2.0, but that's something to think about after the launch.


:)
Posts: 78
Joined: Tue Jan 10, 2012 5:42 pm