Darktrax
Posts: 16
Joined: Sun Nov 30, 2014 1:40 pm

Graphics Acronyms & What works with what?

Sun Jan 11, 2015 2:29 pm

I am new to PI, but what I am trying to read up about does not quite fit in beginner's section.

It would be great to get a bit of an overview of how the common programming languages and resources can best be applied to use the built-in features in the Raspberry Pi. There are so many acronyms, libraries, and much stuff to read that seems not to refer to the other hardware features.

I have a list of things that seem related, or sometimes alternative, or sometimes together, or sometimes in conflict. Naturally, also the questions they bring to mind. I have to rely on some goodwill here, on the grounds that there is no question too silly to ask. If any provoke some derision, in my case, it is probably deserved!

X.org, xorg.conf
Yes, this is about display resolution and (Nouveau?) graphics driver, but also about mouse, keyboard and most other aspects of user input. Something that OpenGL ES seems to bypass?

Framebuffer
Something I thought absolutely everything needs on the way to the display screen. I guess I was wrong!

OpenGL and OpenGL ES - There is a GPU in the ARM processor in the Pi that can use this. There are impressive moving textured examples that show off 3D graphics tricks.
Invoked from C, C++, Java - not sure, and not sure how?

GLES and GLES2??
Are these shortened names for OpenGL ES? Are they library names, or just part of path names?

OpenVG
A feature with "support" apparently built-in to the Raspberry Pi processor. Not sure what form this support takes. To use it may require much import of "libraries". Clearly at least one expert can take us down this road. --> https://github.com/ethanol100/RaspiVidYUV

Question1: Is OpenVG something that can be used along with OpenGL ES together, in the same app, with the benefit of both, though possibly in different threads?

Question2: Can OpenVG be used independently, but simultaneously with OpenGL ES, in different apps?

Question3: Do both of these somehow bypass xorg?
SDL, SDL1, SDL2
I was guessing that these are all the same thing.
We have contributions in this forum --> http://www.raspberrypi.org/forums/viewt ... 67&t=58180
Question1: How does SDL(x) relate to all the other ways things can get displayed?

Question2: Thinking that using 3D things like OpenGL ES to display 2D to relieve the CPU of pixel-by-pixel computations might have big advantages, is SDL2 something that precludes this?

EGL
Another Graphics-related acronym I have come across. I am not sure how it fits with all the above?

GTK and Qt5
Much involved with graphics and widgets display, and both are huge, traditional, though alternative tools for display stuff.
Can one start out making an app for either, and still have the benefits of (say) OpenGL ES or OpenVG (or both)?

Java?
There is apparently built-in JIT features in the Pi processor. Getting a Java app to run on Pi, having first made it on a different PC is very easy, though using C++ in Eclise environment, and a cross-compiler also worked, with a bit more fumble.
Can one invoke OpenGL ES or OpenVG from such souces, and what does it take?

Forgive me if the question set is sometimes inconsistent. I do realise that some questions may be moot, because at this stage, I just don't know.

ghans
Posts: 7878
Joined: Mon Dec 12, 2011 8:30 pm
Location: Germany

Re: Graphics Acronyms & What works with what?

Sun Jan 11, 2015 3:11 pm

X.org, xorg.conf
Yes, this is about display resolution and (Nouveau?) graphics driver, but also about mouse, keyboard and most other aspects of user input. Something that OpenGL ES seems to bypass?
The X server (Xorg server , X11 server) is normally responsible for everything which appears on the screen and manages keyboards and mouses and how
programs communicate with them. OpenGL ES bypasses this all. xorg.conf is the main configuration file , but "Nouveau" is only applicable to PCs (it is a driver).
Framebuffer
Something I thought absolutely everything needs on the way to the display screen. I guess I was wrong!
I think you are still right ?! Somebody else has to comment here .
OpenGL and OpenGL ES - There is a GPU in the ARM processor in the Pi that can use this. There are impressive moving textured examples that show off 3D graphics tricks.
Invoked from C, C++, Java - not sure, and not sure how?
Those APIs (Application Programming Interfaces) are directly exposed to C (and therefore C++ too) only. Other programming languages have to use wrappers
which might be not that good or non-existent.
GLES and GLES2??
Are these shortened names for OpenGL ES? Are they library names, or just part of path names?
Just a shorthand for OpenGL ES.
Question1: Is OpenVG something that can be used along with OpenGL ES together, in the same app, with the benefit of both, though possibly in different threads?

Question2: Can OpenVG be used independently, but simultaneously with OpenGL ES, in different apps?
I would answer yes to both , but i'm not sure.
Question3: Do both of these somehow bypass xorg?
Yes.
GTK and Qt5
Much involved with graphics and widgets display, and both are huge, traditional, though alternative tools for display stuff.
Can one start out making an app for either, and still have the benefits of (say) OpenGL ES or OpenVG (or both)?
Qt5 should use hardware-acceleration behind the scenes , unsure about GTK (might depend on version used).
Java?
There is apparently built-in JIT features in the Pi processor. Getting a Java app to run on Pi, having first made it on a different PC is very easy, though using C++ in Eclise environment, and a cross-compiler also worked, with a bit more fumble.
Can one invoke OpenGL ES or OpenVG from such souces, and what does it take?
Use a OpenGL ES Wrapper/toolkit like JOGL. http://jogamp.org/jogl/www/


ghans
• Don't like the board ? Missing features ? Change to the prosilver theme ! You can find it in your settings.
• Don't like to search the forum BEFORE posting 'cos it's useless ? Try googling : yoursearchtermshere site:raspberrypi.org

Darktrax
Posts: 16
Joined: Sun Nov 30, 2014 1:40 pm

Re: Graphics Acronyms & What works with what?

Tue Jan 13, 2015 11:29 pm

ghans wrote: Can one invoke OpenGL ES or OpenVG from such souces, and what does it take?

Use a OpenGL ES Wrapper/toolkit like JOGL. http://jogamp.org/jogl/www/


ghans
Thanks much for the suggestions ghans.

Probably with a mix of direct experimentation, and searching out what techniques various folk have used, I will come up with something that uses the built-in features to best effect..

It would seem that the best stuff in there is the ability to use OpenGL ES on the GPU, and the built-in hardware that is like a fast open door to Java (JIT).

OpenVG is still a mystery to me, but that is another link I shall be exploring.

x.org, hence xorg server is one of those things that we keep hearing of "upgrades", "forks", replacements for" that are "coming soon". Xorg, despite it's age, is also one of those things that "just works". It does very essential stuff, and will not just disappear anytime soon. There may be newer things that might take over part of the job (eg. Weyland).

There may even be an "upgrade" sometime to the X.Org x-server that will support OpenGL ES, instead of them being separate ways of making stuff appear on the screen.

For my small needs, I am happy to just use the best tools offered, so I can have the Pi work to my purpose.

ghans
Posts: 7878
Joined: Mon Dec 12, 2011 8:30 pm
Location: Germany

Re: Graphics Acronyms & What works with what?

Wed Jan 14, 2015 7:39 am

The Jazelle extensions are unused AFAIK. They aren't that
beneficial to modern Java SE anymore either , so i've heard.

ghans
• Don't like the board ? Missing features ? Change to the prosilver theme ! You can find it in your settings.
• Don't like to search the forum BEFORE posting 'cos it's useless ? Try googling : yoursearchtermshere site:raspberrypi.org

User avatar
hanzelpeter
Posts: 72
Joined: Mon Jul 09, 2012 11:56 am

Re: Graphics Acronyms & What works with what?

Wed Jan 14, 2015 6:09 pm

Are the Jazelle extensions used somewhere? Like in Android?
Do you know how I can use it on RasPi?

ghans
Posts: 7878
Joined: Mon Dec 12, 2011 8:30 pm
Location: Germany

Re: Graphics Acronyms & What works with what?

Sat Jan 17, 2015 1:36 pm

AFAIK Jazelle was all the rage when featurephones with
Java ME were popular. Since Android used Dalvik bytecode
from the beginning and now native code via ART or Dalvik-JIT
Jazelle has fallen out of favour.

No idea how to use this on the Pi. I guess ARM has a vested
interest to keep docs secret or only give them their very
direct costumers (which we don't are).

ghans
• Don't like the board ? Missing features ? Change to the prosilver theme ! You can find it in your settings.
• Don't like to search the forum BEFORE posting 'cos it's useless ? Try googling : yoursearchtermshere site:raspberrypi.org

-rst-
Posts: 1316
Joined: Thu Nov 01, 2012 12:12 pm
Location: Dublin, Ireland

Re: Graphics Acronyms & What works with what?

Tue Jan 20, 2015 4:09 pm

This is in fact fairly interesting question set - probably a source of confusion for many. Therefore some clarifications may be in place... (others feel free to comment, correct and extend - I do not claim to be an expert on this)

Raspberry Pi includes (built into the one 'System On a Chip' (SoC) package) both the ARM 'main processor' / CPU (just like a PC would have one form Intel or AMD) and the VideoCore IV graphics processor / GPU (like a PC would have one from Intel/Nvidia/AMD either built into the motherboard or on a separate graphics card).

The GPU provides hardware accelerated graphics - exposed to application programmers through the standard APIs: OpenGL, OpenVG etc. On the Raspberry Pi the GPU implements the slightly lighter 'mobile device optimised' variant/subset of OpenGL called OpenGLES. OpenGL and OpenGLES are mainly for 3D graphics (can also do 2D) - OpenVG is only for 2D graphics. It should be possible to combine both into same application - at least on two separate graphics surfaces overlaid on top of each other.

'X' (X11, X.org, X Window System) is the 'motor under the hood' of most 'graphical desktop environments' in Linux - it provides the low-level services for input (keyboard, mouse, ...) and output (graphics, sound?, ...). In addition to the 'motor', a typical DTE would also include a window manager and the actual graphical desktop (with user interface widgets etc.). The recommended Linux distribution on Raspberry Pi 'Raspbian' includes a complete set of these components and the current incarnation looks like this - other distributions may have slightly different set of components (different window manager and desktop on top of the common X.org 'motor').

Applications running on the Raspbian desktop can be programmed in many programming languages - some are less 'under-the-hood' concerned than others. Using C it is possible to implement graphical applications linking to the xlib the programming library/API for X.org. Usually one would of course want to use higher level user interface components often referred to as widgets - for this one would use any of the toolkits like GTK, QT(4), SDL, ...) - some possibly requiring C++ as the programming language. There are also other higher level languages like Python (in X one could use PyGTKor Pygame for user interfacing) and Java (OpenJDK 'desktop Java' with AWT/Swing user interface libraries, not 100% sure of the status of this on RPi - better answered on the Java sub-forum I guess), even C# with Mono(?).

The Open* graphics libraries do not provide for display surfaces which (in a windowed desktop environment) one could think of as 'window client areas'. To be able to draw say 3D graphics in an application running within the desktop environment, the program would need to create a desktop window and a graphics surface inside the window - for this there is the GLX library. For Java there is the JOGL library which provides integration with the Open* libraries.

Image
(window manager and desktop libraries omitted (the transparent yellow box) - SDL/GTK/Qt libraries also provide components to use Open* graphics - X does not really know anything about the Open* graphics drawn as the hw composites the X window surface and the GLX surface - at least as far as I know, at the moment the X video driver uses the kernel framebuffer).

However this is a bit different on Raspberry Pi - there is no GLX and a lower level library EGL must be used instead. Also there is the dispmanx library that binds the EGL surface to the actual platform surface.

Image
(EGL surface is not tied to the X window - the application must move the surface if the window is moved ...which may lead to alignment issues like here with Minecraft)

It is also quite possible to run applications without the desktop environment - without the X window system at all. The simplest graphics can be drawn directly to the framebuffer (see my blog in the signature for examples) - dispmanx and EGL (and through EGL the Open* graphics) can be used from a 'console' application too (see the hello_pi examples provided with Raspbian). These are programmed using C language (may of course be wrapped into other languages). With these libraries accessing the mouse and keyboard is not the easiest and there are no ready made user interface widgets.

The SDL library can be used without X too - it provides mouse and keyboard handling and other useful functions - there is also a dispmanx backed version of SDL available which provides flicker free double-buffering.

Qt5 (using C++ programming language) on Raspberry Pi provides the full stack of graphics and user interface components. As does the Oracle JDK 8 with JavaFX. One option is to use Python language and the Pi3D library for graphics (not sure what the user interface functionality provided is).

Image
(NOTE: this is not meant to be in anyway complete as it comes to available libraries/frameworks/etc. - ajstarks/openvg and Cogl (RPi port) comes to mind first)

So it all depends on what you want to do, do you need/want to use the desktop environment, do you want to squeeze the last drop of performance, what is your preferred programming language, ...

Where my knowledge really ends short are the different options possibly replacing X or parts of it in the future like Maynard and Wayland(I understand Qt5 would be using Wayland compositor as part of the stack but can Wl be used in other combinations?). By now it is fairly clear that X is somewhat archaic, designed more for network not for desktop, slow on resource constrained devices, etc. but then again it may be around for some time until better options are mature enough.

UPDATE 2015-11-22: It looks like JavaFX is not anymore included in the latest Oracle JDK build http://www.oracle.com/technetwork/java/ ... l#CACHGFJC
Last edited by -rst- on Thu Jan 22, 2015 4:41 pm, edited 2 times in total.
http://raspberrycompote.blogspot.com/ - Low-level graphics and 'Coding Gold Dust'

User avatar
procount
Posts: 1854
Joined: Thu Jun 27, 2013 12:32 pm
Location: UK

Re: Graphics Acronyms & What works with what?

Tue Jan 20, 2015 4:59 pm

+1
Brilliant post -rst- ! :D
I've been looking for something like this for ages and I'm sure it will answer many beginners' questions. My only remaining questions are: which libraries are used to control the surfaces in the compositor (i.e. which one is in front etc), and is there any limit to the number of surfaces there can be?
PINN - NOOBS with the extras... https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=142574

-rst-
Posts: 1316
Joined: Thu Nov 01, 2012 12:12 pm
Location: Dublin, Ireland

Re: Graphics Acronyms & What works with what?

Tue Jan 20, 2015 5:37 pm

Thanks, procount!

The library for controlling the compositor surfaces on RPi is dispmanx. This may be the only info on the limits http://www.raspberrypi.org/forums/viewt ... 59#p568678
http://raspberrycompote.blogspot.com/ - Low-level graphics and 'Coding Gold Dust'

User avatar
AndyD
Posts: 2333
Joined: Sat Jan 21, 2012 8:13 am
Location: Melbourne, Australia
Contact: Website

Re: Graphics Acronyms & What works with what?

Tue Jan 20, 2015 8:42 pm

That is a great explanation -rst-. It would be great to see this in the Raspberry Pi Documentation.

senderj
Posts: 14
Joined: Mon Jan 19, 2015 9:54 am

Re: Graphics Acronyms & What works with what?

Wed Jan 21, 2015 8:19 am

Very good description. Exactly what I am looking for. Thank you -rst-.

Would you please point out where is things like PIL and SimpleCV in your Pi diagram?

-rst-
Posts: 1316
Joined: Thu Nov 01, 2012 12:12 pm
Location: Dublin, Ireland

Re: Graphics Acronyms & What works with what?

Thu Jan 22, 2015 12:37 pm

senderj wrote:Would you please point out where is things like PIL and SimpleCV in your Pi diagram?
Not very familiar with these two, but based on my limited knowledge and some quick googling...

PIL would sit on the same level as Pi3D - with a link from the application to PIL (and back) only. If the programmer wants to get something loaded/processed with PIL and displayed using Open*, the application must grab the image from PIL and push it to Open* using another library. Not sure if Pi3D has some PIL integration that would make this simpler - there is also a package PyOpenGL that could possibly be used - or PyGame and this https://github.com/stephanh42/rpigl.

Same seems to apply to using Python+PIL running in X with the addition of Python -> X libraries like PyGTK or PyQt required. There are helper modules for converting PIL images to PyQt and PyGTK images (in this case there would be three library boxes (instead of Pi3D): PyQt/PyGTK + ImageQt/GTK + PIL side-by-side and connected.

SimpleCV seems to use the Open* libraries (?) so it would sit in the picture on the same level as Pi3D and have the same connection to Open* in the HW.

Hope that helps.

If anyone with more experience can spot misinformation, please correct.
http://raspberrycompote.blogspot.com/ - Low-level graphics and 'Coding Gold Dust'

User avatar
paddyg
Posts: 2440
Joined: Sat Jan 28, 2012 11:57 am
Location: UK

Re: Graphics Acronyms & What works with what?

Thu Jan 22, 2015 9:45 pm

Very comprehensive resume of everything graphics on the pi. Just to add little to the pi3d comment. a) it requires PIL (or Pillow as now called) for image loading, text etc. So this is bound to be available and can provide much functionality reasonably quickly and easily. b) There is a gui of a fairly rough and ready kind (drop-down menus, sliders, buttons, text inputs). But it's python so won't cope with too much going on at once and it doesn't run with an event driven structure like all the other standard uis (because it's really aimed at 3D animation I suppose). On the pi it doesn't use the xserver which has pros and cons (on 'big' linux it does use x for the drawing surface and mesa for OpenGL ES emulation. Just getting it to work on android which is using SDL via kivy/python-for-android)
also https://groups.google.com/forum/?hl=en-GB&fromgroups=#!forum/pi3d

darryletter
Posts: 76
Joined: Thu Jan 01, 2015 8:52 pm
Location: KC,MO

Re: Graphics Acronyms & What works with what?

Mon Feb 23, 2015 3:35 am

great tread i have learned a lot from this as i am trying to do some graphics programming for my tank controller
Life After 60 sucks

darryletter
Posts: 76
Joined: Thu Jan 01, 2015 8:52 pm
Location: KC,MO

Re: Graphics Acronyms & What works with what?

Mon Feb 23, 2015 11:14 am

-rst- wrote: Applications running on the Raspbian desktop can be programmed in many programming languages - some are less 'under-the-hood' concerned than others. Using C it is possible to implement graphical applications linking to the xlib the programming library/API for X.org. Usually one would of course want to use higher level user interface components often referred to as widgets - for this one would use any of the toolkits like GTK, QT(4), SDL, ...) - some possibly requiring C++ as the programming language. There are also other higher level languages like Python (in X one could use PyGTKor Pygame for user interfacing) and Java (OpenJDK 'desktop Java' with AWT/Swing user interface libraries, not 100% sure of the status of this on RPi - better answered on the Java sub-forum I guess), even C# with Mono(?).
So 2 questions :
1) Is the Raspbian desktop a port of grome
2) Would I have to run the mate desktop to use GTK+
Life After 60 sucks

User avatar
AndyD
Posts: 2333
Joined: Sat Jan 21, 2012 8:13 am
Location: Melbourne, Australia
Contact: Website

Re: Graphics Acronyms & What works with what?

Mon Feb 23, 2015 11:22 am

darryletter wrote: So 2 questions :
1) Is the Raspbian desktop a port of grome
No it is lxde. This article describes the differences.
darryletter wrote: 2) Would I have to run the mate desktop to use GTK+
No, LXDE is based on GTK+ version 2 according to Wikipedia.

Return to “Graphics programming”