EGL - How to use fb1?


21 posts
by yo1dog » Tue Oct 22, 2013 11:24 pm
I posted this question on StackOverflow but with no responses I think this may be a better place to ask.

I am trying to modify the hello_triangle example (/opt/vc/src/hello_pi/hello_triangle) on my Raspberry Pi to output to fb1 instead of fb0.

I have searched everything I can think of with no answers (it could be I don't know what to search for). It seems like the line I need to modify is this line:
Code: Select all
state->display = eglGetDisplay(EGL_DEFAULT_DISPLAY);


The docs state the parameter is of type NativeDisplayType and "Specifies the display to connect to. EGL_DEFAULT_DISPLAY indicates the default display." However, it does not give any information on how to list/find available displays.

EGL_DEFAULT_DISPLAY is defined as (NativeDisplayType)0 so I guessed that 0 referred to fb0, but using (NativeDisplayType)1 for fb1 caused eglGetDisplay to return EGL_NO_DISPLAY.

The original example outputs to fb0 as it should, no problems there.
I also know my display is working correctly as I can use fbi to display images on fb1 just fine.
Posts: 11
Joined: Tue Oct 22, 2013 11:18 pm
by texy » Wed Oct 23, 2013 6:39 am
Hi,
not exaclty cutting-edge for your particular needs, but there is a utility that copies fb0 to fb1 as a background task - search the forum for fbcp

Hope this helps,
Texy
"2.8inch TFT LCD + Touch screen" add-on boards for sale here :
http://www.raspberrypi.org/phpBB3/viewtopic.php?f=93&t=65566
50p goes to the Foundation ;-)
Forum Moderator
Forum Moderator
Posts: 2210
Joined: Sat Mar 03, 2012 10:59 am
Location: Berkshire, England
by yo1dog » Wed Oct 23, 2013 3:40 pm
I suspect a couple problems:

1. The displays attached to each framebuffer have different resolutions and color bit-depths. Although, I bet I could manually set/override this.
2. How fast does this update? Will it cause lag or tearing? I guess I should just try it but I am at work currently.

Thanks for this solution. I will try it out later. I would still like to figure out my original question as this seems a bit hacky.
Posts: 11
Joined: Tue Oct 22, 2013 11:18 pm
by Paeryn » Wed Oct 23, 2013 10:44 pm
The opengl output isn't drawn into the linux framebuffers, it's drawn into an entirely separate layer which is composited by the Pi's internal display manager together with any other layers (of which fb0 is one such layer) to generate the image which is shown.

On the RPi there is only one display. Taken from the source code for eglGetDisplay() for the RPi :-
Code: Select all
   Implementation notes:

   We only support one display. This is assumed to have a native display_id
   of 0 (==EGL_DEFAULT_DISPLAY) and an EGLDisplay id of 1
She who travels light - forgot something.
User avatar
Posts: 65
Joined: Wed Nov 23, 2011 1:10 am
Location: Sheffield, England
by yo1dog » Thu Oct 24, 2013 4:21 am
Hmm... So you are saying it is impossible for egl to use fb1. I wonder why they would implement that restriction. I won't pretend to remotely understand what goes on at that level.

If that is the case. Is there anyway to make the default video output (on-board HDMI/Composite) fb1 and my display fb0 (the default display) so it can be used by egl and openGL ES?
Posts: 11
Joined: Tue Oct 22, 2013 11:18 pm
by Paeryn » Thu Oct 24, 2013 6:21 pm
yo1dog wrote:Hmm... So you are saying it is impossible for egl to use fb1. I wonder why they would implement that restriction. I won't pretend to remotely understand what goes on at that level.

Yes, EGL on the RPi doesn't use the linux framebuffers at all, not fb1 nor fb0. The graphics hardware works at a higher level (dispmanx), the display is made up of multiple layers, when linux starts it creates one layer to cover the full screen for fb0. When you create a window with EGL you also create a new layer and tell the hardware to show it in front of the others.

As far as I understand, the display you see is generated on the fly by the compositing hardware, so if you have a fullscreen fb0 with an EGL window in the middle showing your OpenGL / OpenVG / OpenMAX then there isn't a single bitmap image containing what you see.

If that is the case. Is there anyway to make the default video output (on-board HDMI/Composite) fb1 and my display fb0 (the default display) so it can be used by egl and openGL ES?

If you want to show your OpenGL window on a secondary display (reading your other post http://www.raspberrypi.org/phpBB3/viewtopic.php?f=29&t=59081#p442721 you have a USB display - I'm assuming you want it showing on there) then you could do it by having the EGL layer behind the framebuffer layer (ideally you'd want to create the layer but not have it actively visible but I'm not sure how you'd go about that, other than to have it hidden behind all other layers). Then when you've finished your OpenGL drawing make a copy of the window with glReadPixels() and write the data out to fb1.
She who travels light - forgot something.
User avatar
Posts: 65
Joined: Wed Nov 23, 2011 1:10 am
Location: Sheffield, England
by yo1dog » Fri Oct 25, 2013 2:11 am
Any idea how I write to fb1?
Posts: 11
Joined: Tue Oct 22, 2013 11:18 pm
by AndyD » Fri Oct 25, 2013 3:00 am
yo1dog wrote:Any idea how I write to fb1?

There is an extensive blog written by a (former?) member of this forum called Raspberry Compote that explains writing to the frame-buffer on the Raspberry Pi. The source code that copies from the Raspberry Pi compositor to fb1 may also be of interest. It might be worth experimenting I have a feeling that using glReadPixels() may be slower than using vc_dispmanx_snapshot() (as is used in rpi-fbcp). I have no evidence to support that, but I vaguely remember someone in another thread complaining the glReadPixels() was slow.
Posts: 648
Joined: Sat Jan 21, 2012 8:13 am
Location: Melbourne, Australia
by yo1dog » Fri Oct 25, 2013 4:25 am
Going off that source I was able to add the following to the hello_triangle example:
Code: Select all
//...

static int screen_width    = 480;
static int screen_height   = 272;
static int screen_bitdepth = 2;
static int screen_datasize;

static DISPMANX_DISPLAY_HANDLE_T display;
static DISPMANX_RESOURCE_HANDLE_T screen_resource;
static int fbfd = 0;
static char *fbp = 0;

static VC_RECT_T rect;

//...

static void init_ogl(CUBE_STATE_T *state)
{
   screen_datasize = screen_width * screen_height * screen_bitdepth;

   display = vc_dispmanx_display_open(0);
   // do check and exit if failed

   fbfd = open("/dev/fb1", O_RDWR);
   // do check and exit if failed

   uint32_t image_prt;
   screen_resource = vc_dispmanx_resource_create(VC_IMAGE_RGB565, screen_width, screen_height, &image_prt);
   // do check and exit if failed

   fbp = (char*)mmap(0, screen_datasize, PROT_READ | PROT_WRITE, MAP_SHARED, fbfd, 0);
   // do check and exit if failed

   vc_dispmanx_rect_set(&rect, 0, 0, screen_width, screen_height);

   printf("Everything worked\n");

//...

   while (!terminate)
   {
      update_model(state);
      redraw_scene(state);

      printf("start\n");

      int ret = vc_dispmanx_snapshot(display, screen_resource, 0);
      printf("ret1: %i\n", ret);

printf("middle\n");

      ret = vc_dispmanx_resource_read_data(screen_resource, &rect, fbp, screen_width * screen_bitdepth);
      printf("ret2: %i\n", ret);

      printf("end\n");

//...

static void exit_func(void)
{

//...

   munmap(fbp, screen_datasize);
   close(fbfd);
   vc_dispmanx_resource_delete(screen_resource);
   vc_dispmanx_display_close(display);

//...

But the following prints out:

Everything worked
start
ret1: 0
middle
ret2: -1
end
start

And then it hangs forever. So on the first loop I see the cube on fb0 but no change to fb1 and vc_dispmanx_resource_read_data returns -1. On the second loop it hangs on vc_dispmanx_snapshot.

I noticed that when trying to use glReadPixels it also hangs forever.

EDIT: fbcp does not work and also seems to hang.
EDIT2: I put the same print statements around the vc_dispmanx_resource_read_data and vc_dispmanx_snapshot in fbcp source and it prints out exactly the same as my modified hello_triangle.

Any ideas?
Posts: 11
Joined: Tue Oct 22, 2013 11:18 pm
by Paeryn » Fri Oct 25, 2013 2:54 pm
yo1dog wrote:Everything worked
start
ret1: 0
middle
ret2: -1
end
start

And then it hangs forever. So on the first loop I see the cube on fb0 but no change to fb1 and vc_dispmanx_resource_read_data returns -1. On the second loop it hangs on vc_dispmanx_snapshot.

I noticed that when trying to use glReadPixels it also hangs forever.

EDIT: fbcp does not work and also seems to hang.
EDIT2: I put the same print statements around the vc_dispmanx_resource_read_data and vc_dispmanx_snapshot in fbcp source and it prints out exactly the same as my modified hello_triangle.

Any ideas?

It looks like vc_dispmanx_resource_read_data() fails when trying to write to the mmap'ed memory. If you malloc a temporary buffer to read the resource into and the memcpy that to the mmap'ed memory then the program runs fine. Whether it actually copies the display I can't say (my RPi isn't connected to a display at the moment, and I don't have a usb display), I ran it copying into fb0.
She who travels light - forgot something.
User avatar
Posts: 65
Joined: Wed Nov 23, 2011 1:10 am
Location: Sheffield, England
by yo1dog » Fri Oct 25, 2013 3:17 pm
I think I tried that with the same results. I can confirm on Sunday.

I assumed the read is failing because it returns -1, but why is it hanging on vc_dispmanx_snapshot?

fbcp seems to work for everyone else, why would it not work on my setup?
Posts: 11
Joined: Tue Oct 22, 2013 11:18 pm
by Paeryn » Sat Oct 26, 2013 3:27 am
I would guess something along the lines of, the read request locking the resource that holds the screenshot and fails to free it when the read fails. The next screenshot request could then be sat waiting for the resource to be unlocked.

I tried the fbcp program (again changing it to use to fb0), it locks up exactly the same for me.

I've just tried the modified version of your program (reading to malloc'ed memory and then memcpy'ing it to fb0) and it definitely copies the image from when the opengl window was open onto fb0.

The only problem with using vc_dispmanx_snapshot is that your usb display fb1 will only be able to show exactly what is on the main display that the RPi is outputting via HDMI / Composite.
I got the glReadPixels version working without you seeing the actual opengl window on screen, but only in 32bit colour. I couldn't get gles1 to return anything other than a 32bit RGBA bitmap even when explicitly asking for a 16bit 565 window with eglSaneChooseConfigBRCM.
She who travels light - forgot something.
User avatar
Posts: 65
Joined: Wed Nov 23, 2011 1:10 am
Location: Sheffield, England
by yo1dog » Tue Oct 29, 2013 4:49 pm
I was able to read the snapshot into memory and then memcpy it to fb1. That worked and I can now see a scaled version of fb0 on fb1.

Problem: I don't want to scale, I just want to crop 480x272 pixels (resolution of my LCD) of fb0 and display it in fb1.

One solution would be to force fb0 to have the same resolution (480x272) so that no cropping or scaling would be needed and a direct copy could be used. I tried plugging in an HDMI monitor and in /boot/config.txt setting hdmi_mode=3 (480p) and hdmi_force_hotplug=1 (just in case) but the resolution was still higher than 480x272. Can anyone think of a way to force fb0 to be the resolution I need?

Another solution would be to use glReadPixels and read the pixel data I need into fb1. Problem is that the RPI implementation of openGL ES only supports RGB888 but my LCD is 16bit so I need RGB565. Converting the formats manually (using the CPU) is too slow and drops the framerate significantly (maybe the algorithm I am using is inefficient?).

My current solution is to make the snapshot the same size as fb0 and read it into memory. When memcpy-ing into fb1 I only copy the the top-left 480x272 pixels using the following code:

Code: Select all
//...
// get the resolution of fb0
int fb0_screen_width;
int fb0_screen_height;
graphics_get_display_size(0, &fb0_screen_width, &fb0_screen_height);

// set the resolution of fb1
int fb1_screen_width = 480;
int fb1_screen_height = 272;
int fb1_screen_bitdepth = 2; // 16bit RGB565

// allocate some memory for reading the snapshot into as fb0_pointer
unsigned short * fb0_pointer = malloc(fb0_screen_width * fb0_screen_height * fb1_screen_bitdepth); // note we use fb1_screen_bitdepth becuase the resource we define below uses 16bit RGB565 as the pixel format, same as fb1.

// open fb1 and map it to memory as fb1_pointer
int fb1fd = open("/dev/fb1", O_RDWR)
unsigned short * fb1_pointer = mmap(0, fb1_screen_width * fb1_screen_height * fb1_screen_bitdepth, PROT_READ | PROT_WRITE, MAP_SHARED, fb1fd, 0);

// create the rectangle the snapshot should be scaled into (I think?). Use the original size (size of fb0) so nothing actually gets scaled.
VC_RECT_T rect;
vc_dispmanx_rect_set(&rect, 0, 0, fb0_screen_width, fb0_screen_height);

// create the dispmanx resource and use 16bit RGB565 as the format. Not sure what this does or how the width and height are used, or what the image_prt is used for?
uint32_t image_prt;
screen_resource = vc_dispmanx_resource_create(VC_IMAGE_RGB565, fb0_screen_width, fb0_screen_height, &image_prt);

//...

// take a snapshot and read the pixel data into fb0_pointer.
vc_dispmanx_snapshot(display, screen_resource, 0);
vc_dispmanx_resource_read_data(screen_resource, &rect, fb0_pointer, fb1_screen_width * fb1_screen_bitdepth); // not sure what this last parameter does but if you make it less than this it cuts off the bottom on fb1

// copy only the top left pixels to fb1
int y;
for (y = 0; y < fb1_screen_height; y++)
   memcpy(
      fb1_pointer + y * fb1_screen_width, // beginning of the row for fb1
      fb0_pointer + y * fb0_screen_width, // beginning of the row for fb0
      fb1_screen_width * fb1_screen_bitdepth); // one row of pixel data for fb1


As you can see from the comments above, I don't understand everything that is going on here and I bet there is a better way of doing this. The snapshot and reading above drops the framerate a bit. Is there a faster way to do this? Is there really not a way to snapshot just a portion of fb0?

How do programs like fbi and xserver use only fb1 without having to render to fb0?

Is it possible to use the GPU to convert the pixel format so that I can use glReadPixels and don't have to use EGL to render to fb0 and copy to fb1?
Posts: 11
Joined: Tue Oct 22, 2013 11:18 pm
by karlkiste » Tue Oct 29, 2013 7:24 pm
yo1dog wrote:How do programs like fbi and xserver use only fb1 without having to render to fb0?


They just render to fb1. I'm displaying pictures like this:

gm convert picture.jpg -scale ${xres}x${yres}! -depth 8 RGB:/dev/fb1

There's no trick to it.
To just see anything, you can just

cat /dev/urandom >/dev/fb1

You're doing something completely different. It's an EGL buffer you're rendering to, not a framebuffer. If it was the framebuffer, you could just read /dev/fb0 and copy it to /dev/fb1. Using graphicsmagick, you could also scale or crop. To just crop, "dd" could even be sufficient. Or "cut". But it is not fb0 you're rendering to. That's what makes your task a bit more complicated.
Posts: 149
Joined: Tue Jan 22, 2013 8:50 am
Location: berlin, germany
by yo1dog » Tue Oct 29, 2013 9:59 pm
Can I use openGL ES to render without using EGL? Will it still take advantage of the GPU? Is there a way to use the GPU to convert the rendered image from openGL ES to RGB 565 or a more efficient way to do it with the CPU that won't kill my framerate?
Posts: 11
Joined: Tue Oct 22, 2013 11:18 pm
by karlkiste » Wed Oct 30, 2013 4:57 am
yo1dog wrote:Can I use openGL ES to render without using EGL? Will it still take advantage of the GPU? Is there a way to use the GPU to convert the rendered image from openGL ES to RGB 565 or a more efficient way to do it with the CPU that won't kill my framerate?


I'm not very much into this subject, I only understand the overall principle. I don't know the difference between OpenGL ES and EGL, I only think of those as "GPU features". Those features usually go directly to the screen output, without touching the CPU. The framebuffer is a memory area which belongs to the CPU, is read by the GPU, mixed with the "GPU features" and then sent to the display. This mixing is done by dedicated hardware, and goes in parallel to all other function. No time is needed for that.

You're copying data from the "GPU features" back to the CPU. This is something unusual, and it takes time. Therefore, I'm not sure if it is really improving speed to use the "GPU features". You could instead create the graphics directly on the CPU and you would not have to read the result back that way. Your Screen has quite a low resolution, so it's probably possible to use only software rendering.
Posts: 149
Joined: Tue Jan 22, 2013 8:50 am
Location: berlin, germany
by Paeryn » Wed Oct 30, 2013 5:21 am
yo1dog wrote:One solution would be to force fb0 to have the same resolution (480x272) so that no cropping or scaling would be needed and a direct copy could be used. I tried plugging in an HDMI monitor and in /boot/config.txt setting hdmi_mode=3 (480p) and hdmi_force_hotplug=1 (just in case) but the resolution was still higher than 480x272. Can anyone think of a way to force fb0 to be the resolution I need?

I doubt you can set the display that small, I think the smallest you can go is 640x480.

yo1dog wrote:Another solution would be to use glReadPixels and read the pixel data I need into fb1. Problem is that the RPI implementation of openGL ES only supports RGB888 but my LCD is 16bit so I need RGB565. Converting the formats manually (using the CPU) is too slow and drops the framerate significantly (maybe the algorithm I am using is inefficient?).

The RPi does support 565, if you ask for it you will see the colour banding. To get it you have to pick the right config. The first config returned by eglChooseConfig() when asking for a 565 will NOT be 565 but 888 as the spec says that the configs are sorted with colour sizes that biggest first. Broadcom have an extra function eglSaneChooseConfigBRCM() which will return the list sorted with minimum first.

For some reason though glReadPixels only supports GL_RGBA,GL_UNSIGNED_BYTE. It should also support a second format returned from glGet() GL_IMPLEMENTATION_COLOR_READ_FORMAT_OES,GL_IMPLEMENTATION_COLOR_COLOR_TYPE_OES but glGet hardcodes these to return GL_RGBA and GL_UNSIGNED_BYTE rather than letting them have values corresponding to the colour buffer format.

Some notes on your code that you gave,
Code: Select all
// create the dispmanx resource and use 16bit RGB565 as the format. Not sure what this does or how the width and height are used, or what the image_prt is used for?
uint32_t image_prt;
screen_resource = vc_dispmanx_resource_create(VC_IMAGE_RGB565, fb0_screen_width, fb0_screen_height, &image_prt);

It creates an image resource in the gpu's memory to hold the snapshot with a resolution of fb0_screen_width x fb0_screen_height. The image_prt just needs to be a pointer, it's meant to hold a pointer to the bitmap but as that memory isn't available to the ARM it will always be a null pointer.
Code: Select all
vc_dispmanx_resource_read_data(screen_resource, &rect, fb0_pointer, fb1_screen_width * fb1_screen_bitdepth); // not sure what this last parameter does but if you make it less than this it cuts off the bottom on fb1

The last parameter is the stride in bytes between successive lines of the destination bitmap. It's useful if the memory you are copying to is of a bitmap that is wider than the source bitmap (or requires extra padding at the end of each line).
Code: Select all
// copy only the top left pixels to fb1
int y;
for (y = 0; y < fb1_screen_height; y++)
   memcpy(
      fb1_pointer + y * fb1_screen_width, // beginning of the row for fb1
      fb0_pointer + y * fb0_screen_width, // beginning of the row for fb0
      fb1_screen_width * fb1_screen_bitdepth); // one row of pixel data for fb1

The memcpy parameters are wrong, it should be
Code: Select all
   memcpy(
      fb1_pointer + y * fb1_screen_width * fb1_screen_bitdepth, // beginning of the row for fb1
      fb0_pointer + y * fb0_screen_width * fb1_screen_bitdepth, // beginning of the row for fb0
      fb1_screen_width * fb1_screen_bitdepth); // one row of pixel data for fb1

And, just nitpicking, but your variable name fb1_screen_bitdepth isn't descriptive of the value it holds as the bitdepth is 16, fb1_screen_bytes_per_pixel would be more accurate.

yo1dog wrote:As you can see from the comments above, I don't understand everything that is going on here and I bet there is a better way of doing this. The snapshot and reading above drops the framerate a bit. Is there a faster way to do this? Is there really not a way to snapshot just a portion of fb0?

I've tried various ways of getting the egl resource but to no avail. There is an egl function that should copy the colour buffer to a native pixmap - eglCopyBuffers(), but looking at the egl source code native pixmaps aren't supported, the code sets a null pointer for the bitmap data.
She who travels light - forgot something.
User avatar
Posts: 65
Joined: Wed Nov 23, 2011 1:10 am
Location: Sheffield, England
by yo1dog » Wed Oct 30, 2013 6:19 am
Thanks for all the great info and your time.

Paeryn wrote:The RPi does support 565, if you ask for it you will see the colour banding. To get it you have to pick the right config. The first config returned by eglChooseConfig() when asking for a 565 will NOT be 565 but 888 as the spec says that the configs are sorted with colour sizes that biggest first. Broadcom have an extra function eglSaneChooseConfigBRCM() which will return the list sorted with minimum first.

For some reason though glReadPixels only supports GL_RGBA,GL_UNSIGNED_BYTE. It should also support a second format returned from glGet() GL_IMPLEMENTATION_COLOR_READ_FORMAT_OES,GL_IMPLEMENTATION_COLOR_COLOR_TYPE_OES but glGet hardcodes these to return GL_RGBA and GL_UNSIGNED_BYTE rather than letting them have values corresponding to the colour buffer format.

So, the pixels rendered by openGL can be 565 format but when we read the pixel data it always converts it to 888? That seems silly. :P

Paeryn wrote:The memcpy parameters are wrong, it should be
Code: Select all
   memcpy(
      fb1_pointer + y * fb1_screen_width * fb1_screen_bitdepth, // beginning of the row for fb1
      fb0_pointer + y * fb0_screen_width * fb1_screen_bitdepth, // beginning of the row for fb0
      fb1_screen_width * fb1_screen_bitdepth); // one row of pixel data for fb1


Actually, because fb0/fb1_pointer's is of type short, it is not necessary to multiply by fb1_screen_bitdepth (2) because c knows the type is short (which is 2 bytes in size). When you add 1 to a pointer it advances the size of the type (sizeof(short) = 2) in bytes.

Paeryn wrote:And, just nitpicking, but your variable name fb1_screen_bitdepth isn't descriptive of the value it holds as the bitdepth is 16, fb1_screen_bytes_per_pixel would be more accurate.

I appreciate the correction. :)

Paeryn wrote:I've tried various ways of getting the egl resource but to no avail. There is an egl function that should copy the colour buffer to a native pixmap - eglCopyBuffers(), but looking at the egl source code native pixmaps aren't supported, the code sets a null pointer for the bitmap data.


So back to square one, eh?
Posts: 11
Joined: Tue Oct 22, 2013 11:18 pm
by karlkiste » Wed Oct 30, 2013 6:49 am
yo1dog wrote:[...] but the resolution was still higher than 480x272. Can anyone think of a way to force fb0 to be the resolution I need?


Just stumbled over it: Of course you can set that resolution:

in config.txt

disable_overscan=1
framebuffer_width=480
framebuffer_height=272

And, very important, do NOT plug in an HDMI monitor! The monitor will tell via EDID that it can't display such a low resolution, and the pi will set a resolution supported by the monitor. If no HDMI is connected, the output will go to CVBS, and the resolution can be set to whatever you want.
Posts: 149
Joined: Tue Jan 22, 2013 8:50 am
Location: berlin, germany
by yo1dog » Wed Oct 30, 2013 3:33 pm
Thanks! I'll try this tonight.
Posts: 11
Joined: Tue Oct 22, 2013 11:18 pm
by Paeryn » Wed Oct 30, 2013 5:19 pm
yo1dog wrote:Actually, because fb0/fb1_pointer's is of type short, it is not necessary to multiply by fb1_screen_bitdepth (2) because c knows the type is short (which is 2 bytes in size). When you add 1 to a pointer it advances the size of the type (sizeof(short) = 2) in bytes.

Oops, my mistake, the code I wrote I had the pointers as char*, didn't realise you'd declared them as short*. Sorry :oops:
She who travels light - forgot something.
User avatar
Posts: 65
Joined: Wed Nov 23, 2011 1:10 am
Location: Sheffield, England