Packing 4x 8bit EGL_IMAGE_BRCM_MULTIMEDIA_Y into 1x 32bit RBGA for shader processing

[Electroozz]

Hi all,

For a project I've been working on developing a C++ library for (mmal) camera communication, including PWM/flash synchronisation and OpenGL support (https://github.com/HesselM/flashcam).

Currently everything is running fine, but in my project I need to apply different shaders consecutively with some magic in between. I'm able to process Mode-5 images at 10fps, but it should be possible to increase this speed.

In sobel.c I found this comment:

Code: Select all

/* Example Sobel edge detct shader. The texture format for
 * EGL_IMAGE_BRCM_MULTIMEDIA_Y is a one byte per pixel greyscale GL_LUMINANCE.
 * If the output is to be fed into another image processing shader then it may
 * be worth changing this code to take 4 input Y pixels and pack the result
 * into a 32bpp RGBA pixel.
 */

As my library only support the Y-channel, it should be possible to do this, unfortunately, I'm not sure how to approach this.

The process of texture creation/mapping in my library is as follows:

• Videoport is set to use the OPAQUE parameter
• The callback processes frame updates by calling mmalbuf2TextureOES with the OPAQUE frame-data
• `mmalbuf2TextureOES` creates a `EGLImageKHR` in the same way as is done in RaspiTex (using `EGL_IMAGE_BRCM_MULTIMEDIA_Y`).
• Once created, the image is then pushed to a user defined callback
• In this callback I translate the EGLImageKHR image into an `RGBA` texture using the shader:

  Code: Select all

  #define SRC_FSHADER_OES2RGB \
      "#extension GL_OES_EGL_image_external : require\n" \
      "uniform samplerExternalOES tex;\n" \
      "varying vec2 texcoord;\n" \
      "void main(void) {\n" \
      "    gl_FragColor = texture2D(tex, texcoord);\n" \
      "}\n" 
  

And at this point I'm a bit confused. Intuitively I would think that the mapping from the 4x1byte luminance -> 4byte RGBA could be done by changing a single definition: assuming an image internally is represented as `8bit-pixdata 8bit-pixdata 8bit-pixdata 8bit-pixdata` and RGBA pixdata is stored consecutively, showing an image as Luminance or RGBA is just a matter of interpretation (and setting the width appropriately: RGBA should have a width of width/4 with respect to the Luminance). Yet, I fail the get a proper google-result on this matter (perhaps wrong search-keys?)

A different approach would be to update the shader and define a secondary `eglCreatePbufferSurface` and context with width/4, such that I could create an 2D-RGBA texture to which I sample the original EGLImageKHR-linked texture. A shader could then roughly look something like:

Code: Select all

void main(void) {
    vec4 color = vec4(0);
    color.r  = texture2D(tex, texcoord ).r;
    color.g = texture2D(tex, texcoord + 1/pixelwidth).r;
    color.b = texture2D(tex, texcoord + 2/pixelwidth).r;
    color.a = texture2D(tex, texcoord + 3/pixelwidth).r;
    gl_FragColor = color;
}

But this sounds a bit overcomplicating the challenge?

Has anyone some thoughts or experience on this? What would be the `best` way to go?
Many thanks is advance for pointers and/or help!

[Electroozz]

For others which may have similar question in the future: I was able to get in working by using a shader.

Code: Select all

#extension GL_OES_EGL_image_external : require
uniform samplerExternalOES tex;
uniform vec2 texelsize;
varying vec2 texcoord;
void main(void) {
	vec2 txc = texcoord
	txc.x    = txc.x*4.0
	vec4 col = vec4(0)
	col.r = texture2D(tex, txc+vec2(0,0)*texelsize).r
	col.g = texture2D(tex, txc+vec2(1,0)*texelsize).r
	col.b = texture2D(tex, txc+vec2(2,0)*texelsize).r
	col.a = texture2D(tex, txc+vec2(3,0)*texelsize).r
	gl_FragColor = col;
}

Where:

• `tex`: external texture. In my case this is eglImageKhr, created from the MMAL_BUFFER_HEADER
• `texelsize`: vec2(1/width, 1/height) of the original image size of the captured image (or source-image)
• `texcoord`: texturecoordinates for the result/target image. Defined by:

  Code: Select all

  attribute vec2 position;
  varying vec2 texcoord;
  void main(void) {
  	texcoord = position * vec2(0.5) + vec2(0.5);
  	gl_Position = vec4(position, 0.0, 1.0);
  }

The target image is created by:

Code: Select all

...
glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, width/4, height, 0, GL_RGBA, GL_UNSIGNED_BYTE, NULL); 
...

After applying the shader, each RGBA pixel represents four luminance in a row.