Posts: 8
Joined: Thu Jun 01, 2017 10:00 am

Packing 4x 8bit EGL_IMAGE_BRCM_MULTIMEDIA_Y into 1x 32bit RBGA for shader processing

Mon Sep 11, 2017 11:57 am

Hi all,

For a project I've been working on developing a C++ library for (mmal) camera communication, including PWM/flash synchronisation and OpenGL support (https://github.com/HesselM/flashcam).

Currently everything is running fine, but in my project I need to apply different shaders consecutively with some magic in between. I'm able to process Mode-5 images at 10fps, but it should be possible to increase this speed.

In sobel.c I found this comment:

Code: Select all

/* Example Sobel edge detct shader. The texture format for
 * EGL_IMAGE_BRCM_MULTIMEDIA_Y is a one byte per pixel greyscale GL_LUMINANCE.
 * If the output is to be fed into another image processing shader then it may
 * be worth changing this code to take 4 input Y pixels and pack the result
 * into a 32bpp RGBA pixel.
As my library only support the Y-channel, it should be possible to do this, unfortunately, I'm not sure how to approach this.

The process of texture creation/mapping in my library is as follows:
  • Videoport is set to use the OPAQUE parameter
  • The callback processes frame updates by calling mmalbuf2TextureOES with the OPAQUE frame-data
  • `mmalbuf2TextureOES` creates a `EGLImageKHR` in the same way as is done in RaspiTex (using `EGL_IMAGE_BRCM_MULTIMEDIA_Y`).
  • Once created, the image is then pushed to a user defined callback
  • In this callback I translate the EGLImageKHR image into an `RGBA` texture using the shader:

    Code: Select all

    #define SRC_FSHADER_OES2RGB \
        "#extension GL_OES_EGL_image_external : require\n" \
        "uniform samplerExternalOES tex;\n" \
        "varying vec2 texcoord;\n" \
        "void main(void) {\n" \
        "    gl_FragColor = texture2D(tex, texcoord);\n" \
And at this point I'm a bit confused. Intuitively I would think that the mapping from the 4x1byte luminance -> 4byte RGBA could be done by changing a single definition: assuming an image internally is represented as `8bit-pixdata 8bit-pixdata 8bit-pixdata 8bit-pixdata` and RGBA pixdata is stored consecutively, showing an image as Luminance or RGBA is just a matter of interpretation (and setting the width appropriately: RGBA should have a width of width/4 with respect to the Luminance). Yet, I fail the get a proper google-result on this matter (perhaps wrong search-keys?)

A different approach would be to update the shader and define a secondary `eglCreatePbufferSurface` and context with width/4, such that I could create an 2D-RGBA texture to which I sample the original EGLImageKHR-linked texture. A shader could then roughly look something like:

Code: Select all

void main(void) {
    vec4 color = vec4(0);
    color.r  = texture2D(tex, texcoord ).r;
    color.g = texture2D(tex, texcoord + 1/pixelwidth).r;
    color.b = texture2D(tex, texcoord + 2/pixelwidth).r;
    color.a = texture2D(tex, texcoord + 3/pixelwidth).r;
    gl_FragColor = color;
But this sounds a bit overcomplicating the challenge?

Has anyone some thoughts or experience on this? What would be the `best` way to go?
Many thanks is advance for pointers and/or help!

Posts: 8
Joined: Thu Jun 01, 2017 10:00 am

Re: Packing 4x 8bit EGL_IMAGE_BRCM_MULTIMEDIA_Y into 1x 32bit RBGA for shader processing

Sat Sep 16, 2017 8:38 am

For others which may have similar question in the future: I was able to get in working by using a shader.

Code: Select all

#extension GL_OES_EGL_image_external : require
uniform samplerExternalOES tex;
uniform vec2 texelsize;
varying vec2 texcoord;
void main(void) {
	vec2 txc = texcoord
	txc.x    = txc.x*4.0
	vec4 col = vec4(0)
	col.r = texture2D(tex, txc+vec2(0,0)*texelsize).r
	col.g = texture2D(tex, txc+vec2(1,0)*texelsize).r
	col.b = texture2D(tex, txc+vec2(2,0)*texelsize).r
	col.a = texture2D(tex, txc+vec2(3,0)*texelsize).r
	gl_FragColor = col;
  • `tex`: external texture. In my case this is eglImageKhr, created from the MMAL_BUFFER_HEADER
  • `texelsize`: vec2(1/width, 1/height) of the original image size of the captured image (or source-image)
  • `texcoord`: texturecoordinates for the result/target image. Defined by:

    Code: Select all

    attribute vec2 position;
    varying vec2 texcoord;
    void main(void) {
    	texcoord = position * vec2(0.5) + vec2(0.5);
    	gl_Position = vec4(position, 0.0, 1.0);
The target image is created by:

Code: Select all

glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, width/4, height, 0, GL_RGBA, GL_UNSIGNED_BYTE, NULL); 
After applying the shader, each RGBA pixel represents four luminance in a row.

Return to “OpenGLES”