Posts: 1
Joined: Wed Jul 12, 2017 3:52 pm

Streaming/moonlight to GL texture

Wed Jul 12, 2017 5:14 pm

EDIT: it turns out that for some reason the textureID changed, I just make it a global variable and now works!! still, very unstable and laggy but at least I can see the video now

Hello! I need some help =D
I am trying to stream a windows PC application to a raspberry pi with low latency, and apply some effects (locally, in the raspberry pi)...
For the streaming part I found moonlight-embedded, it works really really good! far better than any other solution I found.
But now I need to apply some visual effects that I can easily achieve using shaders, plus using shaders will give better performance!
So I found that moonlight uses OpenMAX for decoding (that's why it works so great) and I found that OpenMAX can render in to a opengles texture using OMX_UseEGLImage...
The problem is that I have no previous knowledge of OpenMAX so I think I am not using it very well, what I've done so far is this:

I grabbed this part of the moonlight code:
https://github.com/irtimmer/moonlight-e ... video/pi.c
and this code (its an OpenMAX to gl texture example, using OMX_UseEGLImage):
https://github.com/Apress/raspberry-pi- ... penMAX/EGL
(that both works flawlesly)
And tried to merge the example into the moonlight's pi.c. and build moonlight. What I did its basically this:

Here is some of the code, the rest it almost the same as the examples

the pi.c file has a function called decoder_renderer_setup, this one its being called at the begining... I removed all the video_render stuffs and used the egl_render instead with the respective ports...

Code: Select all


// callback
ilclient_set_fill_buffer_done_callback(client, my_fill_buffer_done, 0);


if(ilclient_create_component(client, &egl_render, "egl_render", ILCLIENT_DISABLE_ALL_PORTS | ILCLIENT_ENABLE_OUTPUT_BUFFERS) != 0){
  fprintf(stderr, "Can't create video decode\n");
  return -2;

list[1] = egl_render;

set_tunnel(tunnel, video_decode, 131, egl_render, 220);


the pi.c file has a function called decoder_renderer_submit_decode_unit, that its beeing called in a loop in a thread, so its iterative... so to make sure everything its in the same thread I inited all the opengl things (obviously just one time) in there, and executed the gl update things, and the OpenMAX egl render stuffs ( and commented out the pthread_create(&thread1, NULL, video_decode_test, eglImage); in CreateSimpleTexture2D ).

Code: Select all

static int decoder_renderer_submit_decode_unit(PDECODE_UNIT decodeUnit) {
  if((buf = ilclient_get_input_buffer(video_decode, 130, 1)) == NULL){
    fprintf(stderr, "Can't get video buffer\n");

  if( glInited == 0 ){
    UserData  userData;
    esInitContext ( &esContext );
    esContext.userData = &userData;
    esCreateWindow ( &esContext, "Simple Texture 2D", 1280, 720, ES_WINDOW_RGB );
    InitCtx ( &esContext );
    esRegisterDrawFunc ( &esContext, Draw );

    glInited = 1;

  // feed data and wait until we get port settings changed
  dest = buf->pBuffer;

  buf->nFilledLen = 0;

  buf->nOffset = 0;


  if(first_packet) {
    first_packet = 0;

  PLENTRY entry = gs_sps_fix(&decodeUnit->bufferList, GS_SPS_BITSTREAM_FIXUP);
  decodeUnit->bufferList = entry;
  while (entry != NULL) {
    memcpy(dest, entry->data, entry->length);
    buf->nFilledLen += entry->length;
    dest += entry->length;
    entry = entry->next;

    //update gl stuffs
   eglSwapBuffers(esContext.eglDisplay, esContext.eglSurface);

  if(port_settings_changed == 0 &&
    ((buf->nFilledLen > 0 && ilclient_remove_event(video_decode, OMX_EventPortSettingsChanged, 131, 0, 0, 1) == 0) ||
    (buf->nFilledLen == 0 && ilclient_wait_for_event(video_decode, OMX_EventPortSettingsChanged, 131, 0, 0, 1,
                        ILCLIENT_EVENT_ERROR | ILCLIENT_PARAMETER_CHANGED, 10000) == 0))) {
    port_settings_changed = 1;

    if(ilclient_setup_tunnel(tunnel, 0, 0) != 0){
      fprintf(stderr, "Can't setup video\n");

     //ilclient_change_component_state(video_render, OMX_StateExecuting);

     //config all the egl_render stuffs
     // Set egl_render to idle
     ilclient_change_component_state(egl_render, OMX_StateIdle);

     // Enable the output port and tell egl_render to use the texture as a buffer
     //ilclient_enable_port(egl_render, 221); //THIS BLOCKS SO CANT BE USED
     if (OMX_SendCommand(ILC_GET_HANDLE(egl_render), OMX_CommandPortEnable, 221, NULL) != OMX_ErrorNone)
     	    printf("OMX_CommandPortEnable failed.\n");

     if (OMX_UseEGLImage(ILC_GET_HANDLE(egl_render), &eglBuffer, 221, NULL, eglImage) != OMX_ErrorNone)
     	    printf("OMX_UseEGLImage failed.\n");

     // Set egl_render to executing
     ilclient_change_component_state(egl_render, OMX_StateExecuting);

     // Request egl_render to write data to the texture buffer
     if(OMX_FillThisBuffer(ILC_GET_HANDLE(egl_render), eglBuffer) != OMX_ErrorNone)
     	    printf("OMX_FillThisBuffer failed.\n");


  if(OMX_EmptyThisBuffer(ILC_GET_HANDLE(video_decode), buf) != OMX_ErrorNone){
    fprintf(stderr, "Can't empty video buffer\n");

  return DR_OK;
It opens the gl window, in fact if I change the fragment shader to gl_FragColor = vec(1,0,0,1); I do see an all red window. but sampling the texture all I see is black.

I would like to know if this is even possible, since the video example loads the video from file maybe rendering to gl texture from a stream is not possible?? of if someone has a working code that can share? or else some help!!

Thank you so much in advance!!!

Return to “OpenMAX”