tifiou
Posts: 5
Joined: Tue Aug 01, 2017 3:41 pm

OpenGL ES 2.0 - play .mp4 on texture

Tue Aug 01, 2017 3:52 pm

Hi, I am currently looking for a library allowing me to play a .mp4 on an OpenGL texture. I tried to do it in hello_videocube but with an .mp4 the textures are black and after research it appears we cannot play .mp4 (maybe i'm wrong). I saw it is possible with gst-omx but the only example I found it's this : https://github.com/01org/gst-omx/tree/m ... amples/egl, but I didn't understand how it works. Someone can give a sample or explain me how we do with gst-omx, or a library with good documentation or something else ?


Thanks,
best regards

User avatar
dividuum
Posts: 108
Joined: Sun Jun 16, 2013 1:18 pm
Location: Germany
Contact: Website

Re: OpenGL ES 2.0 - play .mp4 on texture

Tue Aug 01, 2017 11:29 pm

I don't think there's any library out there that will help you. hello_videocube is probably the best example for how to play videos on a texture. I don't think you'll find a lot of documentation either. The whole OMX aspect of the Pi is underdocumented and very frustrating....

If you want to play mp4 files, you'll have to demux them. You can't just send the complete mp4 bytestream to the decoder component. You'll have to demux the mp4 video (for example with libav) and only send the h264 video frames (and the decoder config (in video_codec_context->extradata, google keyword might be "OMX_BUFFERFLAG_CODECCONFIG")). Also make sure that you use the exact same defines (those -D arguments) that hello_video uses while compiling your code. Otherwise it won't work and you'll probably not get any helpful error message at all.

Coding for OMX on the Pi isn't a lot of fun. Prepare for pain. Also, FullHD videos are close or over the limit for playing videos through OpenGL. If you want to play videos reliably without framedrops you'll have to limit yourself to 720p.
info-beamer hosted - A user and programmer friendly digital signage platform for the Pi: https://info-beamer.com/hosted

tifiou
Posts: 5
Joined: Tue Aug 01, 2017 3:41 pm

Re: OpenGL ES 2.0 - play .mp4 on texture

Wed Aug 02, 2017 12:58 pm

Oh jesus.... I already feel the headache :cry: but I 'm going to try. Thanks for your answer :)


EDIT : Seriously, even libav has no tutorial ? :|

tifiou
Posts: 5
Joined: Tue Aug 01, 2017 3:41 pm

Re: OpenGL ES 2.0 - play .mp4 on texture

Wed Aug 02, 2017 9:15 pm

Ok, I used libav to convert .mp4 in .h264 it works with hello_video, but not in hello_videocube (always black textures )... any idea, please ?

User avatar
dividuum
Posts: 108
Joined: Sun Jun 16, 2013 1:18 pm
Location: Germany
Contact: Website

Re: OpenGL ES 2.0 - play .mp4 on texture

Fri Aug 04, 2017 5:53 pm

tifiou wrote:
Wed Aug 02, 2017 9:15 pm
Ok, I used libav to convert .mp4 in .h264 it works with hello_video, but not in hello_videocube (always black textures )... any idea, please ?
Any error message at all? Does the video play with omxplayer? What changes did you make to hello_videocube? Or did you only switch out the referenced hello.h264 file?

By the way: When I wrote to use libav, I didn't mean that you should preprocess the video, but open it with the libav API (e.g. avformat_open_input, avformat_find_stream_info), read the packets and only send the video packets to the OMX pipeline.
info-beamer hosted - A user and programmer friendly digital signage platform for the Pi: https://info-beamer.com/hosted

tvjon
Posts: 624
Joined: Mon Jan 07, 2013 9:11 am

Re: OpenGL ES 2.0 - play .mp4 on texture

Sat Aug 05, 2017 7:46 pm

"and after research it appears we cannot play .mp4 (maybe i'm wrong)."

As you can see in the attached screenshot, a demuxed (I use gpac where possible) MP4 will work *providing* the MP4 matches several of "Big Buck Bunny"'s attributes, particularly frame resolution.

As yet I don't know what needs to be changed in the source files to allow different file encoding to work. The same situation applies to hello_teapot.

vcube.jpg
vcube.jpg (63.6 KiB) Viewed 4317 times

tvjon
Posts: 624
Joined: Mon Jan 07, 2013 9:11 am

Re: OpenGL ES 2.0 - play .mp4 on texture

Sun Aug 06, 2017 4:33 pm

"As yet I don't know what needs to be changed in the source files to allow different file encoding to work"

Ok, now I know.

Use

mediainfo

to get your .h264 file details.

In

triangle.c

change these 2 definitions to your video size

#define IMAGE_SIZE_WIDTH 1280
#define IMAGE_SIZE_HEIGHT 674

airlander.jpg
airlander.jpg (40.54 KiB) Viewed 4236 times

tifiou
Posts: 5
Joined: Tue Aug 01, 2017 3:41 pm

Re: OpenGL ES 2.0 - play .mp4 on texture

Mon Aug 14, 2017 4:04 pm

Sorry, I was in the countryside without internet...
dividuum wrote:
Fri Aug 04, 2017 5:53 pm
tifiou wrote:
Wed Aug 02, 2017 9:15 pm
Ok, I used libav to convert .mp4 in .h264 it works with hello_video, but not in hello_videocube (always black textures )... any idea, please ?
Any error message at all? Does the video play with omxplayer? What changes did you make to hello_videocube? Or did you only switch out the referenced hello.h264 file?

By the way: When I wrote to use libav, I didn't mean that you should preprocess the video, but open it with the libav API (e.g. avformat_open_input, avformat_find_stream_info), read the packets and only send the video packets to the OMX pipeline.
I found the problem, I was so focus on video.c that I didn't see the fixed size of the texture in triangle.c ....

I am still try to understand how works OMXPlayer, so when you say "read the packets and only send the video packets to the OMX pipeline" , you mean at line "(OMX_UseEGLImage(ILC_GET_HANDLE(egl_render), &eglBuffer, 221, NULL, eglImage) != OMX_ErrorNone)" instead of eglImage I send the packets ?

thanks again for your help.

@tvjon : thanks for your answer, I changed the fixed size and it worked. But it's sad that we must convert to h264 the sound is enclosed... :/


P.s : I have a little question if you allow it, in the OMX_Core.h, I saw that we can pause the video (in omx_statetype) but not accelerate it, any idea why ?

tifiou
Posts: 5
Joined: Tue Aug 01, 2017 3:41 pm

Re: OpenGL ES 2.0 - play .mp4 on texture

Mon Aug 21, 2017 7:45 pm

I made some progress with openMax but I block in rendering on the texture. In the example they use eglCreateImageKHR, I read the doc but I didn't really undestand how it work like how can it display on the screen without texture ...

here is a part of my source code :

Code: Select all

void componentOMX::readVideo(void* arg){
  int err;

  if(eglImage == 0){
    std::cerr << "eglImage is null.\n";
    exit(1);
  }
  
  err = ilclient_enable_port_buffers(decodeComponent, 130, NULL, NULL, NULL);
  if(err < 0){
    std::cerr << "couldn't enable buffers" << std::endl;
    return;
  }

  ilclient_enable_port(decodeComponent, 130);
  
  err = ilclient_change_component_state(decodeComponent, OMX_StateExecuting);
  if(err < 0){
    std::cerr << "couldn't change state to Executing" << std::endl;
    return;
  }

  config.setDecoderConfig(decodeComponent, demux.getExtradataSize(), demux.getExtradata());
  config.setClockState(clockComponent);
  
  AVPacket pkt = demux.getPacket();
  
  while(av_read_frame(demux.getContext(), &pkt) >= 0){
    AVPacket origPacket = pkt;
    
    if(pkt.stream_index == demux.getVideoIndex()){
      omxBuffer = ilclient_get_input_buffer(decodeComponent, 130, 1);
      if(omxBuffer != NULL)
	config.copyInBufferAndEmpty(&pkt, decodeComponent, omxBuffer, demux.getTimebase());
      
      err = ilclient_wait_for_event(decodeComponent, OMX_EventPortSettingsChanged, 131, 0, 0, 1, ILCLIENT_EVENT_ERROR | ILCLIENT_PARAMETER_CHANGED, 0);

      if(err >= 0)
	break;

      if(ilclient_remove_event(decodeComponent, OMX_EventPortSettingsChanged, 131, 0, 0, 1) == 0){
	break;
      }
    }
    av_free_packet(&origPacket);
    //    pkt = demux.getPacket();
  }
  
  TUNNEL_T decodeTunnel;
  set_tunnel(&decodeTunnel, decodeComponent, 131, schedulerComponent, 10);
  if( (err = ilclient_setup_tunnel(&decodeTunnel, 0, 0)) < 0){
    std::cerr << "error setting up decode tunnel" << std::endl;
    return;
  }

  TUNNEL_T schedulerTunnel;
  set_tunnel(&schedulerTunnel, schedulerComponent, 11, renderComponent, 220);//90);
  if( (err = ilclient_setup_tunnel(&schedulerTunnel, 0, 1000)) < 0){
    std::cerr << "error setting up scheduler tunnel" << std::endl;
    return;
  }
  
  TUNNEL_T clockTunnel;
  set_tunnel(&clockTunnel, clockComponent, 80, schedulerComponent, 12);
  if( (err = ilclient_setup_tunnel(&clockTunnel, 0, 0)) < 0){
    std::cerr << "error setting up clock tunnel" << std::endl;
    return;
  }
  else
    ilclient_change_component_state(clockComponent, OMX_StateExecuting);
  
  OMX_SendCommand(ilclient_get_handle(decodeComponent), OMX_CommandPortEnable, 131, NULL);
  ilclient_enable_port(decodeComponent, 131);

  OMX_SendCommand(ilclient_get_handle(clockComponent), OMX_CommandPortEnable, 80, NULL);
  ilclient_enable_port(clockComponent, 80);

  OMX_SendCommand(ilclient_get_handle(schedulerComponent), OMX_CommandPortEnable, 10, NULL);
  ilclient_enable_port(schedulerComponent, 10);
  OMX_SendCommand(ilclient_get_handle(schedulerComponent), OMX_CommandPortEnable, 11, NULL);
  ilclient_enable_port(schedulerComponent, 11);
  OMX_SendCommand(ilclient_get_handle(schedulerComponent), OMX_CommandPortEnable, 12, NULL);
  ilclient_enable_port(schedulerComponent, 12);
  
  // OMX_SendCommand(ilclient_get_handle(renderComponent), OMX_CommandPortEnable, 221, NULL);
  //   ilclient_enable_port(renderComponent, 221);

  err = ilclient_change_component_state(decodeComponent, OMX_StateExecuting);
  if(err < 0){
    std::cerr << "couldn't change state to Executing" << std::endl;
    return ;
  }

  // err = ilclient_change_component_state(renderComponent, OMX_StateExecuting);
  // if(err < 0){
  //   std::cerr << "couldn't change state to Executing" << std::endl;
  //   return ;
  // }
  
  err = ilclient_change_component_state(schedulerComponent, OMX_StateExecuting);
  if(err < 0){
    std::cerr << "couldn't change state to Executing" << std::endl;
    return ;
  }

  err = ilclient_change_component_state(clockComponent, OMX_StateExecuting);
  if(err < 0){
    std::cerr << "couldn't change state to Executing" << std::endl;
    return ;
  }

  //  av_seek_frame(demux.getContext(), demux.getVideoIndex(), 0, AVSEEK_FLAG_BACKWARD);
  while(1){  
    while(av_read_frame(demux.getContext(), &pkt) >= 0){
      if(pkt.stream_index != demux.getVideoIndex())
  	continue;

      omxBuffer = ilclient_get_input_buffer(decodeComponent, 130, 1);
      
      if(omxBuffer != NULL)
  	config.copyInBufferAndEmpty(&pkt, decodeComponent, omxBuffer, demux.getTimebase());

      err = ilclient_wait_for_event(decodeComponent, OMX_EventPortSettingsChanged, 131, 0, 0, 1, ILCLIENT_EVENT_ERROR | ILCLIENT_PARAMETER_CHANGED, 10000);

      if(err >= 0)
  	std::cerr << "Another port setting change" << std::endl;

      if(ilclient_setup_tunnel(&schedulerTunnel, 0, 1000) != 0){
	break;
      }

      err = ilclient_change_component_state(renderComponent, OMX_StateIdle);
      
      if (OMX_SendCommand(ilclient_get_handle(renderComponent), OMX_CommandPortEnable, 221, NULL) != OMX_ErrorNone){
      	printf("OMX_CommandPortEnable failed.\n");
      	exit(1);
      }

      if (OMX_UseEGLImage(ILC_GET_HANDLE(renderComponent), &omxBuffer, 221, NULL, eglImage) != OMX_ErrorNone){
	std::cerr << "OMX_UseEGLImage failed.\n";
	exit(1);
      }
      
      ilclient_change_component_state(renderComponent, OMX_StateExecuting);
      if(err < 0){
      	std::cerr << "couldn't change state to Executing for EGLRENDER" << std::endl;
      	return ;
      }
      
      glUniformMatrix4fv(glGetUniformLocation(pIdVideos, "projection"), 1, GL_FALSE, glm::value_ptr(projectionVideos));
      glUniformMatrix4fv(glGetUniformLocation(pIdVideos, "view"), 1, GL_FALSE, glm::value_ptr(viewVideos));
      glUniformMatrix4fv(glGetUniformLocation(pIdVideos, "model"), 1, GL_FALSE, glm::value_ptr(modelVideos));
    }
    av_seek_frame(demux.getContext(), demux.getVideoIndex(), 0, 0); 
  }
  ilclient_wait_for_event(renderComponent, OMX_EventBufferFlag, 90, 0, OMX_BUFFERFLAG_EOS, 0, ILCLIENT_BUFFER_FLAG_EOS, 10000);
  return;
}

void componentOMX::videoToTexture(EGLDisplay display,  EGLContext context, glm::mat4 projection, glm::mat4 view, GLint pIdVideo){
  pIdVideos = pIdVideo;
  
  projectionVideos = projectionVideos;
  viewVideos = view;
  
  GLint tex0 = glGetUniformLocation(pIdVideos, "texture0");
  if(tex0  < 0){
    std::cerr << "texture video failed..." << std::endl;
    return;
   }
  int size = demux.getWidth() * demux.getHeight() * 4;
  unsigned char* pix = new unsigned char[size];
  
  glGenTextures(1, &tex);

  glBindTexture(GL_TEXTURE_2D, tex);
  glTexImage2D(GL_TEXTURE_2D, 0, GL_RGBA, demux.getWidth(), demux.getHeight(), 0,
	       GL_RGBA, GL_UNSIGNED_BYTE, pix);

  glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, GL_NEAREST);
  glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST);
  glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
  glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);


  std::cerr << "wezsh weseh" << std::endl;
  
  if(display == NULL){
    std::cerr << "display is null\n";
    exit(1);
  }

  if(context == NULL){
    std::cerr << "context is null\n";
    exit(1);
  }

  
  eglImage = eglCreateImageKHR(
			       display,
			       context,
			       EGL_GL_TEXTURE_2D_KHR,
			       (EGLClientBuffer)tex,
			       0);


  if(eglImage != EGL_NO_IMAGE_KHR)
    std::cerr << "mouais ..." << std::endl;

  glUseProgram(pIdVideos);
  glUniform1i(tex0, 0);
  glActiveTexture(GL_TEXTURE0);
  glBindTexture(GL_TEXTURE_2D, tex);
  drawQuad->draw();
  //  glEGLImageTargetTexture2DOES(GL_TEXTURE_EXTERNAL_OES, eglImage);
  readVideo(eglImage);
  //  std::thread thread1(&componentOMX::readVideo, this, eglImage);

  glBindTexture(GL_TEXTURE_2D, 0);
  //std::cerr << "....." << std::endl;
  glUseProgram(0);
  //  thread1.join();
}


any help would be appreciated, thanks

luc4
Posts: 33
Joined: Mon Nov 12, 2012 12:28 am

Re: OpenGL ES 2.0 - play .mp4 on texture

Thu Nov 23, 2017 9:29 pm

If you want my code uses OMX_UseEGLImage to play video to texture. This is the relevant portion: https://github.com/carlonluca/pot/blob/ ... XVideo.cpp. This is the result: https://www.youtube.com/watch?v=SeJxQN-W2uA&t=3s you can get. Performance is worst than omxplayer.
Regards.

Return to “OpenGLES”

Who is online

Users browsing this forum: No registered users and 1 guest