Pi4: MMAL video in GL (on DRM). Flickering texture. Any idea what might be wrong?

[dividuum]

Hi.

I've managed a basic MMAL based video playback on a DRM GL surface on the Pi4. I've now run into an odd issue that I don't have any explanation for. Have a look at this image:



Edit: Here's a video.

What you see in screen capture is a red gl surface with a single mmal decoded video rendered two times. What I observe is that the second time I draw the video texture it constantly flickers in an unpredictable way. In the snapshow above, both textures should have the same height and the right one is randomly cut off in the lower part. Some observations:

• Decoding a second video with another mmal pipeline and showing its texture works. No flickering in that case. Showing any of those textures a second time results in flickering for this additional texture.
• The flickering occurs even if the first texture is rendered completely outside the visible area.
• Drawing the texture happens in the main thread, while all the mmal work is in a background thread. Similar to this code here I swap out the reference to the current buffer. As drawing is a a separate action in my code, I've locked this section so that swapping out `current_buffer` only happens while the main thread isn't holding that lock while GlDraw'ing that texture. Unfortunately this doesn't help.
• I tried to only swap out the `current_buffer` both just before or after a eglSwapBuffers. Still flickers.
• I've incremented the number of EGL_LINUX_DMA_BUF_EXT buffers I cycle through. From my understanding I need at least two, so one is being worked on while the other is currently available to draw on the screen. I've upped that value to 3 and 10 and besides the additional memory usage it still flickers.
• Usually the drawing is triggered by user code that then calls the C code actually issuing the glDraw commands. I've modified that part so the C does does those two call instead upon a single invocation. That way there's way there's no way anything else between those two calls might change the GL state. It flickers.
• Oddly enough using a single glDrawArrays call with a glVertexAttribPointer pointing to 8 instead of the 4 vertices (essentially drawing two squares awkwardly connected at a single edge) works without flickering.
• It even flickers if I suspend the thread handing the mmal buffer headers containing new frames (So essentially if I sleep(10) here). So the buffer header holding the vcsm_handle shouldn't be touch in my understanding.

I'm running out of ideas at this point. It almost seems like drawing a GL_TEXTURE_EXTERNAL_OES texture once results in some internal state that then prevents it from being drawn a second time in the same frame without flickering. But that's pure speculation and so far I haven't been able to reproduce this issue by modifying the linked code. Of course it's not directly comparable as that code uses a X GL surface instead of DRM and I'm not sure if that might influence what happens. Any hint might be useful.

[aire39]

I've noticed that the colors of the image seem weird is that what the image should look like? which drm fourcc are you using for your egl import? Is the incoming image RGB or YUV?

What happens if you add a glFinish call before the eglSwapBuffers call?

[dividuum]

I've noticed that the colors of the image seem weird is that what the image should look like? which drm fourcc are you using for your egl import? Is the incoming image RGB or YUV?

DRM fourcc should be XB24. The isp component converts to RGBA and the colors are correct (it's 1:14 into this video). Here are the in/out ports of both video_decode and isp:

Code: Select all

-----[ mmal port ]------------
vc.ril.video_decode:in:0(H264)(0xa4107cd0)
-----[ mmal format ]------------
type: video, fourcc: H264
 bitrate: 0, framed: 1
 extra data: 43, 0xa41036f0
 width: 640, height: 480, (0,0,640,480)
 pixel aspect ratio: 1/1, frame rate: 1966080/65536
 buffers num: 128(opt 20, min 1), size: 16384(opt 81920, min: 2048), align: 0
-----[ mmal port ]------------
vc.ril.video_decode:out:0(OPQV)(0xa4102d20)
-----[ mmal format ]------------
type: video, fourcc: OPQV
 bitrate: 0, framed: 0
 extra data: 0, (nil)
 width: 640, height: 368, (0,0,640,360)
 pixel aspect ratio: 1/1, frame rate: 1966080/65536
 buffers num: 10(opt 10, min 3), size: 128(opt 128, min: 128), align: 0

-----[ mmal port ]------------
vc.ril.isp:in:0(OPQV)(0xa4106130)
-----[ mmal format ]------------
type: video, fourcc: OPQV
 bitrate: 0, framed: 0
 extra data: 0, (nil)
 width: 640, height: 368, (0,0,640,360)
 pixel aspect ratio: 1/1, frame rate: 1966080/65536
 buffers num: 10(opt 10, min 3), size: 128(opt 128, min: 128), align: 0
-----[ mmal port ]------------
vc.ril.isp:out:0(RGBA)(0xa4106440)
-----[ mmal format ]------------
type: video, fourcc: RGBA
 bitrate: 0, framed: 0
 extra data: 0, (nil)
 width: 640, height: 368, (0,0,640,360)
 pixel aspect ratio: 1/1, frame rate: 1966080/65536
 buffers num: 3(opt 1, min 1), size: 942080(opt 942080, min: 942080), align: 0

What happens if you add a glFinish call before the eglSwapBuffers call?

Didn't make any difference. I added it both after the texture draw call (so it's called between to two invocations of glDrawArray) and before eglSwapBuffers. Still flickers.

[6by9]

Probably unrelated as I read your issue as being on textures, but I am tracing through an issue with LibreElec at the moment where tearing happens. It looks like we're returning the DRM flip complete event early under some situations. Resolving that is more complex than it at first seems.


Can you add frame numbers to the top and bottom half of the images and check to see if the top and bottom halves are from the same image? I can't quite tell from your image, but it looks like the bottom half is from an earlier frame, in which case I have a guess.

The 3D hardware itself still requires an internal RGB format for the input data, however there is a dedicated Texture Formatter Unit (TFU) to do format conversions for the input data. Much of GL, DRM, and that stuff is sequenced using fences to block processing until the input frame is ready.
I'd hazard a guess that setting up the output to be dependent on the TFU conversion of 2 jobs is the issue (is GL smart enough to know that the two textures are taking the same source image and therefore only convert once?). That would mean that 3D has been signalled to produce the output frame before the second TFU job has completed, and therefore it is potentially reading stale data if it manages to overtake the TFU.
Having viewed the source video it may be the other way around as the diagnonal line in the bottom left is moving right to left. That would mean the 3D has started with stale data, and the TFU overtakes it.

Sorry, this is all conjecture, and I'm not familiar enough with the GL side to say for certain or have debug tools immediately to hand to confirm it.
Having the framenumbers would confirm whether top and bottom of the frames are coming from the same source or not. If so then we do have people we can call on to do a more indepth analysis, better still if we can have a simple test case that provokes the issue reliably.

[dividuum]

Can you add frame numbers to the top and bottom half of the images and check to see if the top and bottom halves are from the same image? I can't quite tell from your image, but it looks like the bottom half is from an earlier frame, in which case I have a guess.

Unfortunately that doesn't seem to be the case. I'm now playing a black/white flickering video and I see either a garbled white texture or a garbled black texture. Nothing mixed. Here's a short video (excuse the moiré effect).

I also slowed down the frequency of new video frames in my previous attempts (by sleep()'ing in the MMAL code that receives new output frames) and the texture produced seems correct. It's only when drawing it twice that it gets messed up.

The 3D hardware itself still requires an internal RGB format for the input data, however there is a dedicated Texture Formatter Unit (TFU) to do format conversions for the input data. Much of GL, DRM, and that stuff is sequenced using fences to block processing until the input frame is ready.

Interesting insight. Thanks. Does that also mean that the isp component transcodes from the opaque internal format from the H264 video decoder into RGBA and then the TPU has to do another pass? There's no direct way for that? Or does that overhead(?) not matter?

I'd hazard a guess that setting up the output to be dependent on the TFU conversion of 2 jobs is the issue (is GL smart enough to know that the two textures are taking the same source image and therefore only convert once?). That would mean that 3D has been signalled to produce the output frame before the second TFU job has completed, and therefore it is potentially reading stale data if it manages to overtake the TFU.
Having viewed the source video it may be the other way around as the diagnonal line in the bottom left is moving right to left. That would mean the 3D has started with stale data, and the TFU overtakes it.

Sounds plausible. I've noticed that taking a snapshot with vc_dispmanx_resource_read_data seems to fix it briefly (for less than 1s). Higher resolution videos that result in less than 60fps output also look correct most of the time. So it really looks like some kind of data race.

[6by9]

Can you add frame numbers to the top and bottom half of the images and check to see if the top and bottom halves are from the same image? I can't quite tell from your image, but it looks like the bottom half is from an earlier frame, in which case I have a guess.

Unfortunately that doesn't seem to be the case. I've now playing a black/white flickering video and I see either a garbled white frame or a garbled black texture. Nothing mixed. Here's a short video (excuse the moiré effect).

Are those freshly created textures on the change? In which case something may well be memsetting it to 0, leaving a transparent alpha channel.

I also slowed down the frequency of new video frames in my previous attempts (by sleep()'ing in the MMAL code that receives new output frames) and the texture produced seems correct. It's only when drawing it twice that it gets messed up.

Don't believe that's going to help as I'm suspecting it is the presentation to GL that is the issue.

The 3D hardware itself still requires an internal RGB format for the input data, however there is a dedicated Texture Formatter Unit (TFU) to do format conversions for the input data. Much of GL, DRM, and that stuff is sequenced using fences to block processing until the input frame is ready.

Interesting insight. Thanks. Does that also mean that the isp component transcodes from the opaque internal format from the H264 video decoder into RGBA and then the TPU has to do another pass? There's no direct way for that? Or does that overhead(?) not matter?

The TFU will quite happily convert YUV to the internal format (UIF), so no need to go through the ISP for that step.
The TFU and 3D blocks are the only thing that utilise UIF, therefore there is no option for the ISP to produce it. Let the TFU do the work for you.

I'd hazard a guess that setting up the output to be dependent on the TFU conversion of 2 jobs is the issue (is GL smart enough to know that the two textures are taking the same source image and therefore only convert once?). That would mean that 3D has been signalled to produce the output frame before the second TFU job has completed, and therefore it is potentially reading stale data if it manages to overtake the TFU.
Having viewed the source video it may be the other way around as the diagnonal line in the bottom left is moving right to left. That would mean the 3D has started with stale data, and the TFU overtakes it.

Sounds plausible. I've noticed that taking a snapshot with vc_dispmanx_resource_read_data seems to fix it briefly (for less than 1s). Higher resolution videos that result in less than 60fps output also look correct most of the time. So it really looks like some kind of data race.

How many output buffers are you using? That is where the DRM issue I'm looking at could hit you as GL could be updating the new buffer whilst it is still on the screen. I don't think that would cause the effect you're seeing though.

[dividuum]

Are those freshly created textures on the change? In which case something may well be memsetting it to 0, leaving a transparent alpha channel.

I'm not sure I fully understand what you're asking, but right now the code creates a configurable number (I tried from 2 to 10) number of textures with the code like this (error handling removed) once the decoder know the video format (MMAL_EVENT_FORMAT_CHANGED):

Code: Select all

EGLImage image = eglCreateImageKHR(display, EGL_NO_CONTEXT, EGL_LINUX_DMA_BUF_EXT, NULL, attribs);
glGenTextures(1, &buf->gl.texture);
glBindTexture(GL_TEXTURE_EXTERNAL_OES, buf->gl.texture);
glTexParameteri(GL_TEXTURE_EXTERNAL_OES, GL_TEXTURE_MIN_FILTER, GL_LINEAR);
glTexParameteri(GL_TEXTURE_EXTERNAL_OES, GL_TEXTURE_MAG_FILTER, GL_LINEAR);
glEGLImageTargetTexture2DOES(GL_TEXTURE_EXTERNAL_OES, image);
eglDestroyImageKHR(display, image);

Each of the `buf`s in the above code is tied to an MMAL output buffer of the isp component. Once a new mmal buffer is produced, similar to the drm_mmal example code I search for its buffer and then use that buffers gl.texture to draw it using code that essentially looks like this:

Code: Select all

glBindTexture(GL_TEXTURE_EXTERNAL_OES, buf->gl.texture);
glVertexAttribPointer(current_shader->postext, 4, GL_FLOAT, GL_FALSE, 0, vertices);
glEnableVertexAttribArray(current_shader->postext);
glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);

Don't believe that's going to help as I'm suspecting it is the presentation to GL that is the issue.

Yep. Doesn't help. The video I've linked shows that. In the linked video I `sleep(10)` after being handed a new mmal buffer from the isp output port and after keeping note on which of the generated textures is now filled with the new frame. The flickering occurs even when I disable the mmal connection between video_decoder and isp and stop feeding new raw h264 data into the decoder. From my understanding this should result in mmal being completely idle. So it indeed seems like a pure GL issue when rendering the same GL_TEXTURE_EXTERNAL_OES twice in a single vsync cycle.

The TFU and 3D blocks are the only thing that utilise UIF, therefore there is no option for the ISP to produce it. Let the TFU do the work for you.

Ok

How many output buffers are you using? That is where the DRM issue I'm looking at could hit you as GL could be updating the new buffer whilst it is still on the screen. I don't think that would cause the effect you're seeing though.

Definitely more than one. I tried between 2 and 10 with no difference.