mavo
Posts: 16
Joined: Wed Aug 01, 2012 2:08 pm

video_render, OMX_EmptyThisBuffer, VSYNC / tearing

Tue Mar 12, 2019 8:43 am

Hi,

I'm working on a video application using the OMX video_decoder + video_render components, using a DPI display on the GPIO.

I'm not tunelling data between decoder out and render in, since i need to software-modify the data. So my application handles data between decode out and render in ports.

Since i'm not using the video_scheduler and clock components i'm timing the frames directly against the DPI VSYNC signal, to a clean 60 fps.

I'm using OMX_EmptyThisBuffer on the render component, when VSYNC occurs. My issue is that this causes tearing on the image on screen, as the data is updated mid-screen.

So my question is, where in OMX / raspberry pi is the frame presentation aligned with VSYNC? Does the video_render component not wait for vsync when sending data to it's input port?

The tearing must be caused by the time taken from OMX_EmptyThisBuffer until data is presented we are passed the VSYNC.

Any idea on how to fix the tearing? Would it be fixed by implementing the video_scheduler and does this also require the clock component?

My goal is to just serve 1 frame per VSYNC (does not matter if it's 58fps or 62fps).

When running my video using hello_video example, it does not tear. But this example also uses the video_scheduler.

Thanks.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7578
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: video_render, OMX_EmptyThisBuffer, VSYNC / tearing

Tue Mar 12, 2019 11:05 am

video_render sits on top of DispmanX, which is programmed by providing an "update" that is latched on the next vsync ready for the next frame. If you provide 2 updates before the vsync, then the first is discarded, the resources returned, and the second will take effect.
It is very hard to get that to tear.

Taking hello_video I've hacked out video_scheduler and clock - https://github.com/6by9/userland/tree/hello_video
Use youtube-dl to download something like https://www.youtube.com/watch?v=Cyxixzi2dgQ, and I don't see any tearing. There are skipped frames as the decoder can run faster than realtime and so there are multiple updates happening within a single rendered frame period.

Sorry, I haven't the time to hack it further to pull the decoded frames back to the ARM and then pass them to video_render. If you can post your test case then I can have a look.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

mavo
Posts: 16
Joined: Wed Aug 01, 2012 2:08 pm

Re: video_render, OMX_EmptyThisBuffer, VSYNC / tearing

Tue Mar 12, 2019 4:15 pm

Hi,

Thanks for responding.

My approach was to supply data to video_render using OMX_EmptyThisBuffer, on each VSYNC. The VSYNC is fixed at 60fps, and that's exactly what the frame data should match aswell.

What i experience is that tearing happens at a fixed location on screen, indicating that data is updated "late" but in sync with VSYNC.

I understand that calling OMX_EmptyThisBuffer twice without VSYNC will skip a frame, if i delay my OMX_EmptyThisBuffer i should then get 30 FPS. However it does not seem like OMX_EmptyThisBuffer it waiting for VSYNC to display the data.

Is there any way to check if the data already sent to the render using OMX_EmptyThisBuffer, has been displayed and it's safe to throw the next OMX_EmptyThisBuffer? Or any other way to sync against VSYNC?

Could this issue be caused by using an DPI LCD on GPIO, not HDMI?

I'll have a look at the link you sent.

Thanks.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7578
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: video_render, OMX_EmptyThisBuffer, VSYNC / tearing

Tue Mar 12, 2019 4:33 pm

Without your code or a test case I can't help further.

Calling OMX_EmptyThisBuffer actually has to do a memcpy of the image data from ARM memory to GPU memory. Achieving a tear in that memcpy is therefore pretty impressive, or you are mishandling the buffers and when they are submitted. How many buffers have you allocated video_renders input port? And you are waiting for the buffer to be returned before updating it?

HDMI, DSI, DPI, and composite video all go through the same paths, it's just the final destination of the pipe that is different. That final stage is responsible for creating the correct pixel clock and encoding the output, but that's about it.

Are you transposing? That has to go through a different path and can tear under some circumstances.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 24192
Joined: Sat Jul 30, 2011 7:41 pm

Re: video_render, OMX_EmptyThisBuffer, VSYNC / tearing

Tue Mar 12, 2019 4:51 pm

Is the tear diagonal? That would indicate a transposer tear.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I think it’s wrong that only one company makes the game Monopoly.” – Steven Wright

mavo
Posts: 16
Joined: Wed Aug 01, 2012 2:08 pm

Re: video_render, OMX_EmptyThisBuffer, VSYNC / tearing

Wed Mar 13, 2019 7:34 am

Hi,

The tearing is horizontal, exactly across 1 line. So top 20 lines shows the previous image, next 220 lines shows the new image.

I'll try to make a small version of my application which shows the issue.

Thanks.

mavo
Posts: 16
Joined: Wed Aug 01, 2012 2:08 pm

Re: video_render, OMX_EmptyThisBuffer, VSYNC / tearing

Wed Mar 13, 2019 9:47 am

Hi,

I have tried the hello_video stripped of clock and scheduler:
https://github.com/6by9/userland/tree/hello_video

Seems to work fine without tearing. This still uses tunelling though.

I'm not sure what transposing means, i am however using OMX_CONFIG_DISPLAYREGIONTYPE displayRegion; to create a region. I'm playing a square video in the center of a 4:3 screen, and using displayRegion to center the video output.

Thanks.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7578
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: video_render, OMX_EmptyThisBuffer, VSYNC / tearing

Wed Mar 13, 2019 11:42 am

Transpose - https://en.wikipedia.org/wiki/Transpose
Flipping on the line X=Y, as required by rotating 90 or 270 degreees.
it sounds like you aren't doing that.

The hacked hello_video does still use tunnelling as stripping that out is more significantly more effort, particularly with IL. If you really can't sort a simple test app I'll see if I can hack it about enough.
You could push your full app as a private repo on Github and ask me to look at it there with pointers to where the relevant IL code is. I mainly wanting to see roughly how you are handling the buffers rather than any specifics. I mainly want to know things like how many buffers are you using on the video_decode output and video_render input. I asked these questions earlier and you haven't answered them, so I really can't help further.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

mavo
Posts: 16
Joined: Wed Aug 01, 2012 2:08 pm

Re: video_render, OMX_EmptyThisBuffer, VSYNC / tearing

Wed Mar 13, 2019 12:15 pm

Hi,

Thanks for your support.

I'm sorry i did not answer all your questions, i was gathering information and trying things out.

The render input is set up with 3 buffers of 92160 bytes each.

While i was investigating the input buffers i found the source of the tearing issue. The render input buffers were not handled correctly, i was (mistakenly) not cycling through the 3 input buffers, but resubmitting the 1. buffer again and again. This kept the input fifo drained constantly.

I would however not have found this without your questions, and explanation on how the renderer is working. So thank you very much!

Thanks!

mavo
Posts: 16
Joined: Wed Aug 01, 2012 2:08 pm

Re: video_render, OMX_EmptyThisBuffer, VSYNC / tearing

Wed Mar 13, 2019 3:10 pm

Hi again,

What is the best solution to manually keep the 3 render input buffers full? Wait for EmptyBufferDone on the previously submitted buffer before using OMX_EmptyThisBuffer on the same buffer?

If i use OMX_EmptyThisBuffer as soon as there is 1 (of 3) EmptyBufferDone, it seems the render is skipping every other frame. Effectively running 60fps, but 60s of video only lasts 30s. I receive 2xEmptyBufferDone per VSYNC.

If i use OMX_EmptyThisBuffer when there are 2 (of 3) EmptyBufferDone, rendering seems to be perfect, but it does not seem like it's utilizing the entire fifo of 3.


Thanks.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7578
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: video_render, OMX_EmptyThisBuffer, VSYNC / tearing

Wed Mar 13, 2019 3:29 pm

mavo wrote:
Wed Mar 13, 2019 3:10 pm
What is the best solution to manually keep the 3 render input buffers full? Wait for EmptyBufferDone on the previously submitted buffer before using OMX_EmptyThisBuffer on the same buffer?
That is mandatory. Once you've called OMX_EmptyThisBuffer the buffer is not yours to modify until you've had the EmptyBufferDone callback.
Ignore that and you will get unexpected behaviours.
mavo wrote:If i use OMX_EmptyThisBuffer as soon as there is 1 (of 3) EmptyBufferDone, it seems the render is skipping every other frame. Effectively running 60fps, but 60s of video only lasts 30s. I receive 2xEmptyBufferDone per VSYNC.

If i use OMX_EmptyThisBuffer when there are 2 (of 3) EmptyBufferDone, rendering seems to be perfect, but it does not seem like it's utilizing the entire fifo of 3.
It's not a FIFO of 3, it is 3 buffers allocated. And actually nBufferCountMin is 2 as that is the minimum that is required by the component.

Submit buffer A and it will be put on screen until the vsync AFTER you submit buffer B.
At the vsync you'll get the EmptyBufferDone for buffer A. That would be the sensible point to submit buffer C (that you have prefilled), and start filling buffer A again. At the next vsync buffer C will be displayed, and buffer B will be returned through EmptyBufferDone, so submit buffer A.

The component requires 2 buffers to function, but to be more efficient having one additional buffer for the application to be filling makes more sense, so 3 buffers total.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

mavo
Posts: 16
Joined: Wed Aug 01, 2012 2:08 pm

Re: video_render, OMX_EmptyThisBuffer, VSYNC / tearing

Wed Mar 13, 2019 3:55 pm

Hi,

Thanks for explaining the inner workings, it makes sense know.

This also explains why resubmitting buffer 1 caused the tearing, since it is displayed immediately.


Thanks again!

Return to “OpenMAX”