video_scheduler / clock components -- playing video too fast

[ErinaceousJones]

I'm writing a program based on hello_video which is intended to seamlessly load .h264 video files on the fly -- one will be playing on a loop, and with a command sent through a FIFO, the program opens a new video file and fills the video_decode buffer with data straight away.

That's working -- it switches seamlessly between videos without any flickering or glitches, so that's good

However, no matter what I try, I can't get the program to stop playing "catch up" with freshly opened videos!

Here's what I've observed:

• Tell program to open `video1.h264` and play it on loop
• Program begins streaming that file at the correct framerate.
• Several seconds later, send command telling program to open `video2.h264`.
• Program begins streaming `video2`, but for several seconds, plays the video far too fast. After a while, it seems to "catch up" and continue playing it at normal speed.
• For all subsequent new videos opened, the time it takes to "catch up" becomes longer and longer -- "catch up" time seems to be directly related to the running time of the program, so the longer the program runs for, the longer the catch up time.

I've pored over various bits of documentation e.g. the OpenMAX IL specification
(reading the messy draft version at https://www.khronos.org/registry/omxil/ ... edline.pdf) -- and they describe how the clock component is responsible for telling video_scheduler when to dispatch frames to the renderer component (I think?).

Is there a way to reset the clock when I'm playing a new video? Or at least tell it that a new video has started?

I've tried setting OMX_BUFFERFLAG_STARTTIME flag on the video_decode buffer whenever I start a new video. Otherwise, it's set to OMX_BUFFERFLAG_TIME_UNKNOWN. Removing the STARTTIME flag stops the video from playing at all.

I've also tried setting the clock's start time to the current wall time (feeding OMX_IndexConfigTimeCurrentWallTime to the clock state's 'nStartTime' property), makes no difference either (read only property?).

The messy source code of my program is here:
https://gist.github.com/erinaceous/9ff14df424761073223b

[hjimbens]

Hi, I have similar code that does not exhibit the same problem. The only difference I can spot is that after setting the field nFlags of the buffer, I clear the timestamp:

Code: Select all

buf->nTimeStamp = omx_ticks_from_s64 (0);

[ErinaceousJones]

Thanks for the tip unfortunately zeroing out nTimeStamp also had no effect.
[bangs head against wall]

I've tried reinitializing the clock component now as well -- still no joy. Maybe one of the components has some internal counter which I need to reset? e.g. something in the video_scheduler? The documentation I've found for the broadcom components is very sparse and not very helpful:
http://home.nouwen.name/RaspberryPi/doc ... clock.html
http://home.nouwen.name/RaspberryPi/doc ... duler.html

[hjimbens]

Hi, another issue is that you set the start time flag more than once.

Code: Select all

buf->nFlags = OMX_BUFFERFLAG_STARTTIME;

In my code that happens only for the first packet of the first video.

[ErinaceousJones]

Ah, that may be why it's playing catchup.

Without setting STARTTIME at the beginning of every new video, it never actually plays the next videos, if I flush the buffer before playing them. The program continues running (I continue getting debug output), but no new video is rendered to the screen.

Possibly I should be simply adding to the video_decode buffer instead of flushing it -- doing that makes the next video play, but introduces a roughly 5-second lag as it processes the remaining video data in the buffer before reaching the new video's data. I tried changing the buffer offset when I open a new video --
buf->nOffset = buf->nFilledLen;
-- but that also seems to make no difference.

[etam89]

Hi,
I also have problem to figure how to pause a video. My problem is that after I pause the video and restart again. It goes very fast - seems program try to catch up with the amount of time I pause.

If you have a working code, could you kindly enough to post it?

Thanks in advance
Eugene

[SteveBooth]

I just had a breakthrough with this, and thought I would post my findings. I got videos to pause and play at the correct rate. The secret is to carefully follow the directions in the OMX Spec. Get v 1.1.2 of the doc here: https://www.khronos.org/registry/omxil/ ... cation.pdf.

Look at Pp 288. You will see:

To implement a seek on a chain of components, an IL client shall perform the following operations in order:
1. Pause the component through the use of OMX_SendCommand requesting a state transition to OMX_StatePause.
2. Stop the clock component’s media clock through the use of OMX_SetConfig on OMX_TIME_CONFIG_CLOCKSTATETYPE requesting a transition to OMX_TIME_ClockStateStopped.
3. Seek to the desired location through the use of OMX_SetConfig on OMX_IndexConfigTimePosition requesting the desired timestamp.
4. Flush all components.
5. Start the clock component’s media clock through the use of OMX_SetConfig on OMX_TIME_CONFIG_CLOCKSTATETYPE requesting a transition to either OMX_TIME_ClockStateRunning or OMX_TIME_ClockStateWaitingForStartTime.
6. Un-pause the component through the use of OMX_SendCommand requesting a state transition to OMX_StateExecuting.

In my case, I had to use OMX_SetParameter, and not OMX_SetConfig, and the OMX_IndexConfigTimePosition was invalid for me on the Pi. I had to use OMX_IndexConfigTimeClientStartTime, but at least it's working a LOT better now.

Hope that helps.

[oomek]

I know this is an old thread, but I have stumbled across the same issue. My tunneling setup is as follows:

Code: Select all

video_decode ---> video_scheduler ---> egl_render
                       ^
                       |
clock ---------------------> audio_render

I feed the frames from the ffmpeg splitter to the video_decode. When I reach the last frame I call av_seek_frame with a timestamp 0 hoping to get a gapless loop of the video. After rewind the video is tryig to catchup and is playing with the maximum speed. The audio playback is unaffected and is playing at normal speed.

So here is my question: Is there any way to just tell the clock or the scheduler to start counting from 0 without flushing the components?

[6by9]

It's a long time since I looked at the detail of this.

IIRC audio_render only supports being a clock master, updating the clock component as it receives data to be played.

video_scheduler will request callbacks from the clock just before the timestamp of the buffers it is given. If the time has already passed, then that callback happens immediately, hence the catchup that you get.

I don't think you can safely reset the time without a flush. Should there be any remaining buffers with video_scheduler at the point you reset then those will never be released, and the codec may be unable to decode the stream due to insufficient buffers. The codec may also need a flush to clear out old reference frames (although passing a new IDR frame should do that anyway).
The seek event sequence is documented in the spec (section 6.1.3 of 1.1.2 spec), and step 4 is flush all components.

What you could probably do is manipulate the timestamps coming from ffmepeg so that they are always incrementing. eg should your clip end at 10seconds, you feed the buffers back in to IL with 10seconds added to all the timestamps. The clock can then keep running as is, and everything should stay in sync.

[oomek]

Thanks for the reply. Regarding the video timestamp compensation. This is what I'm actually doing at the moment, but I was treating it as a workaround and hoping to find a better solution.

In short, Since an audio is the master I get the latest audio timestamp, multiply it by the repeat count and add it to the video timestamp. Unfortunately this solution makes the audio drift after few repeats, and it depends I think on the fps of the video.

I compensate that drift in the following way

For 60fps video I can just add

Code: Select all

omx_video_buffer->nTimeStamp.nLowPart += (last_audio_timestamp) * repeat_count);

For the 30fps videos I have to add

Code: Select all

omx_video_buffer->nTimeStamp.nLowPart += ((last_audio_timestamp + audio_frame_time) * repeat_count);

I think I need to find a better source of the audio end time value to be framerate agnostic.

[6by9]

fps of the video should have no effect.

last_audio_timestamp is likely to be for the start of the last frame, so you need to add the frame duration to it as well (relatively easy as it is uncompressed audio)

[oomek]

I did some tests and I finally know where this audio drift originates from.

It depends on the difference between the audio and video stream dutation, since both streams have different time bases when I reencode the audio ie. from 48kHz to 16kHz the drift changes.

I'm still stuck I'm afraid.

[oomek]

It seems I will have to stick to flushing the tunnels to make the video loop without any audio drifts, but there is one tiny roadblock that I have no idea how to overcome.

I'm not sure how can I check for EOS in the callback, so I do not stop the clock or flush the tunnels before the last callback has launched and the egl texture filled.

Code: Select all

static void fill_egl_texture_buffer ( void* data, COMPONENT_T* c )
{
	if ( OMX_FillThisBuffer (ILC_GET_HANDLE ( egl_render ), omx_egl_buffers[current_buffer] != OMX_ErrorNone )
		printf ( "OMX_FillThisBuffer failed for egl buffer in callback\n" );
	// I would need to signal here the main thread to flush and rewind, but only on the last frame
}

Can I somehow utilize this "data" pointer so it points to the current buffer?
Are flags carried over through the video_scheduler to the egl_render input port when I set them on the input port of video_decode?

Edit: ok, it was easy

Code: Select all

int is_last_frame = omx_egl_buffers[current_buffer]->nFlags & OMX_BUFFERFLAG_EOS;

But now I'm stuck again, when using egl_render it's impossible to pause properly. setting xScale to 0 on the clock has exactly the same problem. The video is pausing often too late and then it's trying to catch up on fast forward, but it fails anyway.

Why this API has to be such a pain in the back? Nothing is straight forward, you solve one issue, another two appear out of nowhere.